Skip to content
  • There are no suggestions because the search field is empty.

Tutorial: Transform Pipeline

What Does This Article Cover?

This tutorial will walk you through building a Pipeline to facilitate use cases that require transformations to the data structure. This Tutorial will feature Flatten, Breakup and Write New stages.

  • Tutorial Prerequisites
  • Tutorial Preparation
  • Tutorial Instructions
  • Tutorial Summary

Tutorial Prerequisites

  1. You have completed the initial Installation of the Intelligence Hub
  2. You have applied your Intelligence Hub license to your environment
  3. It is recommended to read the Getting Started Series before completing this tutorial exercise
  4. It is recommended to complete "Tutorial: Connections", "Tutorial: Models and Instances" and "Tutorial: Templating"

Tutorial Preparation

  1. Enable Intelligence Hub MQTT broker

    • In the left-hand navigation panel, navigate to Manage, and click Settings
    • Under the MQTT Broker section enable the broker, if ports 1885 and 1886 are being utilized on your Intelligence Hub server, update to ports of your choosing, otherwise accept the defaults and click save

  2. Import the required Connections

    • In the left-hand navigation panel, navigate to Manage, and click Project
    • Within the Import screen, ensure Full Project is off (otherwise your existing project will be overwritten)
    • Change the Import Type to JSON and paste the following code block into the Project box and click the import button

Configuration Download

  1. Update the imported Connections as required

    • Navigate to Configure and click Connections, Click Tutorial_MQTT and Update the MQTT settings as required based on the prior preparation step #1

    • Navigate to Configure and click Connections, Click "Tutorial_CMMS_servicelogs" and type the following within the password field and click "Save"

                                password

  2. Setup UNS Client

    • In the left-hand navigation panel, navigate to Tools and right click UNS client and open Link in New Tab
    • Enter login information
    • For Connection select Tutorial_MQTT
    • For Subscribed Topics remove the default wildcard entry # and Subscribe to Topic "Tutorial/#"
    • Click Add
    • Click Connect and confirm UNS client says "Connected to Tutorial_MQTT"
    • Return to the previous tab

Tutorial Instructions

  1. Review the source data

    • Navigate to Instances

    • Open Enterprise_CNC_Model_Instance

    • Click Test Instance to review the source data

      • Take note we have a complex payload as a result of the parent - child model configuration

  2. Build a new Pipeline

    • Navigate to Pipelines
    • Click New Pipeline in the upper right corner
    • Within the Pipeline Start stage, specify "Tutorial_Transform_Pipeline" for the Name

  3. Add a WriteNew Stage to the Pipeline to write out original payload

    • Within the Stages panel, add a WriteNew stage by dragging it to the working space
    • Enter WriteNew_Original as the WriteNew stage name
    • Connect the Pipeline start with the WriteNew stage
    • Update the WriteNew stage name to WriteNew_Original
    • Using the References panel, map Tutorial_MQTT to the Use Connection field within the Write New stage
    • Within the new Write New stage, enter "Tutorial/Transform/Original" as the Topic
    • Enable the retain flag
    • Click submit

  1. Create a Flow to the Buffer Pipeline

    • Navigate to Flows
    • Click new Flow
    • Within the New Flow details view, enter "Tutorial_Transform_Pipeline_Flow"
    • Click next
    • Within the New Flow Sources & Targets view, use the reference panel and set the Type to Instance
    • Map Enterprise_CNC_Model_Instance to the Sources panel
    • From the References panel, set type to Pipeline
    • Map Tutorial_Transform_Pipeline to the Targets panel
    • Click next
    • Set the interval to 5 seconds, enable the flow and click submit
    • Review the results within your UNS client

  2. Add an additional transform to the Pipeline

    • Navigate to Pipelines

    • Click Tutorial_Transform_Pipeline

    • Add a Flatten stage by dragging it to the working space

    • Connect the Pipeline start with the Flatten stage

    • Within the Stages panel, add a WriteNew stage by dragging it to the working space

    • Update the WriteNew stage name to WriteNew_Flatten

    • Using the References panel, map Tutorial_MQTT to the Use Connection field within the Write New stage

    • Within the new Write New stage, enter "Tutorial/Transform/Flattened" as the Topic

    • Enable the retain flag

    • Click Save

    • Review the results within your UNS client

      • Take note the child instance attributes "Values" and "ProgramInfo" have been flatten into a single JSON object

  1. Add an additional transform to the Pipeline

    • Navigate to Pipelines

    • Click Tutorial_Transform_Pipeline

    • Add a Breakup stage by dragging it to the working space

    • Connect Flatten stage with the Breakup stage

    • Within the Breakup Stage, set the Breakup Type to All

    • Within the Stages panel, add a WriteNew stage by dragging it to the working space

    • Update the WriteNew stage to name WriteNew_Breakup

    • Connect the Pipeline start with the WriteNew_Breakup Stage

    • Using the References panel, map Tutorial_MQTT to the Use Connection field within the Write New stage

    • Within the new Write New stage, enter "Tutorial/Transform/Breakup/{{event.metadata.breakupName}}" as the Topic

    • Enable the retain flag

    • Review the results within your UNS client

      • NOTE: We can see the values are now broken up into individual MQTT topics using the {{event.metadata.breakupName}} value in our topic, this includes our flattened child attributes

Tutorial Summary

In this tutorial you built a Pipeline composed on Flatten, Breakup and Write New stages. These stages are commonly utilized in use cases requiring data structure change. By utilizing Pipelines, Data Engineers can ensure the data is outputted to target system(s) in the required or expected format.