Skip to content
  • There are no suggestions because the search field is empty.

Tutorial: File Connections

This step-by-step interactive tutorial provides an introduction to the Intelligence Hub File Connection.

What Does This Tutorial Cover?

This tutorial will walk you through the basics of working with File Connections.  A Connection represents a pathway to either a data source or a target, serving as the starting point for all use cases.

  • Tutorial Prerequisites
  • Tutorial Preparation
  • Tutorial Instructions
  • Tutorial Summary

Tutorial Prerequisites

  • You have completed the initial Installation of the Intelligence Hub.
  • It is recommended to read the Getting Started Series before completing this tutorial exercise.
  • It is recommended to complete "Tutorial: Connections".

Tutorial Preparation

  1. Enable Intelligence Hub MQTT broker

    • In the left-hand navigation panel, navigate to Manage, and click Settings.
    • Under the MQTT Broker section enable the broker, if ports 1885 and 1886 are being utilized on your Intelligence Hub server, update to ports of your choosing, otherwise accept the defaults and click the Save button.

  2. Import the required Connections

    • In the left-hand navigation panel, navigate to Manage, and click Project.
    • Within the Import screen, ensure Full Project is off (otherwise your existing project will be overwritten).
    • Copy the JSON provided here. Project JSON
    • Change the Import Type to JSON and paste the provided JSON into the Project box and click the Import button.

  3. Update the imported Connections as required

    • Navigate to Configure and click Connections, click "HB_Tutorial_MQTT" and Update the MQTT settings as required based on the prior preparation step #1.
    • Navigate to Configure and click Connections, click "HB_Tutorial_File" and update the file directory to a local directory on the Intelligence Hub server.  This tutorial will not utilize the processed or error directory.
  4. Import the source files
    • Right click and save the following three images to the local file directory from the previous step.
      • cnc_1001.jpg

      • cnc_1002.jpg

      • cnc_1003.jpg



  5. Setup UNS Client
    • In the left-hand navigation panel, navigate to Tools and right click UNS Client and select Open Link in New Tab.
    • Enter login information.

    • For Connection select "HB_Tutorial_MQTT".

    • For Subscribed Topics enter "Tutorial/#".

    • Click Add.

    • Click the "x" to remove topic "#".

    • Click the Connect button.

    • Confirm the UNS Client says "Connected to HB_Tutorial_MQTT".

    • Return to the previous web browser tab.

Instructions

  1. Login to your Intelligence Hub environment.

  2. Review the source data.

    • Navigate to Connections and select "HB_Tutorial_File".

    • Navigate to the Inputs tab.

    • Select "HB_Tutorial_File_Input_Read_Wildcard".

      • Take note of the file name structure, this is the default new input for File/CSV connections.  Consider it a wildcard that will read the whole directory without any additional logic.

    • Within the right hand References Panel, set type to Input.

    • Set Connections to "HB_Tutorial_File".

    • Find the Input you opened and take note of the object’s shape.

    • Now enable “Include Metadata” within the Input details

    • Click the Test Input button and review the results from the Results and Reference Panels.

    • Take note we now have the File Name available as additional metadata and the object shape has changed within the Reference Panel.

  3. Create a new Input

    • Take note the existing "HB_Tutorial_File_Input_Read_Wildcard" Input returns the first file in the directory alphabetically.

    • Click the Copy button in the upper right of the Input tab.

    • Enter "HB_Tutorial_File_Input_Read_By_File_Type" as the new Input Name.

    • Update the File Name to [(.+jpg)] and do not include the brackets. 

    • Click the Save button.

    • Click the Test Input button and review the results, you should notice the _filename and value have changed based on the new File Name expression.

      • Note if you only have the three source image files in this directory and no other files, the input will return the same value.

  4. Create an Output for the File Connection.

    • Navigate to Connections and select "HB_Tutorial_MQTT".

    • Navigate to the Outputs Tab.

    • Click New Output and enter the following and click the Create button.

      • Name: HB_Tutorial_MQTT_Output_File
      • Topic: Tutorial/File
  5. Create a Pipeline to publish the file data via MQTT.
      • Navigate to Pipelines 
      • Click New Pipeline.
      • Enter “HB_Tutorial_File_To_MQTT” for the Name.
      • Select the Build Flow option to create a Pipeline with Sources, Targets, and a Trigger.
      • Click the Next button.
      • Within the Sources & Target view, use the References Panel and set Type to Input.
      • Set Connections to "HB_Tutorial_File".
      • Drag “HB_Tutorial_File_Input_Read_By_File_Type” into the Sources box.
      • Change Type to Output.
      • Set Connections to "HB_Tutorial_MQTT".
      • Drag "HB_Tutorial_MQTT_Output_File" to the Targets box.
      • Click the Next button.
      • Change the interval to 10 seconds.
      • Click the Create button. 
      • Click on the Pipeline "Flow" stage and Enable the Pipeline.  
      • Review the results in the UNS client.
  6. Read the File value from our UNS

    • Navigate to Connections and select "HB_Tutorial_MQTT".

    • Click the Inputs Tab.

    • Click the New Input button and enter the following:

      • Name: HB_Tutorial_MQTT_Input_File
      • Topic: Tutorial/File

    • Click the Create button.

    • Click the Test Input button to review the data

      • Take note, the value attribute is the binary string of the File. We will need to ensure we specify this attribute in our File output.
  7. Create a File output.
    • Navigate to Connections.

    • Select "HB_Tutorial_File".

    • Select the Outputs tab.

    • Click the New Output button, enter the following and click the Create button.

      • Name: HB_Tutorial_File_Output_File

      • File Name: Tutorial_fileoutput.jpg

      • Payload Reference: {{this.value}}

      • Base64 Decoding: On

  8. Create a Pipeline to read the data from MQTT and write out a new file.  Note we will only output one of the three files.
    • Navigate to Pipelines.
    • Click the New Pipelines button.
    • Enter name as “HB_Tutorial_Read_File_From_MQTT”
    • Select the Build Flow option to create a Pipeline with Sources, Targets, and a Trigger.
    • Click the Next button.
    • Within the Sources & Target view, use the References Panel and set Type to Input.
    • Set Connections to "HB_Tutorial_MQTT"
    • Drag your root object "HB_Tutorial_MQTT_Input_File" to the Sources box.
    • Set Type to Output.
    • Set Connections to "HB_Tutorial_File".
    • Drag the object “HB_Tutorial_File_Output_File” to the Targets box.
    • Click the Next button.
    • Change the interval to 10 seconds.
    • Click the Create button. 
    • Click on the Pipeline "Flow" stage and Enable the Pipeline.  
    • Review the results in the file directory.
  9. Review the results
    • Navigate to your File directory to view the file.
    • Alternatively you could create a Connection Input to view the newly created file. 

 

The answer to this tutorial can be found in the beginning of the next tutorial section.   It is found in the Project JSON in the next tutorial.  The Project JSON in each section is cumulative.  It is built based on the previous tutorial sections as well as new components needed for the current section.

Typically, a file use case involves multiple deployments of the Intelligence Hub to securely move Files from one network to another. For the sake of simplicity, we will simulate the consumption of a file from the UNS from a single deployment.

Tutorial Summary

This exercise focused on the basics of working with a File Connection. You should now be more familiar with how to Input and Output raw files using the Intelligence Hub. We covered the basic principles of using regex to build our file names in the File Connection. These same basic principles can be used with CSV and Parquet Connections. This exercise also covered how to move files to other target sources (such as MQTT) and read and decode the files to additional locations.

Next Tutorial