Skip to content
  • There are no suggestions because the search field is empty.

Tutorial: File Connections for Remote Training

This step-by-step interactive tutorial provides an introduction to the Intelligence Hub File Connection.

 

This article provides an alternative to the Tutorial: File Connectors article.  If the user does not have access to the file system where Intelligence Hub is installed, this article assumes that the *.jpg files will be accessible from the ../appData folder.  The files should be placed in this folder by the system administrator. It also assumes that files can be written to the ../appData folder.   

What Does This Tutorial Cover?

This tutorial will walk you through the basics of working with File Connections.  A Connection represents a pathway to either a data source or a target, serving as the starting point for all use cases.

  • Tutorial Prerequisites
  • Tutorial Preparation
  • Tutorial Instructions
  • Tutorial Summary

Tutorial Prerequisites

  • You have completed the initial Installation of the Intelligence Hub.
  • It is recommended to read the Getting Started Series before completing this tutorial exercise.
  • It is recommended to complete "Tutorial: Connections".

Tutorial Preparation

  1. Enable Intelligence Hub MQTT broker

    • In the left-hand navigation panel, navigate to Manage, and click Settings.
    • Under the MQTT Broker section enable the broker, if ports 1885 and 1886 are being utilized on your Intelligence Hub server, update to ports of your choosing, otherwise accept the defaults and click the Save button.

  2. Import the required Connections

    • In the left-hand navigation panel, navigate to Manage, and click Project.
    • Within the Import screen, ensure Full Project is off (otherwise your existing project will be overwritten).
    • Copy the JSON provided here. Project JSON
    • Change the Import Type to JSON and paste the provided JSON into the Project box and click the Import button.

  3. Update the imported Connections as required

    • Navigate to Configure and click Connections, click "HB_Tutorial_MQTT" and Update the MQTT settings as required based on the prior preparation step #1.
    • Navigate to Configure and click Connections, click "HB_Tutorial_File".  Make sure the File Directory is set to "../appData".   This location is the application directory folder that in under the HighByte Intelligence Hub installation folder.  This tutorial will not utilize the processed or error directory since users do not have access to interact with these folders.
  4. The training environments has been setup to use these 3 files.  These files have already been saved in the ../appData folder on your remote training environment.
    • cnc_1001.jpg

    • cnc_1002.jpg

    • cnc_1003.jpg



  5. Setup UNS Client
    • In the left-hand navigation panel, navigate to Tools and right click UNS Client and select Open Link in New Tab.
    • Enter login information.

    • For Connection select "HB_Tutorial_MQTT".

    • For Subscribed Topics enter "Tutorial/#".

    • Click Add.

    • Click the "x" to remove topic "#".

    • Click the Connect button.

    • Confirm the UNS Client says "Connected to HB_Tutorial_MQTT".

    • Return to the previous web browser tab.

Instructions

  1. Login to your Intelligence Hub environment.

  2. Review the source data.

    • Navigate to Connections and select "HB_Tutorial_File".

    • Navigate to the Inputs tab.

    • Select "HB_Tutorial_File_Input_Read_Wildcard".

      • Take note of the file name structure, this is the default new input for File/CSV connections.  Consider it a wildcard that will read the whole directory without any additional logic.

    • Within the right-hand References Panel, set type to Input.

    • Set Connections to "HB_Tutorial_File".

    • Find the Input you opened and take note of the object’s shape.

    • Now enable “Include Metadata” within the Input details

    • Click the Test Input button and review the results from the Results and Reference Panels.

    • Take note we now have the File Name available as additional metadata and the object shape has changed within the Reference Panel.

  3. Create a new Input

    • Take note the existing "HB_Tutorial_File_Input_Read_Wildcard" Input returns the first file in the directory alphabetically.

    • Click the Copy button in the upper right of the Input tab.

    • Enter "HB_Tutorial_File_Input_Read_By_File_Type" as the new Input Name.

    • Update the File Name to [(.+jpg)] and do not include the brackets. 

    • Click the Save button.

    • Click the Test Input button and review the results, you should notice the _filename and value have changed based on the new File Name expression.

      • Note if you only have the three source image files in this directory and no other files, the input will return the same value.

  4. Create an Output for the MQTT Connection.

    • Navigate to Connections and select "HB_Tutorial_MQTT".

    • Navigate to the Outputs Tab.

    • Click New Output and enter the following and click the Create button.

      • Name: HB_Tutorial_MQTT_Output_File
      • Topic: Tutorial/File
  5. Create a Pipeline to publish the file data via MQTT.
      • Navigate to Pipelines 
      • Click New Pipeline.
      • Enter “HB_Tutorial_File_To_MQTT” for the Name.
      • Select the Build Flow option to create a Pipeline with Sources, Targets, and a Trigger.
      • Click the Next button.
      • Within the Sources & Target view, use the References Panel and set Type to Input.
      • Set Connections to "HB_Tutorial_File".
      • Drag “HB_Tutorial_File_Input_Read_By_File_Type” into the Sources box.
      • Change Type to Output.
      • Set Connections to "HB_Tutorial_MQTT".
      • Drag "HB_Tutorial_MQTT_Output_File" to the Targets box.
      • Click the Next button.
      • Change the interval to 10 seconds.
      • Click the Create button. 
      • Click on the Pipeline "Flow" stage and Enable the Pipeline.  
      • Click on the Save button.
      • Review the results in the UNS client.
  6. Read the File value from our UNS

    • Navigate to Connections and select "HB_Tutorial_MQTT".

    • Click the Inputs Tab.

    • Click the New Input button and enter the following:

      • Name: HB_Tutorial_MQTT_Input_File
      • Topic: Tutorial/File

    • Click the Create button.

    • Click the Test Input button to review the data

      • Take note, the value attribute is the binary string of the File. We will need to ensure we specify this attribute in our File output.
  7. Create a File output.
    • Navigate to Connections.

    • Select "HB_Tutorial_File".

    • Select the Outputs tab.

    • Click the New Output button, enter the following and click the Create button.
      • Name: HB_Tutorial_File_Output_File
      • File Name: Tutorial_fileoutput.jpg
      • Payload Reference:  {{this.value}}
      • Base64 Decoding: On
  8. Create a Pipeline to read the data from MQTT and write out a new file.  Note: only one of the three files will be outputted.
    • Navigate to Pipelines.
    • Click the New Pipelines button.
    • Enter name as “HB_Tutorial_Read_File_From_MQTT”
    • Select the Build Flow option to create a Pipeline with Sources, Targets, and a Trigger.
    • Click the Next button.
    • Within the Sources & Target view, use the References Panel and set Type to Input.
    • Set Connections to "HB_Tutorial_MQTT"
    • Drag your root object "HB_Tutorial_MQTT_Input_File" to the Sources box.
    • Set Type to Output.
    • Set Connections to "HB_Tutorial_File".
    • Drag the object “HB_Tutorial_File_Output_File” to the Targets box.
    • Click the Next button.
    • Change the interval to 10 seconds.
    • Click the Create button. 
    • Click on the Pipeline "Flow" stage and Enable the Pipeline.  
    • Click on the Save button.
  9. Review the results 
    • Navigate to the HB_Tutorial_File connector.
    • Select Inputs
    • Open HB_Tutorial_File_Input_Read_By_File_Type input.
    • Click on Copy button.
    • Specify name as "HB_Tutorial_File_Input_Read_Output" and press Copy.
    • Set File Name: Tutorial_fileoutput.jpg
    • Click on Save button.
    • Click on Test Input button.
    • Results should be populated with output from the HB_Tutorial_Read_File_From_MQTT pipeline defined in step #8.
Here is the completed Project JSON from this tutorial section.

 

Typically, a file use case involves multiple deployments of the Intelligence Hub to securely move Files from one network to another. For the sake of simplicity, we will simulate the consumption of a file from the UNS from a single deployment.

Tutorial Summary

This exercise focused on the basics of working with a File Connection. You should now be more familiar with how to Input and Output raw files using the Intelligence Hub. We covered the basic principles of using regex to build our file names in the File Connection. These same basic principles can be used with CSV and Parquet Connections. This exercise also covered how to move files to other target sources (such as MQTT) and read and decode the files to additional locations.

Next Tutorial

  1. Tutorial: Models and Instances