Reference Solution: AVEVA PI System to the Cloud

What Does This Article Cover?

Intelligence Hub is often used to obtain data from PI System, curate a use case aligned data payload, and send the data payload to a cloud data lake or data warehouse. The data payload may include real-time data, historical time-series data, and or configuration data from PI Asset Framework. The data from PI System may be combined with data from additional data sources to create a composite data payload. The destination data repository might contain structured or unstructured data and it might be file based or table based. Finally, it might be important to optimize latency or alternatively cloud compute costs. So, there are many design considerations and solution scenarios. This article attempts to capture the key considerations.

Use Case Considerations

The overall design approach is determined by the specific use case. The following considerations might be helpful for defining the use case.

What role or persona will consume the data and for what purpose?
How will the data be consumed by the role or persona?
What specific data lake or data warehouse technology is being used?
How will the data be queried and for what type of analytics?
What is the required shape of the data in the target data store?
Is low latency important and what is the design requirement?
How can cloud storage or compute costs be minimized?

Helpful Questions

Answers to the following questions might help in defining the Intelligence Hub solution.

Is PI Asset Framework being used and are the assets well organized?
How many assets are configured in PI Asset Framework?
How many root level assets are configured in PI Asset Framework?
How many levels of asset hierarchy are configured in PI Asset Framework?
How many PI Point are configured and associated to PI Asset Framework asset attributes?
Generally, how frequently are PI Point values changing?
What is the expected latency and resolution of the data?
Are all values needed or is polling acceptable? At what rate?
Is historical backfill of data required?
Is the design approach wide or narrow tables?
Will metadata tables be created in the data lake?
What network is the PI System located on?
Where will Intelligence Hub be installed?
Will Intelligence Hub have access to write directly to the cloud endpoint?

Important Design Considerations

Consider the following when designing an Intelligence Hub solution related to AVEVA PI System.

The scope may include PI System assets, asset metadata, PI Point and or attribute values, and historical data
The Intelligence Hub PI Agent should be installed on the same network as the PI System.
The Intelligence Hub Server should have at least 16 to 32 GB of RAM and a processor with four to eight cores.
Intelligence Hub Connections are single threaded. Therefore, assume that a solution may utilize many Intelligence Hub Connections to obtain data from the PI System.
The Intelligence Hub PI Connections should be configured with long Request Timeout settings (for example 90 seconds) and Gzip compression.
To optimize throughput consider Intelligence Hub settings including the selections for Store and Forward on Connection Outputs (turn off), Dynamic References and the Breakup Arrays setting on Connection Output (turn off if Dynamic References are not needed), the Ignore Write Results setting on Pipeline Write Stage (turn on), and Pipeline Track Activity (turn off).
Buffer transactions using a strategy that aligns with the solution requirements for latency. It may be beneficial to buffer transactions for a cloud data warehouse, like Snowflake for example.

Summary

Please reference the questions and design considerations listed above when using Intelligence Hub to create a solution that obtains data from AVEVA PI System and sends curated data payloads to a cloud data store.