Historians: Read Value Changes

What Does This Article Cover?

Some solutions require obtaining value changes from a historian, including late-arriving data. This article provides starter solutions for this specific scenario.

What is late-arriving data?

Late-arriving data refers to data changes that arrive at the historian with a timestamp that is earlier than the most recent data change already stored for a given point or tag. Late data may arrive due to the following.

Network latency or outages
Buffering at the source (e.g., a PLC, RTU, or edge device)
Store-and-forward
Time synchronization issues
Editing in the historian or external application writing to the historian

General Intelligence Hub design considerations for obtaining value changes

This design approach should be used when it is necessary to subscribe to value changes, including late-arriving data. Consider that obtaining data by polling or aggregation may result in less data processing and better data pipeline performance.
The resulting data payload might be written to a database, data lake, or data warehouse.
When creating a solution, start with a small number of PI Points and a short time span.
- Avoid long-duration queries.
- Attempt to use a query that executes in a few seconds.
Avoid Pipeline Debug mode and Enabling Tracking Activity if the Intelligence Hub Pipeline is processing more than many dozen value changes.
This solution type is conducive to a narrow table format with a column for values.
- Consider normalizing data types in a narrow table and the respective logic.
- It might be practical to have a column for numeric values and a separate column for non-numeric values.
- Model the desired output schema using an Intelligence Hub Model.
- Asset and PI Point context might be obtained infrequently by a parallel pipeline and stored in a different table in the destination system.
In this scenario, the time obtained by the Intelligence Hub Connection Input is the time that the respective value was written to the historian.

AVEVA PI Data Archive design considerations for obtaining value changes

These considerations pertain to obtaining value changes for PI Data Archive PI Points.

The Intelligence Hub AVEVA PI System Connection Point Changes Input is used to obtain value changes. It returns all adds, updates, inserts, and deletes for all PI Points identified in the query.
A Reference list of PI Points may be used or specific named PI Points. Consider using the Point Browse Connection to obtain the list of Point Names and optimize the Cache Lifetime setting.
The Connection Input creates a subscription via PI Data Pipes. The subscription remains active for ten minutes and the subscription is reactivated each time it is queried.
The Connection uses the query as defined at the time of subscription creation. If PI Points are added or deleted the subscription may need to be updated.
Use the Max Per Read to optimize the pipeline throughput and avoid processing floods of value changes.
The starter solution captures the PI Point's Point ID. The assumption is that the narrow table is linked to a table containing PI Point metadata.

A project file may be downloaded [here].

Navigate to the interactive demonstration [here].

AVEVA PI System Asset Framework design considerations for obtaining value changes

These considerations pertain to obtaining value changes for PI Asset Framework Attributes.

The Intelligence Hub PI Connection Asset Changes Input enables you to capture all additions, updates, inserts, and deletions for every asset attribute, including attributes linked to PI Points and those that are not.
- This method is recommended if you need to monitor changes to Static Values (asset attributes not mapped to PI Points) or require asset context to be part of the payload.
- If only PI Point value changes are needed, you may obtain the PI Points directly and leverage the PI Data Archive approach.
The Connection Input creates a subscription via PI Data Pipes. The subscription remains active for ten minutes, and the subscription is reactivated each time it is queried.
The Connection uses the query as defined at the time of subscription creation. If assets are added or deleted in PI, the subscription may need to be updated.
The starter solution includes the element ID and attribute ID in the payload. This approach assumes the narrow table is associated with destination system tables that store element (asset) and attribute metadata.