Sharing state databases between Intelligence Hubs
What Does This Article Cover:
The article shows how a pipeline can be used to share state values between Intelligence Hubs. The pipeline will run on a fixed interval and will use the Intelligence Hub File connection to copy the SQLite State database(s) from a running Intelligence Hub to another file location. The other file location could be a network drive, or it could be the appData folder of a secondary Intelligence Hub in a redundancy configuration.
It is recommended that the appData folder for Intelligence Hub to be store on local storage or SAN (low latency) file storage. The recommended storage for appData is similar to storage requirement needed for a database transaction log. Due to latency, network share drives are not recommended to store Intelligent Hub appData.
Here is the configuration used in this article:
{
"productInfo": {
"company": "HighByte",
"product": "IntelligenceHub",
"version": "4.0.1",
"build": "2024.11.5.4",
"stage": "Release"
},
"project": {
"version": 9,
"connections": [
{
"name": "FileSyncs",
"uri": "file://FileSyncs",
"tags": [
"FilesSync"
],
"writes": {
"flattenModeledValues": false
},
"subscriptions": {},
"storeForward": {
"enabled": false,
"maxEntries": 100,
"waitOnFailureInterval": {
"duration": 1,
"units": "Seconds"
}
},
"settings": {
"communicationProtocol": {
"type": "FILE_PROTOCOL",
"options": {}
},
"directory": "../appData"
}
}
],
"inputs": [
{
"name": "cacheDB",
"connection": "FileSyncs",
"type": "file",
"qualifier": {
"fileName": "intelligencehub-cache.db"
},
"cacheLifetime": {
"enabled": false
},
"template": {
"type": "Off"
}
},
{
"name": "stateDB",
"connection": "FileSyncs",
"type": "file",
"qualifier": {
"fileName": "intelligencehub-state.db"
},
"cacheLifetime": {
"enabled": false
},
"template": {
"type": "Off"
}
}
],
"outputs": [
{
"name": "cacheDB",
"connection": "FileSyncs",
"type": "file",
"qualifier": {
"breakupArrays": false,
"relativePath": "../syncAppData",
"fileName": "intelligencehub-cache.db"
}
},
{
"name": "stateDB",
"connection": "FileSyncs",
"type": "file",
"qualifier": {
"breakupArrays": false,
"relativePath": "../syncAppData",
"fileName": "intelligencehub-state.db"
}
}
],
"modeling": {
"models": [],
"instances": []
},
"conditions": [],
"functions": [],
"tags": [
{
"name": "FilesSync"
}
],
"pipelines": [
{
"name": "cacheDBSync",
"uri": "pipeline",
"tags": [
"FilesSync"
],
"settings": {
"inputStages": [
"Write"
],
"trackActivity": false,
"triggers": [
{
"name": "FlowTrigger",
"config": {
"type": ".TriggerFlow",
"enabled": true,
"flowEvaluation": {
"type": "Polled",
"interval": {
"duration": 5,
"units": "Seconds"
},
"mode": "Always",
"expression": "",
"delay": {
"duration": 0,
"units": "Seconds"
}
},
"publishMode": "All",
"template": {
"type": "Off"
},
"inReferences": [
""
]
},
"display": {
"position": {
"x": -454,
"y": -73
}
}
}
],
"stages": [
{
"name": "Write",
"outputs": [],
"config": {
"type": ".WriteConfig",
"failureOutputs": [],
"ignoreResult": false,
"references": [
""
]
},
"display": {
"position": {
"x": 212,
"y": -75
}
}
}
]
}
},
{
"name": "stateDBSync",
"uri": "pipeline",
"tags": [
"FilesSync"
],
"settings": {
"inputStages": [
"Write"
],
"trackActivity": false,
"triggers": [
{
"name": "FlowTrigger",
"config": {
"type": ".TriggerFlow",
"enabled": true,
"flowEvaluation": {
"type": "Polled",
"interval": {
"duration": 5,
"units": "Seconds"
},
"mode": "Always",
"expression": "",
"delay": {
"duration": 0,
"units": "Seconds"
}
},
"inReferences": [
""
],
"publishMode": "OnlyChanges",
"template": {
"type": "Off"
}
},
"display": {
"position": {
"x": -454,
"y": -73
}
}
}
],
"stages": [
{
"name": "Write",
"outputs": [],
"config": {
"type": ".WriteConfig",
"failureOutputs": [],
"references": [
""
],
"ignoreResult": false
},
"display": {
"position": {
"x": 212,
"y": -75
}
}
}
]
}
}
],
"namespace": []
},
"network": {
"groups": [],
"hubs": []
}
}
Example Preparation
-
This example assumes that a virtual folder has been setup at the same level as the Intelligence Hub appData folder.
-
On the Primary instance, the syncAppData folder in the /usr/local/highbyte folder points to appData folder Secondary instance.
-
On the Secondary instance, the syncAppData folder in the /usr/local/highbyte folder points appData folder on the Primary instance. With this setup, the Secondary should update the Primary when the Primary comes back online. 
-
- In the configuration, there is only one connector named FileSyncs. The file directory is the ../appData folder. This is the folder that contains the SQLite state databases.
-
The inputs point directly to the SQLite databases. This input is pointing to the Pipeline State database (intelligencehub-state.db).
- The outputs point to the SQLite databases in the ../syncAppData folder. This output is pointing to the Pipeline State database in the virtual folder (intelligencehub-state.db).
-
The last component of this solution is the pipeline. The stateDBSync pipeline uses a FlowTrigger to poll the local state database and copy it to the state database in the virtual folder.