What Does This Article Cover:
Using sub-pipelines is a great way to distribute work to use other processing resources. Sub pipelines are an easy way to add addition computing power to the processing of messages. This article explains how it can be done and how to decide which sub pipeline to use.
Here is the configuration used in this article:
Using a sub-pipeline:
Any pipeline can call another pipeline to off-load its work. By using a Write stage, another pipeline can be chosen to process the remaining work on a pipeline. The sub pipeline is just a write target. The sub pipeline can be selected in the References panel and dragged into the target property. The parent pipeline will not wait for the sub pipeline to complete. It just hands off the message and move onto its next stage.
If you enable the UsingSubPipeline and look at the statistics, you will see that here are not any messages queued. Although the PLCIdx Flow stage generates 4 message every second and SubPipeline1 can only process 1 message per second, none are queue in the UsingSubPipeline pipeline.
The queueing of the messages happens in SubPipeline1. Since there are 3 times more messages than it can process, the Queued in SubPipline1 keeps growing over time. 3 additional sub pipelines need to be defined to keep up with the processing needs of UsingSubPipeline. Once defined, the work needs to be routed to the sub pipelines based on some sort of strategy to balance the workload. 2 strategies are mentioned in the subsequent sections of this article.
Selecting Sub Pipelines Based on Round Robin:
In this pipeline, messages are being distributed to sub pipelines based on a round robin.....E.g. 1, 2, 3, 4, 1, 2,3, 4,1...etc.
Now, if you look at the number Queued in SubPipeline1, it is close to zero since the work is being distributed.
Selecting Sub Pipeline Based On Load:
In this pipeline, messages are being distributed to sub pipelines based on a load. It first checks to see if SubPipeline1 is idle. If it is, it routes it to SubPipeline1. If SubPipeline1 is busy, it checks SubPipeline2. If SubPipeline2 is busy, it checks SubPipeline3. If SubPipeline3 is busy, it sends it to SubPipeline4. This might perform better than round robin is processing times in the sub pipelines are variable.
