What Does This Article Cover:
This article will cover how to read files from a folder one at a time in ascending order based on the file update time. After each file is read, the last update time is recorded as a state in the pipeline. This state is used as a filter the next time the pipeline is run.
Here is the configuration used in this article:
File Inputs:
In version 4.1, we added the capability to browse files in a folder. This capability provides the ability to only search for files based on their creation and update time. The Time Filter Type property is used to filter the browse either by Creation or Update.
In this example, it will use an input parameter that will browse files that are greater than the LastUpdateTime. This pipeline will call this input get an array of files to select from. From this array, the pipeline filters the array to only find files and then sorts the array ascending based on updateTime.
After finding the file that need to be read, the GetFile input is used to read the file. The input uses an input parameter named FileName for the FileName. This will read the file set it the input parameter.
Pipeline:
With these input in place, the pipeline will read the files in sequence based on updateTime. To read from the Browser input, the pipeline will first get the LastUpdateTime from the pipeline state and store it into metadata. The Browser is read using the LastUpdateTime input parameter and the result will return an array of Files and Directories whose latest update time is greater than the LastUpdateTime.
From the array, a transform is used to first filter the array based on type = File. Then, the array is sorted in ascending order based on updateTime. By sorting the array by updateTime will ensure that the files will be read by updateTime.
This pipeline is setup to read only one file at a time. Using a breakup stage, it will break up the sorted array. Once broken up, a switch stage is used to only select the first file (e.g. breakupIndex = 0). The file is then Read using the name as the value for the FileName parameter into the GetFiles input. Finally, the updateTime of the file is recorded into the LastUpdateTime state to provide a bookmark on where to start the next time the pipeline is run.
