Create a Scheduled Trigger – Create and Manage Batch Processing and Pipelines-2

The capability to create dependencies to other triggers exists within the Advanced grouping. When you expand that group, you will see the option to add trigger dependencies. To create a dependency between the DailyBrainwaveIngestion trigger and other tumbling window triggers, such as DailyElectLoad and DailyFreqLoad, you select them from the Trigger drop‐down text box. You will see only tumbling triggers in the list box; the other types of triggers are not supported. The two triggers illustrated in Figure 6.37 are DailyElectLoad and DailyFreqLoad. Since the initial load of the brainjammer brain wave data happens via the DailyBrainwaveIngestion trigger, it needs to happen before jobs that will pull electrode and frequency data from that Azure SQL pool table.

Next to the Trigger option are two additional options, Offset and Window Size. In general terms, an offset is the positive or negative value difference between two points. In this case, consider that the start time for this tumbling window trigger is 00:00, and the offset is zero. This means that these dependent triggers will fire after the upstream pipeline run completes. Had the value been 30 minutes, then the downstream trigger would be fired 30 minutes after completion of this upstream dependent job. If you want to increase or decrease the time window in which this dependent job can run, place that value in the Window Size text box. Consider that the DailyBrainwaveIngestion trigger is scheduled to run every hour if you need to reduce the time window for the DailyElectLoad trigger to run due to further dependencies, then you can configure that restriction. As mentioned previously, the management of interdependencies of jobs is very complicated. Making changes to them is very risky and, in many cases, it is not possible to test completely the impact of a change. Therefore, if you experience some issue, for example, with the DailyElectLoad trigger because another process locks the table at 45 after the hour, you can change the Window Size down to avoid that locking at 45 after the hour. Especially if you are certain the trigger completes within that timeframe, it is better to do this than to try to reschedule jobs. Rescheduling jobs can cause a domino effect, and you end up potentially changing the behavior of a large number of jobs, which can have a major negative impact.

Another point to make about a tumbling window trigger is that you can have a maximum of five dependencies. That means one tumbling window trigger can have five tumbling window triggers in the drop‐down list boxes in the Advanced section. The amount of time a dependent trigger will wait for an upstream job to finish is 7 days. After 7 days, the downstream dependencies will time out and have a status of Failed. Finally, a tumbling window trigger has a one‐to‐one relationship with a pipeline. A tumbling window trigger cannot trigger more than a single pipeline.

Bill Mettler

Learn More

Leave a Reply

Your email address will not be published. Required fields are marked *