STORAGE EVENT TRIGGERS – Study guide for Exam DP-203: Data Engineering

Handle Failed Batch Loads Microsoft DP-203 STORAGE EVENT TRIGGERS

Implement Azure Synapse Link and Query the Replicated Data – Create and Manage Batch Processing and Pipelines

Bill Mettler 05/30/2024 0 Comment

Chapter 2 briefly introduced data processing types such as OLTP, OLAP, and HTAP, and Chapter 10 covered them in a bit more detail. The overlying principle with these data...

Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS

Data Streaming– Create and Manage Batch Processing and Pipelines

Bill Mettler 06/26/2023 0 Comment

For managing delays in a data streaming scenario, there is a concept known as a watermark. The details are covered in detail in Chapter 7, but as the context...

Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS

Implement Incremental Data Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 03/26/2023 0 Comment

@getPastTime(1, ‘Day’) FIGURE 6.55 Incremental data loads—Get Metadata activity FIGURE 6.56 Incremental data loads—ForEach activity FIGURE 6.57 Incremental data loads—Execute Pipeline activity FIGURE 6.58 Incremental data loads—pipeline output That...

Handle Missing Data Microsoft DP-203 STORAGE EVENT TRIGGERS

Revert Data to a Previous State – Create and Manage Batch Processing and Pipelines

Bill Mettler 02/25/2023 0 Comment

As mentioned numerous times in this book, batch jobs and pipelines fail. Sometimes a failure has no impact on the data in your data lake, but in some cases...

Handle Failed Batch Loads Microsoft DP-203 STORAGE EVENT TRIGGERS

Handle Missing Data– Create and Manage Batch Processing and Pipelines

Bill Mettler 10/30/2022 0 Comment

In Exercise 5.5 you worked with a CSV file that contained null values. The action in that exercise to avoid the null values was to remove the record from...

Handle Failed Batch Loads Microsoft DP-203 STORAGE EVENT TRIGGERS

Handle Duplicate Data – Create and Manage Batch Processing and Pipelines

Bill Mettler 04/22/2022 0 Comment

The impact of having multiple rows of identical data in your table and database depends on your requirements. In most cases, however, having a duplication of a row is...

Handle Failed Batch Loads Handle Missing Data Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS Trigger Batches

Handle Failed Batch Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 02/20/2022 0 Comment

Monitoring and troubleshooting pipelines that contain batch loads is covered in more detail in Chapter 9 and Chapter 10. An option for handling a failed batch load is to...

Microsoft DP-203 STORAGE EVENT TRIGGERS

Validate Batch Loads by Using a Validation Activity – Create and Manage Batch Processing and Pipelines

Bill Mettler 01/30/2022 0 Comment

FIGURE 6.63 Validate batch loads with the Validation activity The dataset from Exercise 6.9 that you chose as the location to perform the validation on is the same location...

Microsoft DP-203 STORAGE EVENT TRIGGERS Trigger Batches

Design and Develop Slowly Changing Dimensions – Create and Manage Batch Processing and Pipelines

Bill Mettler 12/29/2021 0 Comment

The design of an SCD table was discussed in detail in Chapter 3, in the section “Design the Serving Layer.” The notion of a star schema was also introduced...

Handle Missing Data Microsoft DP-203 STORAGE EVENT TRIGGERS

STORAGE EVENT TRIGGERS – Create and Manage Batch Processing and Pipelines

Bill Mettler 09/26/2021 0 Comment

In the previous two trigger type scenarios, the data scheduled to be processed is expected to be in a specified location before the pipeline is triggered. An alternative to...