Study guide for Exam DP-203: Data Engineering

Microsoft DP-203

Upsert Data– Create and Manage Batch Processing and Pipelines

Bill Mettler 07/12/2023 0 Comment

FIGURE 6.49 Upsert data—update methods FIGURE 6.50 Upsert data—sink data preview The Clone feature mentioned in step 1 can be very useful. Instead of needing to re‐create and reconfigure...

Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS

Data Streaming– Create and Manage Batch Processing and Pipelines

Bill Mettler 06/26/2023 0 Comment

For managing delays in a data streaming scenario, there is a concept known as a watermark. The details are covered in detail in Chapter 7, but as the context...

Handle Missing Data Microsoft DP-203 Trigger Batches

Hashed Column– Create and Manage Batch Processing and Pipelines

Bill Mettler 05/30/2023 0 Comment

Another interesting approach for implementing upsert logic is to use what is called a row hash. A very popular hashing algorithm is MD5. This algorithm is no longer secure,...

Handle Failed Batch Loads Microsoft DP-203 Trigger Batches

Scale Resources – Create and Manage Batch Processing and Pipelines

Bill Mettler 04/25/2023 0 Comment

Chapter 10 focuses on optimizing the compute and availability of your data workloads running on Azure. Remember that when you provisioned the nodes that will run the batch job,...

Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS

Implement Incremental Data Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 03/26/2023 0 Comment

@getPastTime(1, ‘Day’) FIGURE 6.55 Incremental data loads—Get Metadata activity FIGURE 6.56 Incremental data loads—ForEach activity FIGURE 6.57 Incremental data loads—Execute Pipeline activity FIGURE 6.58 Incremental data loads—pipeline output That...

Handle Missing Data Microsoft DP-203 STORAGE EVENT TRIGGERS

Revert Data to a Previous State – Create and Manage Batch Processing and Pipelines

Bill Mettler 02/25/2023 0 Comment

As mentioned numerous times in this book, batch jobs and pipelines fail. Sometimes a failure has no impact on the data in your data lake, but in some cases...

Microsoft DP-203 Revert Data to a Previous State

Validate Batch Loads by Using a Lookup Activity – Create and Manage Batch Processing and Pipelines

Bill Mettler 01/22/2023 0 Comment

FIGURE 6.65 Validate batch loads with the Lookup activity When the pipeline is completed, check the output of the Lookup activity. You should see something like the following, which...

Handle Missing Data Microsoft DP-203 Trigger Batches

Handle Duplicate Data with a Data Flow – Create and Manage Batch Processing and Pipelines-2

Bill Mettler 12/15/2022 0 Comment

FIGURE 6.47 Handle duplicate data—data flow sink The first change you may have noticed in Exercise 6.9 versus some of the others has to do with the data. In...

Handle Failed Batch Loads Microsoft DP-203 STORAGE EVENT TRIGGERS

Handle Missing Data– Create and Manage Batch Processing and Pipelines

Bill Mettler 10/30/2022 0 Comment

In Exercise 5.5 you worked with a CSV file that contained null values. The action in that exercise to avoid the null values was to remove the record from...

Handle Failed Batch Loads Microsoft DP-203 Revert Data to a Previous State

Handle Duplicate Data with a Data Flow – Create and Manage Batch Processing and Pipelines-1

Bill Mettler 09/08/2022 0 Comment

FIGURE 6.43 Handle duplicate data—data flow source FIGURE 6.44 Handle duplicate data—data flow aggregate group by FIGURE 6.45 Handle duplicate data—data flow aggregate FIGURE 6.46 Handle duplicate data—data flow...