2022 – Study guide for Exam DP-203: Data Engineering

Handle Missing Data Microsoft DP-203 Trigger Batches

Handle Duplicate Data with a Data Flow – Create and Manage Batch Processing and Pipelines-2

Bill Mettler 12/15/2022 0 Comment

FIGURE 6.47 Handle duplicate data—data flow sink The first change you may have noticed in Exercise 6.9 versus some of the others has to do with the data. In...

Handle Failed Batch Loads Microsoft DP-203 STORAGE EVENT TRIGGERS

Handle Missing Data– Create and Manage Batch Processing and Pipelines

Bill Mettler 10/30/2022 0 Comment

In Exercise 5.5 you worked with a CSV file that contained null values. The action in that exercise to avoid the null values was to remove the record from...

Handle Failed Batch Loads Microsoft DP-203 Revert Data to a Previous State

Handle Duplicate Data with a Data Flow – Create and Manage Batch Processing and Pipelines-1

Bill Mettler 09/08/2022 0 Comment

FIGURE 6.43 Handle duplicate data—data flow source FIGURE 6.44 Handle duplicate data—data flow aggregate group by FIGURE 6.45 Handle duplicate data—data flow aggregate FIGURE 6.46 Handle duplicate data—data flow...

Handle Missing Data Microsoft DP-203 Revert Data to a Previous State Trigger Batches

Create and Schedule an Azure Databricks Workflow Job – Create and Manage Batch Processing and Pipelines

Bill Mettler 07/06/2022 0 Comment

Azure Databricks When you perform data analytics in Azure Databricks, it means you are using an Apache Spark cluster. It is possible to use Apache Spark pools in Azure...

Handle Failed Batch Loads Microsoft DP-203 STORAGE EVENT TRIGGERS

Handle Duplicate Data – Create and Manage Batch Processing and Pipelines

Bill Mettler 04/22/2022 0 Comment

The impact of having multiple rows of identical data in your table and database depends on your requirements. In most cases, however, having a duplication of a row is...

Handle Missing Data Microsoft DP-203 Revert Data to a Previous State

Streaming – Create and Manage Batch Processing and Pipelines

Bill Mettler 03/30/2022 0 Comment

In addition to handling duplications in data that is being ingested and stored into your data lake for scheduled processing, you may need to handle duplications while streaming data....

Handle Failed Batch Loads Handle Missing Data Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS Trigger Batches

Handle Failed Batch Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 02/20/2022 0 Comment

Monitoring and troubleshooting pipelines that contain batch loads is covered in more detail in Chapter 9 and Chapter 10. An option for handling a failed batch load is to...

Microsoft DP-203 STORAGE EVENT TRIGGERS

Validate Batch Loads by Using a Validation Activity – Create and Manage Batch Processing and Pipelines

Bill Mettler 01/30/2022 0 Comment

FIGURE 6.63 Validate batch loads with the Validation activity The dataset from Exercise 6.9 that you chose as the location to perform the validation on is the same location...