2023 – Study guide for Exam DP-203: Data Engineering

Handle Missing Data Microsoft DP-203 Revert Data to a Previous State

Handle Security and Compliance Requirements – Create and Manage Batch Processing and Pipelines

Bill Mettler 12/24/2023 0 Comment

The in‐depth coverage of security‐related concepts is coming in Chapter 8, “Keeping Data Safe and Secure.” The different aspects of security are vast, but here you will get a...

Handle Missing Data Microsoft DP-203 Trigger Batches

Directory Structure – Create and Manage Batch Processing and Pipelines

Bill Mettler 11/21/2023 0 Comment

The following directory structure should not be something new to you at this point. And perhaps you can already visualize how this directory can be used for the incremental...

Microsoft DP-203 Revert Data to a Previous State Trigger Batches

Design and Implement Incremental Data Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 10/19/2023 0 Comment

The design part of incremental loading was covered in Chapter 3, in the section “Design for Incremental Loading.” As shown in Figure 3.3, the serving layer is part of...

Microsoft DP-203 Revert Data to a Previous State Trigger Batches

Sink – Create and Manage Batch Processing and Pipelines

Bill Mettler 09/14/2023 0 Comment

The Settings tab of a sink in a data flow includes an attribute named Batch Size. This setting is valid in the context of columnar storage and caching, which...

Microsoft DP-203

Upsert Data– Create and Manage Batch Processing and Pipelines

Bill Mettler 07/12/2023 0 Comment

FIGURE 6.49 Upsert data—update methods FIGURE 6.50 Upsert data—sink data preview The Clone feature mentioned in step 1 can be very useful. Instead of needing to re‐create and reconfigure...

Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS

Data Streaming– Create and Manage Batch Processing and Pipelines

Bill Mettler 06/26/2023 0 Comment

For managing delays in a data streaming scenario, there is a concept known as a watermark. The details are covered in detail in Chapter 7, but as the context...

Handle Missing Data Microsoft DP-203 Trigger Batches

Hashed Column– Create and Manage Batch Processing and Pipelines

Bill Mettler 05/30/2023 0 Comment

Another interesting approach for implementing upsert logic is to use what is called a row hash. A very popular hashing algorithm is MD5. This algorithm is no longer secure,...

Handle Failed Batch Loads Microsoft DP-203 Trigger Batches

Scale Resources – Create and Manage Batch Processing and Pipelines

Bill Mettler 04/25/2023 0 Comment

Chapter 10 focuses on optimizing the compute and availability of your data workloads running on Azure. Remember that when you provisioned the nodes that will run the batch job,...

Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS

Implement Incremental Data Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 03/26/2023 0 Comment

@getPastTime(1, ‘Day’) FIGURE 6.55 Incremental data loads—Get Metadata activity FIGURE 6.56 Incremental data loads—ForEach activity FIGURE 6.57 Incremental data loads—Execute Pipeline activity FIGURE 6.58 Incremental data loads—pipeline output That...

Handle Missing Data Microsoft DP-203 STORAGE EVENT TRIGGERS

Revert Data to a Previous State – Create and Manage Batch Processing and Pipelines

Bill Mettler 02/25/2023 0 Comment

As mentioned numerous times in this book, batch jobs and pipelines fail. Sometimes a failure has no impact on the data in your data lake, but in some cases...