Trigger Batches – Study guide for Exam DP-203: Data Engineering

Microsoft DP-203 Trigger Batches

Enable Copy Activity Data Consistency Verification – Create and Manage Batch Processing and Pipelines

Bill Mettler 02/26/2024 0 Comment

The Copy Data activity is a very powerful and convenient way to ingest data to your data lake. Exercise 3.13 was your first exposure to the Copy Data activity....

Handle Missing Data Microsoft DP-203 Trigger Batches

Directory Structure – Create and Manage Batch Processing and Pipelines

Bill Mettler 11/21/2023 0 Comment

The following directory structure should not be something new to you at this point. And perhaps you can already visualize how this directory can be used for the incremental...

Microsoft DP-203 Revert Data to a Previous State Trigger Batches

Design and Implement Incremental Data Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 10/19/2023 0 Comment

The design part of incremental loading was covered in Chapter 3, in the section “Design for Incremental Loading.” As shown in Figure 3.3, the serving layer is part of...

Microsoft DP-203 Revert Data to a Previous State Trigger Batches

Sink – Create and Manage Batch Processing and Pipelines

Bill Mettler 09/14/2023 0 Comment

The Settings tab of a sink in a data flow includes an attribute named Batch Size. This setting is valid in the context of columnar storage and caching, which...

Handle Missing Data Microsoft DP-203 Trigger Batches

Hashed Column– Create and Manage Batch Processing and Pipelines

Bill Mettler 05/30/2023 0 Comment

Another interesting approach for implementing upsert logic is to use what is called a row hash. A very popular hashing algorithm is MD5. This algorithm is no longer secure,...

Handle Failed Batch Loads Microsoft DP-203 Trigger Batches

Scale Resources – Create and Manage Batch Processing and Pipelines

Bill Mettler 04/25/2023 0 Comment

Chapter 10 focuses on optimizing the compute and availability of your data workloads running on Azure. Remember that when you provisioned the nodes that will run the batch job,...

Handle Missing Data Microsoft DP-203 Trigger Batches

Handle Duplicate Data with a Data Flow – Create and Manage Batch Processing and Pipelines-2

Bill Mettler 12/15/2022 0 Comment

FIGURE 6.47 Handle duplicate data—data flow sink The first change you may have noticed in Exercise 6.9 versus some of the others has to do with the data. In...

Handle Missing Data Microsoft DP-203 Revert Data to a Previous State Trigger Batches

Create and Schedule an Azure Databricks Workflow Job – Create and Manage Batch Processing and Pipelines

Bill Mettler 07/06/2022 0 Comment

Azure Databricks When you perform data analytics in Azure Databricks, it means you are using an Apache Spark cluster. It is possible to use Apache Spark pools in Azure...

Handle Failed Batch Loads Handle Missing Data Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS Trigger Batches

Handle Failed Batch Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 02/20/2022 0 Comment

Monitoring and troubleshooting pipelines that contain batch loads is covered in more detail in Chapter 9 and Chapter 10. An option for handling a failed batch load is to...

Microsoft DP-203 STORAGE EVENT TRIGGERS Trigger Batches

Design and Develop Slowly Changing Dimensions – Create and Manage Batch Processing and Pipelines

Bill Mettler 12/29/2021 0 Comment

The design of an SCD table was discussed in detail in Chapter 3, in the section “Design the Serving Layer.” The notion of a star schema was also introduced...