Revert Data to a Previous State – Study guide for Exam DP-203: Data Engineering

Handle Failed Batch Loads Microsoft DP-203 Revert Data to a Previous State

Use PolyBase to Load Data to a SQL Pool – Create and Manage Batch Processing and Pipelines

Bill Mettler 03/16/2024 0 Comment

PolyBase is a very powerful technology that works behind the scenes to enable the execution of SQL queries on data files, where the data files are, for example, in...

Handle Missing Data Microsoft DP-203 Revert Data to a Previous State

Handle Security and Compliance Requirements – Create and Manage Batch Processing and Pipelines

Bill Mettler 12/24/2023 0 Comment

The in‐depth coverage of security‐related concepts is coming in Chapter 8, “Keeping Data Safe and Secure.” The different aspects of security are vast, but here you will get a...

Microsoft DP-203 Revert Data to a Previous State Trigger Batches

Design and Implement Incremental Data Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 10/19/2023 0 Comment

The design part of incremental loading was covered in Chapter 3, in the section “Design for Incremental Loading.” As shown in Figure 3.3, the serving layer is part of...

Microsoft DP-203 Revert Data to a Previous State Trigger Batches

Sink – Create and Manage Batch Processing and Pipelines

Bill Mettler 09/14/2023 0 Comment

The Settings tab of a sink in a data flow includes an attribute named Batch Size. This setting is valid in the context of columnar storage and caching, which...

Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS

Data Streaming– Create and Manage Batch Processing and Pipelines

Bill Mettler 06/26/2023 0 Comment

For managing delays in a data streaming scenario, there is a concept known as a watermark. The details are covered in detail in Chapter 7, but as the context...

Microsoft DP-203 Revert Data to a Previous State STORAGE EVENT TRIGGERS

Implement Incremental Data Loads – Create and Manage Batch Processing and Pipelines

Bill Mettler 03/26/2023 0 Comment

@getPastTime(1, ‘Day’) FIGURE 6.55 Incremental data loads—Get Metadata activity FIGURE 6.56 Incremental data loads—ForEach activity FIGURE 6.57 Incremental data loads—Execute Pipeline activity FIGURE 6.58 Incremental data loads—pipeline output That...

Microsoft DP-203 Revert Data to a Previous State

Validate Batch Loads by Using a Lookup Activity – Create and Manage Batch Processing and Pipelines

Bill Mettler 01/22/2023 0 Comment

FIGURE 6.65 Validate batch loads with the Lookup activity When the pipeline is completed, check the output of the Lookup activity. You should see something like the following, which...

Handle Failed Batch Loads Microsoft DP-203 Revert Data to a Previous State

Handle Duplicate Data with a Data Flow – Create and Manage Batch Processing and Pipelines-1

Bill Mettler 09/08/2022 0 Comment

FIGURE 6.43 Handle duplicate data—data flow source FIGURE 6.44 Handle duplicate data—data flow aggregate group by FIGURE 6.45 Handle duplicate data—data flow aggregate FIGURE 6.46 Handle duplicate data—data flow...

Handle Missing Data Microsoft DP-203 Revert Data to a Previous State Trigger Batches

Create and Schedule an Azure Databricks Workflow Job – Create and Manage Batch Processing and Pipelines

Bill Mettler 07/06/2022 0 Comment

Azure Databricks When you perform data analytics in Azure Databricks, it means you are using an Apache Spark cluster. It is possible to use Apache Spark pools in Azure...

Handle Missing Data Microsoft DP-203 Revert Data to a Previous State

Streaming – Create and Manage Batch Processing and Pipelines

Bill Mettler 03/30/2022 0 Comment

In addition to handling duplications in data that is being ingested and stored into your data lake for scheduled processing, you may need to handle duplications while streaming data....