You integrated your Azure Synapse Analytics workspace with GitHub in Exercise 3.6. Each time you clicked the Commit button, the configuration changes you made were stored into your selected branch—in this case, the main branch of the GitHub repository. When you then clicked the Publish button, the result was a deployment of those changes to the production workspace. If multiple users or developers are working on the same workspace, changes made by a commit will not be available to all users on that workspace until a publish is performed. This may not have been obvious, you have most likely been working through all these examples as an individual, not as a team. In Exercise 3.6 you configured the GitHub repository so that your changes were applied to the main branch, similar to Figure 6.66.

FIGURE 6.66 Implementing version control for pipeline artifacts

In general, that is not good practice, because you are effectively committing changes and deploying them directly into the production environment. Instead, you need to consider having numerous branches to host your changes that flow through your release process. A release process typically consists of a development, user acceptance testing (UAT), and a production environment. A development environment changes often, and new features or bug fixes are initially created. Once the change is in an acceptable state, the code can be merged with the main branch, which is then deployed to the UAT environment. The environment is much more stable and is often referred to as a preproduction environment. It is the place where your user acceptance and regression testing take place. Once all the changes are approved, the changes can be applied to a branch that is destined for the production environment. CI/CD tools like GitHub Actions or Azure DevOps pipelines are commonly used to perform the release management and deployments through the release management processes.

You now have some experience with GitHub, so Azure DevOps will become the focus. Azure DevOps was introduced in Chapter 1 and briefly touched on in both Chapter 2 and Chapter 3. Azure DevOps provides the same capabilities you find when using GitHub. The choice is mostly a matter of the skill set of the team responsible for release management. Table 6.7 describes the components available in Azure DevOps.

TABLE 6.7 Azure DevOps components

Figure 6.67 illustrates how each of the Azure DevOps components work together to form a release management process that includes CI/CD. The first component is Azure Boards, which you can use to track, log, and assign work to the developers or data engineers on your team. A term commonly used in this context is work item. A work item can be a new feature, a bug fix, a documentation request, or pretty much anything that needs to be done to support your data solution. Once the work item is assigned, the individual performing the work will test this on a nonproduction Azure Synapse Analytics workspace. After the changes are tested locally and published, that development workspace will be configured to write changes to a development code branch.

FIGURE 6.67 Implementing version control for pipeline artifacts, Azure DevOps release flow

The code and/or configuration changes are pushed to Azure Repos, which is where the production code and configuration are stored. When you first create your Azure DevOps project and Azure repo, there exists only a single branch, typically called the main or master branch. It is common and recommended to have more than a single branch so that multiple individuals can work on the same code for different project deliverables in parallel. Once they both finish, a pull request is made, the code is merged, and it flows through the remainder of the CI/CD process. When the code is committed to the Azure repo branch, it automatically triggers an Azure pipeline, which builds the application. If the build requires the compilation of source code into an executable, for example, this is where that happens. If the application has a dependency on, for example, NuGet packages, npm packages, or some other assembly, Azure Artifacts is referenced to acquire them. It is possible to pull packages directly from the source; however, if you want to make sure each build uses the same exact package or binary, then they can be stored in and retrieved from Azure Artifacts.

Test‐driven development (TDD) and basic functional testing should be performed before publishing to both the development and UAT environments. Azure Test Plans is a great tool for managing the status of tests, as well as for signing off on changes that will be deployed into the production environment. Each stage of the release deployment, from development to UAT to production, should require some form of review and approval prior to performing the release to the different environments. If automated testing scenarios are successful, that result can be regarded as an agreement to deploy into the targeted environment. When you have a fully automated process starting from a pull request to change a code, to publish, to development that flows through the other environments, then to production in a matter of minutes or hours, then you have achieved a true CI/CD release process.