In Azure DevOps, a pipeline is a workflow that defines how your code is built, tested, and deployed. Pipelines automate the process of continuous integration (CI) and continuous delivery (CD) to help ensure high-quality software releases. Here’s an in-depth look at pipelines in Azure DevOps:

Key Components of Azure DevOps Pipelines

1. Pipelines:
   • A pipeline is a series of stages, jobs, and steps that define the process for building, testing, and deploying code.
2. Stages:
   • Stages are high-level divisions of the pipeline, often used to separate the process into phases such as build, test, and deploy.
   • Example: Build, Test, Deploy.
3. Jobs:
   • Jobs are collections of steps that run sequentially on an agent. Each job can run on a different agent or environment.
   • Example: A job to compile code, a job to run tests, a job to deploy code.
4. Steps:
   • Steps are individual tasks or scripts that are executed as part of a job. They can include running a script, invoking a build tool, deploying an application, etc.
   • Example: Running a command line script, calling a REST API, using built-in Azure DevOps tasks.
5. Agents:
   • Agents are the workers that run your pipeline jobs. They can be Microsoft-hosted or self-hosted.

Overview

• A trigger tells a Pipeline to run.

• A pipeline is made up of one or more stages. A pipeline can deploy to one or more environments.

• A stage is a way of organizing jobs in a pipeline and each stage can have one or more jobs.

• Each job runs on one agent. A job can also be agentless.

• Each agent runs a job that contains one or more steps.

• A step can be a task or script and is the smallest building block of a pipeline.

• A task is a pre-packaged script that performs an action, such as invoking a REST API or publishing a build artifact.

• An artifact is a collection of files or packages published by a run.

Some Terminology

Agent

When your build or deployment runs, the system begins one or more jobs. An agent is computing infrastructure with installed agent software that runs one job at a time. For example, your job could run on a Microsoft-hosted Ubuntu agent.

Approvals

Approvals define a set of validations required before a deployment runs. Manual approval is a common check performed to control deployments to production environments. When checks are configured on an environment, pipelines will stop before starting a stage that deploys to the environment until all the checks are completed successfully.

Artifact

An artifact is a collection of files or packages published by a run. Artifacts are made available to subsequent tasks, such as distribution or deployment. For more information, see Artifacts in Azure Pipelines.

Continuous delivery

Continuous delivery (CD) is a process by which code is built, tested, and deployed to one or more test and production stages. Deploying and testing in multiple stages helps drive quality. Continuous integration systems produce deployable artifacts, which include infrastructure and apps. Automated release pipelines consume these artifacts to release new versions and fixes to existing systems. Monitoring and alerting systems run constantly to drive visibility into the entire CD process. This process ensures that errors are caught often and early.

Continuous integration

Continuous integration (CI) is the practice used by development teams to simplify the testing and building of code. CI helps to catch bugs or problems early in the development cycle, which makes them easier and faster to fix. Automated tests and builds are run as part of the CI process. The process can run on a set schedule, whenever code is pushed, or both. Items known as artifacts are produced from CI systems. They’re used by the continuous delivery release pipelines to drive automatic deployments.

Deployment

Classic pipelines – For Classic pipelines, a deployment is the action of running the tasks for one stage, which can include running automated tests, deploying build artifacts, and any other actions are specified for that stage. YAML pipelines – For YAML pipelines, a deployment typically refers to a deployment job. A deployment job is a collection of steps that are run sequentially against an environment. You can use strategies like run once, rolling, and canary for deployment jobs

Environment

An environment is a collection of resources, where you deploy your application. It can contain one or more virtual machines, containers, web apps, or any service that’s used to host the application being developed. A pipeline might deploy the app to one or more environments after build is completed and tests are run.

Library

A library is a collection of build and release assets for an Azure DevOps project. Assets defined in a library can be used in multiple build and release pipelines of the project. The Library tab can be accessed directly in Azure Pipelines.

Task Group

A task group allows you to encapsulate a sequence of tasks, already defined in a build or a release pipeline, into a single reusable task that can be added to a build or release pipeline, just like any other task. You can choose to extract the parameters from the encapsulated tasks as configuration variables, and abstract the rest of the task information.

Deployment group

A deployment group is a set of deployment target machines that have agents installed. A deployment group is just another grouping of agents, like an agent pool. You can set the deployment targets in a pipeline for a job using a deployment group. Learn more about provisioning agents for deployment groups.

Creating a Pipeline

Creating a pipeline in Azure DevOps involves defining the steps your code goes through from development to production. This can be done using the YAML syntax for a declarative approach or using the classic editor for a more visual approach.

Example: YAML Pipeline

Here’s an example of a simple YAML pipeline for a .NET application:

trigger:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - task: UseDotNet@2
            inputs:
              packageType: 'sdk'
              version: '5.x'
          - script: dotnet build
            displayName: 'Build the application'
  - stage: Test
    jobs:
      - job: TestJob
        steps:
          - script: dotnet test
            displayName: 'Run unit tests'
  - stage: Deploy
    jobs:
      - deployment: Production
        environment: 'production'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: '<service-connection>'
                    appName: '<app-name>'
                    package: '$(Build.ArtifactStagingDirectory)/**/*.zip'

Breakdown of the Example:

1. Trigger:
   • The pipeline triggers on changes to the main branch.
2. Pool:
   • The jobs run on a Microsoft-hosted Ubuntu virtual machine.
3. Stages:
   • Build Stage:
     • Uses the .NET SDK.
     • Builds the application.
   • Test Stage:
     • Runs unit tests on the built application.
   • Deploy Stage:
     • Deploys the application to an Azure Web App.

Creating a Pipeline Using the Classic Editor

1. Navigate to Pipelines:
   • Go to your Azure DevOps project and navigate to Pipelines > Pipelines.
   • Click on “New pipeline” and follow the wizard.
2. Select a Source:
   • Choose your source repository (Azure Repos, GitHub, etc.).
3. Select a Pipeline Template:
   • Choose a pipeline template or start with an empty job.
4. Configure Pipeline Tasks:
   • Add tasks to the pipeline for building, testing, and deploying your application.
   • Configure each task with the necessary parameters.
5. Save and Run:
   • Save the pipeline and run it to see the results.

Live Environment Flow

1. Code Commit:
   • Developers commit code to the repository.
2. Pipeline Trigger:
   • The pipeline is triggered automatically based on the defined triggers (e.g., on push to the main branch).
3. Build Stage:
   • The pipeline compiles the code and generates build artifacts.
4. Test Stage:
   • The pipeline runs automated tests to ensure code quality.
5. Deploy Stage:
   • The pipeline deploys the build artifacts to the specified environment (e.g., staging, production).
6. Feedback Loop:
   • The results of each stage are reported back to Azure DevOps, providing feedback on the build and deployment status.

Classic UI vs YAML Pipelines

Creating a Pipeline Using the Classic Editor with Screenshot

Select the Build Source

Select the Build Template

Configure the Build Tasks

Select the Build Agent

Types of Pipeline

In Azure DevOps, there are primarily two types of pipelines that you can use to automate your workflows: Build Pipelines and Release Pipelines. Each type of pipeline serves a different purpose and is designed to help streamline different stages of the software development lifecycle.

1. Build Pipelines

Purpose: Build pipelines focus on compiling code, running tests, and producing artifacts that can be deployed later. They are primarily concerned with continuous integration (CI).

Key Features:

• Compilation and building of code.
• Running unit tests and other types of automated tests.
• Producing build artifacts.
• Triggering on code changes (e.g., commits, pull requests).

Example Workflow:

1. Code Commit: A developer commits code to the repository.
2. Build Trigger: The build pipeline is triggered automatically by the commit.
3. Build Steps: The pipeline compiles the code, runs unit tests, and packages the application.
4. Artifact Creation: The pipeline produces artifacts (e.g., compiled binaries, Docker images) and stores them in a repository.

Example YAML for a Build Pipeline:

trigger:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: UseDotNet@2
    inputs:
      packageType: 'sdk'
      version: '5.x'
  - script: dotnet build
    displayName: 'Build the application'
  - script: dotnet test
    displayName: 'Run unit tests'
  - task: PublishBuildArtifacts@1
    inputs:
      PathtoPublish: '$(Build.ArtifactStagingDirectory)'
      ArtifactName: 'drop'

2. Release Pipelines

Purpose: Release pipelines focus on deploying build artifacts to various environments such as staging, production, etc. They are primarily concerned with continuous delivery (CD) and continuous deployment (CD).

Key Features:

• Deployment to multiple environments.
• Manual or automated approval gates.
• Post-deployment validation and tests.
• Rollback capabilities.

Example Workflow:

1. Artifact Source: The release pipeline is triggered by the availability of new build artifacts.
2. Deploy to Staging: The pipeline deploys the artifacts to a staging environment.
3. Approval Gate: Manual or automated approvals are set up before promoting to the next environment.
4. Deploy to Production: After approval, the pipeline deploys the artifacts to the production environment.
5. Post-deployment Tests: The pipeline runs tests to validate the deployment.

Example YAML for a Release Pipeline:

stages:
  - stage: Deploy
    jobs:
      - deployment: Staging
        pool:
          vmImage: 'ubuntu-latest'
        environment: 'staging'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: DownloadBuildArtifacts@0
                  inputs:
                    buildType: 'specific'
                    project: 'MyProject'
                    pipeline: 'BuildPipeline'
                    buildVersionToDownload: 'latest'
                    downloadPath: '$(Pipeline.Workspace)'
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: '<service-connection>'
                    appName: '<staging-app-name>'
                    package: '$(Pipeline.Workspace)/drop/*.zip'
  - stage: Production
    dependsOn: Deploy
    condition: succeeded()
    jobs:
      - deployment: Production
        pool:
          vmImage: 'ubuntu-latest'
        environment: 'production'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: '<service-connection>'
                    appName: '<production-app-name>'
                    package: '$(Pipeline.Workspace)/drop/*.zip'

3. Multi-stage Pipelines

Purpose: Multi-stage pipelines combine build and release pipelines into a single YAML definition. This allows for end-to-end automation of both CI and CD within a single pipeline configuration.

Key Features:

• Define multiple stages in a single YAML file.
• Combine build, test, and deployment processes.
• Manage complex workflows with dependencies between stages.

Example Workflow:

1. Code Commit: A developer commits code to the repository.
2. Build Stage: The pipeline compiles the code and runs tests.
3. Deploy Stage: The pipeline deploys the artifacts to staging and production environments sequentially.
4. Approval Gates and Tests: Include approvals and tests between stages.

Example YAML for a Multi-stage Pipeline:

trigger:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - task: UseDotNet@2
            inputs:
              packageType: 'sdk'
              version: '5.x'
          - script: dotnet build
            displayName: 'Build the application'
          - script: dotnet test
            displayName: 'Run unit tests'
          - task: PublishBuildArtifacts@1
            inputs:
              PathtoPublish: '$(Build.ArtifactStagingDirectory)'
              ArtifactName: 'drop'

  - stage: Deploy
    dependsOn: Build
    jobs:
      - deployment: Staging
        pool:
          vmImage: 'ubuntu-latest'
        environment: 'staging'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: DownloadBuildArtifacts@0
                  inputs:
                    buildType: 'specific'
                    project: 'MyProject'
                    pipeline: 'BuildPipeline'
                    buildVersionToDownload: 'latest'
                    downloadPath: '$(Pipeline.Workspace)'
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: '<service-connection>'
                    appName: '<staging-app-name>'
                    package: '$(Pipeline.Workspace)/drop/*.zip'

  - stage: Production
    dependsOn: Deploy
    condition: succeeded()
    jobs:
      - deployment: Production
        pool:
          vmImage: 'ubuntu-latest'
        environment: 'production'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureWebApp@1
                  inputs:
                    azureSubscription: '<service-connection>'
                    appName: '<production-app-name>'
                    package: '$(Pipeline.Workspace)/drop/*.zip'

Summary

• Build Pipelines: Focus on compiling code, running tests, and producing artifacts. They handle continuous integration (CI).
• Release Pipelines: Focus on deploying artifacts to various environments with gates and approvals. They handle continuous delivery (CD) and continuous deployment (CD).
• Multi-stage Pipelines: Combine both build and release pipelines into a single YAML definition for end-to-end automation of CI/CD.

Build Details

Build Variables

Custom Variables

$(variablename)  Build In Variables

Variables From PowerShell

Secrets

Environment Variable

Build Triggers

Continuous Integration  Branch Filters

Pull Request

Gated Check-in

Build Options

Build Job Properties  Demands

Build Number Format  Work-Items

Oauth Token

Build Retention & History

Retention Policy 

Change History

Key Components of a Build in Azure DevOps

In Azure DevOps, a build is a process that compiles your source code, runs tests, and produces artifacts that can be deployed to various environments. Here’s a detailed breakdown of the build process, its components, and how to configure and manage builds in Azure DevOps.

Key Components of a Build in Azure DevOps

1. Build Pipeline:
   • The definition that describes how your code is built and tested. It can be defined using a YAML file or through the classic editor.
2. Triggers:
   • Conditions that start the build process, such as code changes (CI), scheduled times, or manual triggers.
3. Pool:
   • Specifies the agent pool (either Microsoft-hosted or self-hosted) that will run the build jobs.
4. Stages:
   • High-level phases of the build process. A build can have multiple stages (e.g., build, test, package).
5. Jobs:
   • A job is a collection of steps that run sequentially on the same agent. Each job runs on a separate agent.
6. Steps:
   • Individual tasks or scripts that run within a job. Steps can include actions like running a script, calling a build tool, or publishing artifacts.
7. Artifacts:
   • Outputs of the build process, such as compiled binaries, packages, or other files, which can be used in release pipelines.

Configuring a Build Pipeline

Using YAML

Example YAML Build Pipeline:

trigger:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - task: UseDotNet@2
            inputs:
              packageType: 'sdk'
              version: '5.x'
          - script: dotnet build
            displayName: 'Build the application'
          - script: dotnet test
            displayName: 'Run unit tests'
          - task: PublishBuildArtifacts@1
            inputs:
              PathtoPublish: '$(Build.ArtifactStagingDirectory)'
              ArtifactName: 'drop'

Using the Classic Editor

1. Navigate to Pipelines:
   • Go to your Azure DevOps project and select Pipelines > Pipelines.
2. Create a New Pipeline:
   • Click on “New pipeline” and follow the steps to connect to your source repository (e.g., Azure Repos, GitHub).
3. Select a Template:
   • Choose a pre-defined template or start with an empty job.
4. Add Tasks:
   • Add tasks to compile the code, run tests, and publish artifacts.
   • Example tasks:
     • DotNetCoreCLI: To restore, build, and test .NET projects.
     • PublishBuildArtifacts: To publish the build outputs.
5. Save and Queue the Build:
   • Save the pipeline and manually queue a build to see it in action.

Detailed Breakdown of the Example YAML

1. Trigger:
   • The pipeline is configured to trigger on changes to the main branch.

trigger:
  branches:
    include:
      - main

2.Pool:

• Specifies the use of a Microsoft-hosted agent with the ubuntu-latest image.

pool:
  vmImage: 'ubuntu-latest'

3.Stages:

• Defines a single stage named Build.

stages:
  - stage: Build

4.Jobs:

• Within the Build stage, a job named BuildJob is defined.

  jobs:
    - job: BuildJob

5.Steps:

• UseDotNet@2 Task: Installs the .NET SDK version 5.x.

    steps:
      - task: UseDotNet@2
        inputs:
          packageType: 'sdk'
          version: '5.x'

Script Step: Runs dotnet build to compile the application

      - script: dotnet build
        displayName: 'Build the application'

Script Step: Runs dotnet test to execute unit tests.

      - script: dotnet test
        displayName: 'Run unit tests'

PublishBuildArtifacts@1 Task: Publishes the build artifacts to be used in subsequent stages or release pipelines.

      - task: PublishBuildArtifacts@1
        inputs:
          PathtoPublish: '$(Build.ArtifactStagingDirectory)'
          ArtifactName: 'drop'

Managing Builds

1. Monitoring Builds:
   • After triggering a build, you can monitor its progress in the Pipelines section.
   • View logs for each step to diagnose issues.
2. Build Results:
   • The build results page shows the status of each job and step.
   • Provides information on test results, code coverage, and build artifacts.
3. Build Artifacts:
   • Artifacts produced by the build can be downloaded or used in release pipelines.
   • Example: A compiled application package, Docker image, or other deployable units.

Summary

Build pipelines in Azure DevOps automate the process of compiling, testing, and packaging code. They consist of triggers, pools, stages, jobs, steps, and artifacts, which together define the workflow for continuous integration. By using YAML or the classic editor, you can configure and manage build pipelines to ensure your code is always in a deployable state. This automation enhances efficiency, consistency, and reliability in the software development lifecycle.