Terraform Pipeline
A Terraform pipeline automates the process of planning, applying, and managing infrastructure as code (IaC) with Terraform. By integrating Terraform with a CI/CD pipeline, you can ensure consistent, repeatable, and automated deployments of infrastructure changes. This helps maintain the integrity of your infrastructure and reduces the risk of human error.
Steps to Set Up a Terraform Pipeline
- Version Control Integration: Use a version control system (e.g., Git) to store and manage your Terraform configurations.
- CI/CD Tool Selection: Choose a CI/CD tool (e.g., Jenkins, GitHub Actions, GitLab CI, CircleCI) to automate the Terraform workflow.
- Pipeline Configuration: Define the pipeline stages and steps for linting, formatting, planning, applying, and destroying Terraform configurations.
- Environment Management: Manage multiple environments (e.g., dev, staging, prod) using Terraform workspaces or separate configuration files.
Example: Setting Up a Terraform Pipeline with GitHub Actions
Prerequisites
- A GitHub repository containing your Terraform configurations.
- An AWS account (or any cloud provider) configured with appropriate IAM roles and permissions.
- GitHub Actions enabled for your repository.
Step 1: Terraform Configuration
Create a simple Terraform configuration in your repository.
File: main.tf
provider "aws" {
region = "us-east-1"
}
resource "aws_instance" "example" {
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t2.micro"
tags = {
Name = "example-instance"
}
}
Step 2: GitHub Actions Workflow Configuration
Create a GitHub Actions workflow file to automate the Terraform process.
File: .github/workflows/terraform.yml
name: Terraform CI
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
terraform:
name: 'Terraform'
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: 'us-east-1'
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.0
- name: Terraform Init
run: terraform init
- name: Terraform Format
run: terraform fmt -check
- name: Terraform Validate
run: terraform validate
- name: Terraform Plan
run: terraform plan
- name: Terraform Apply
if: github.ref == 'refs/heads/main'
run: terraform apply -auto-approve
- name: Terraform Destroy
if: github.event.pull_request.merged == true
run: terraform destroy -auto-approve
name: Terraform CI
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
terraform:
name: 'Terraform'
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: 'us-east-1'
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.0
- name: Terraform Init
run: terraform init
- name: Terraform Format
run: terraform fmt -check
- name: Terraform Validate
run: terraform validate
- name: Terraform Plan
run: terraform plan
- name: Terraform Apply
if: github.ref == 'refs/heads/main'
run: terraform apply -auto-approve
- name: Terraform Destroy
if: github.event.pull_request.merged == true
run: terraform destroy -auto-approve
Explanation of the Workflow
- name: The name of the workflow.
- on: Specifies the events that trigger the workflow (push to main branch and pull requests to main branch).
- jobs: Defines the job to be run, named
terraform
. - runs-on: Specifies the type of runner to use (
ubuntu-latest
). - env: Sets environment variables for AWS credentials.
- steps:
- Checkout code: Uses the
actions/checkout
action to clone the repository. - Set up Terraform: Uses the
hashicorp/setup-terraform
action to install Terraform. - Terraform Init: Initializes Terraform.
- Terraform Format: Checks the Terraform files for proper formatting.
- Terraform Validate: Validates the Terraform configuration.
- Terraform Plan: Runs
terraform plan
to show the changes that will be applied. - Terraform Apply: Applies the changes if the branch is
main
. - Terraform Destroy: Destroys the resources if the pull request is merged.
- Checkout code: Uses the
Step 3: Managing Secrets
- Add AWS Credentials to GitHub Secrets:
- Go to your GitHub repository settings.
- Navigate to “Secrets” and then “Actions”.
- Add new repository secrets for
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
.
Step 4: Creating and Merging Pull Requests
- Create a Branch: Create a new branch and make changes to your Terraform configuration.
- Open a Pull Request: Open a pull request from your branch to the
main
branch. - Review and Merge: Review the pull request. The pipeline will run
terraform plan
on the pull request. - Merge the Pull Request: Once approved, merge the pull request. The pipeline will then run
terraform apply
on themain
branch.
Detailed Explanation for above code
name: Terraform CI
This sets the name of the workflow, which is “Terraform CI”. It is used to identify the workflow in the GitHub Actions interface.
on:
This specifies the events that trigger the workflow.
- push: The workflow will run when there is a push to the
main
branch. - pull_request: The workflow will run when a pull request is opened, synchronized, or reopened targeting the
main
branch.
jobs:
This defines the jobs to be run as part of the workflow. In this case, there is one job named terraform
.
terraform:
This is the identifier for the job and contains the steps that will be executed.
name: 'Terraform'
This sets the display name of the job to “Terraform”.
runs-on: ubuntu-latest
This specifies the type of runner to use for the job. ubuntu-latest
is a GitHub-hosted runner with the latest version of Ubuntu.
env:
This section sets environment variables for the job. These variables are used to authenticate with AWS.
- AWS_ACCESS_KEY_ID: Retrieved from GitHub Secrets, this is the AWS access key ID.
- AWS_SECRET_ACCESS_KEY: Retrieved from GitHub Secrets, this is the AWS secret access key.
- AWS_DEFAULT_REGION: Sets the AWS region to
us-east-1
.
steps:
This defines the individual steps to be executed in the job.
Step 1: Checkout Code
- name: Checkout code
uses: actions/checkout@v2
- name: Checkout code: The name of the step.
- uses: actions/checkout@v2: Uses the
actions/checkout
action to clone the repository’s code into the runner.
Step 2: Set up Terraform
- name: Set up Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.0
- name: Set up Terraform: The name of the step.
- uses: hashicorp/setup-terraform@v1: Uses the
hashicorp/setup-terraform
action to set up Terraform. - with: Specifies the version of Terraform to install. Here,
terraform_version: 1.0.0
is specified.
Step 3: Terraform Init
- name: Terraform Init
run: terraform init
- name: Terraform Init: The name of the step.
- run: terraform init: Runs the
terraform init
command to initialize the Terraform configuration.
Step 4: Terraform Format
- name: Terraform Format
run: terraform fmt -check
- name: Terraform Format: The name of the step.
- run: terraform fmt -check: Runs the
terraform fmt -check
command to check the formatting of the Terraform files.
Step 5: Terraform Validate
- name: Terraform Validate
run: terraform validate
- name: Terraform Validate: The name of the step.
- run: terraform validate: Runs the
terraform validate
command to validate the Terraform configuration files.
Step 6: Terraform Plan
- name: Terraform Plan
run: terraform plan
- name: Terraform Plan: The name of the step.
- run: terraform plan: Runs the
terraform plan
command to generate and show an execution plan.
Step 7: Terraform Apply
- name: Terraform Apply
if: github.ref == 'refs/heads/main'
run: terraform apply -auto-approve
Step 8: Terraform Destroy
- name: Terraform Destroy
if: github.event.pull_request.merged == true
run: terraform destroy -auto-approve
- name: Terraform Destroy: The name of the step.
- if: github.event.pull_request.merged == true: This condition ensures that the step only runs if a pull request was merged.
- run: terraform destroy -auto-approve: Runs the
terraform destroy
command with the-auto-approve
flag to destroy the resources without requiring interactive approval.
This GitHub Actions workflow automates the Terraform process, ensuring that infrastructure changes are consistently and automatically applied. Here’s what each part of the workflow does:
- Triggers: The workflow runs on pushes to the
main
branch and on pull requests targeting themain
branch. - Job Setup: It runs on an
ubuntu-latest
runner and uses environment variables for AWS credentials. - Steps:
- Checkout Code: Clones the repository.
- Set Up Terraform: Installs Terraform.
- Terraform Init: Initializes Terraform.
- Terraform Format: Checks Terraform file formatting.
- Terraform Validate: Validates the Terraform configuration.
- Terraform Plan: Generates a Terraform execution plan.
- Terraform Apply: Applies the Terraform changes (only on the
main
branch). - Terraform Destroy: Destroys the infrastructure if a pull request is merged.
This setup ensures that infrastructure changes are reviewed, validated, and applied in a controlled manner, leveraging the automation capabilities of GitHub Actions.
Example: Managing Multiple Environments
To manage multiple environments, you can use Terraform workspaces or separate configuration files for each environment.
Using Terraform Workspaces
- Initialize Workspaces:
terraform workspace new dev
terraform workspace new prod
- Configure the GitHub Actions Workflow:
Update the workflow to handle different workspaces:
File: .github/workflows/terraform.yml
In this configuration:
- The
Select Workspace
step selects the appropriate workspace based on the branch (prod
formain
anddev
for other branches).
Method 1:
name: Terraform CI
on:
push:
branches:
- main
- dev
pull_request:
branches:
- main
- dev
jobs:
terraform:
name: 'Terraform'
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: 'us-east-1'
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.0
- name: Terraform Init
run: terraform init
- name: Select Workspace
run: terraform workspace select ${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}
- name: Terraform Format
run: terraform fmt -check
- name: Terraform Validate
run: terraform validate
- name: Terraform Plan
run: terraform plan
- name: Terraform Apply
if: github.ref == 'refs/heads/main'
run: terraform apply -auto-approve
- name: Terraform Destroy
if: github.event.pull_request.merged == true
run: terraform destroy -auto-approve
Method 2:
name: Terraform CI
on:
push:
branches:
- main
- dev
pull_request:
branches:
- main
- dev
jobs:
terraform:
name: 'Terraform'
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: 'us-east-1'
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.0
- name: Terraform Init
run: terraform init
- name: Select Workspace
id: workspace
run: |
if [ "${{ github.ref }}" == "refs/heads/main" ]; then
terraform workspace select prod || terraform workspace new prod
else
terraform workspace select dev || terraform workspace new dev
fi
- name: Terraform Format
run: terraform fmt -check
- name: Terraform Validate
run: terraform validate
- name: Terraform Plan
run: terraform plan
- name: Terraform Apply
if: github.ref == 'refs/heads/main'
run: terraform apply -auto-approve
- name: Terraform Apply Dev
if: github.ref == 'refs/heads/dev'
run: terraform apply -auto-approve
- name: Terraform Destroy
if: github.event.pull_request.merged == true
run: |
if [ "${{ github.ref }}" == "refs/heads/main" ]; then
terraform destroy -auto-approve
else
terraform destroy -auto-approve
fi
Method 1: Inline Conditional Expression
- name: Select Workspace
run: terraform workspace select ${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}
Explanation
- Inline Conditional Expression: This method uses a single line of code to determine the workspace based on the branch.
- Syntax: The syntax
github.ref == 'refs/heads/main' && 'prod' || 'dev'
is a ternary-like expression in GitHub Actions. It checks if the branch ismain
. If true, it selects theprod
workspace; otherwise, it selects thedev
workspace.
Pros
- Simplicity: It’s a concise one-liner that’s easy to read and understand for simple conditions.
- Readability: For those familiar with ternary-like expressions, it’s straightforward.
Cons
- No Creation of Workspace: This method only selects an existing workspace. If the workspace does not exist, it will fail and will not create a new workspace.
- Less Flexibility: It’s less flexible for more complex logic or additional commands
Method 2: Multi-Line Conditional Script
- name: Select Workspace
id: workspace
run: |
if [ "${{ github.ref }}" == "refs/heads/main" ]; then
terraform workspace select prod || terraform workspace new prod
else
terraform workspace select dev || terraform workspace new dev
fi
Explanation
- Multi-Line Conditional Script: This method uses a multi-line script to handle the workspace selection and creation.
- Shell Scripting: It utilizes shell scripting to check the branch and either select the existing workspace or create a new one if it doesn’t exist.
Pros
- Workspace Creation: This method can create the workspace if it doesn’t exist (
terraform workspace select prod || terraform workspace new prod
). - Flexibility: It’s more flexible and can be extended to include additional logic or commands.
- Error Handling: It can handle errors more gracefully by trying to create the workspace if selection fails.
Cons
- Complexity: Slightly more complex and verbose compared to the one-liner.
Detailed Comparison
Inline Conditional Expression
- Simplicity:
- name: Select Workspace
run: terraform workspace select ${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}
- This method is straightforward for selecting between two options based on a condition. It uses GitHub Actions’ inline conditional syntax.
- Functionality:
- Selects Workspace: It selects the
prod
workspace if the branch ismain
, anddev
otherwise. - No Creation: It will fail if the specified workspace does not already exist.
- Selects Workspace: It selects the
Multi-Line Conditional Script
- Flexibility:
- name: Select Workspace
id: workspace
run: |
if [ "${{ github.ref }}" == "refs/heads/main" ]; then
terraform workspace select prod || terraform workspace new prod
else
terraform workspace select dev || terraform workspace new dev
fi
- This method provides more flexibility by using a shell script to perform the selection and creation logic. It’s useful for more complex scenarios where additional logic or commands might be needed.
- Functionality:
- Selects or Creates Workspace: It attempts to select the
prod
ordev
workspace and creates it if it doesn’t exist. - Error Handling: More robust in handling scenarios where the workspace may not exist.
- Selects or Creates Workspace: It attempts to select the
Summary
- Use the Inline Conditional Expression for simplicity when you only need to select between two workspaces and are sure they exist.
- Use the Multi-Line Conditional Script for flexibility, especially when you need to ensure that the workspace is created if it doesn’t exist or when additional logic is required.
By setting up a Terraform pipeline, you can automate the deployment and management of your infrastructure. Using GitHub Actions as an example, you can see how to integrate Terraform with CI/CD tools to create a robust and automated workflow. This ensures consistent and repeatable infrastructure deployments, reducing the risk of errors and improving overall efficiency.
Automating infrastructure deployments in the Cloud with Terraform and Azure Pipelines
https://azuredevopslabs.com/labs/vstsextend/terraform/
Automating infrastructure deployments with Terraform and Azure Pipelines can significantly streamline and improve the efficiency of managing cloud resources. Below, I’ll walk you through the steps to set up an Azure Pipeline to automate Terraform deployments in an Azure environment.
Prerequisites
- Azure Subscription: Ensure you have an Azure subscription.
- Azure DevOps Account: Set up an Azure DevOps organization and project.
- Service Principal: Create a service principal to authenticate with Azure.
- Terraform Configuration: Write your Terraform configuration files.
Steps
- Create a Service Principal
- Create an Azure DevOps Project
- Store Terraform Code in a Repository
- Create Azure Pipeline
Detailed Steps
Step 1: Create a Service Principal
Create a service principal in Azure to allow Azure Pipelines to authenticate and deploy resources.
az ad sp create-for-rbac --name terraform-pipeline-sp --role Contributor --scopes /subscriptions/YOUR_SUBSCRIPTION_ID --sdk-auth
his command will output JSON with details required to authenticate, such as client ID, client secret, and tenant ID.
Step 2: Create an Azure DevOps Project
Create a project in Azure DevOps if you don’t have one already.
Step 3: Store Terraform Code in a Repository
Store your Terraform configuration files in a repository (e.g., Azure Repos or GitHub).
Example Terraform Configuration: main.tf
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "example" {
name = "example-resources"
location = "West US"
}
resource "azurerm_storage_account" "example" {
name = "examplestorageacc"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
account_tier = "Standard"
account_replication_type = "LRS"
}
Step 4: Create Azure Pipeline
Create a pipeline in Azure DevOps to automate Terraform deployment.
Create a Pipeline YAML File: .azure-pipelines.yml
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
variables:
# Azure service principal details
ARM_CLIENT_ID: '$(ARM_CLIENT_ID)'
ARM_CLIENT_SECRET: '$(ARM_CLIENT_SECRET)'
ARM_SUBSCRIPTION_ID: '$(ARM_SUBSCRIPTION_ID)'
ARM_TENANT_ID: '$(ARM_TENANT_ID)'
# Terraform backend configuration
TF_VAR_storage_account_name: '$(TF_VAR_storage_account_name)'
TF_VAR_container_name: '$(TF_VAR_container_name)'
TF_VAR_key: '$(TF_VAR_key)'
steps:
- task: AzureCLI@2
inputs:
azureSubscription: '$(azureServiceConnection)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az ad sp create-for-rbac --name terraform-pipeline-sp --role Contributor --scopes /subscriptions/$(ARM_SUBSCRIPTION_ID) --sdk-auth > auth.json
ARM_CLIENT_ID=$(jq -r .clientId auth.json)
ARM_CLIENT_SECRET=$(jq -r .clientSecret auth.json)
ARM_SUBSCRIPTION_ID=$(jq -r .subscriptionId auth.json)
ARM_TENANT_ID=$(jq -r .tenantId auth.json)
echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$ARM_CLIENT_ID"
echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]$ARM_CLIENT_SECRET"
echo "##vso[task.setvariable variable=ARM_SUBSCRIPTION_ID]$ARM_SUBSCRIPTION_ID"
echo "##vso[task.setvariable variable=ARM_TENANT_ID]$ARM_TENANT_ID"
- task: UsePythonVersion@0
inputs:
versionSpec: '3.x'
addToPath: true
- script: |
pip install -r requirements.txt
displayName: 'Install dependencies'
- script: |
terraform init
displayName: 'Terraform Init'
- script: |
terraform plan
displayName: 'Terraform Plan'
- script: |
terraform apply -auto-approve
displayName: 'Terraform Apply'
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
variables:
# Azure service principal details
ARM_CLIENT_ID: '$(ARM_CLIENT_ID)'
ARM_CLIENT_SECRET: '$(ARM_CLIENT_SECRET)'
ARM_SUBSCRIPTION_ID: '$(ARM_SUBSCRIPTION_ID)'
ARM_TENANT_ID: '$(ARM_TENANT_ID)'
# Terraform backend configuration
TF_VAR_storage_account_name: '$(TF_VAR_storage_account_name)'
TF_VAR_container_name: '$(TF_VAR_container_name)'
TF_VAR_key: '$(TF_VAR_key)'
steps:
- task: AzureCLI@2
inputs:
azureSubscription: '$(azureServiceConnection)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az ad sp create-for-rbac --name terraform-pipeline-sp --role Contributor --scopes /subscriptions/$(ARM_SUBSCRIPTION_ID) --sdk-auth > auth.json
ARM_CLIENT_ID=$(jq -r .clientId auth.json)
ARM_CLIENT_SECRET=$(jq -r .clientSecret auth.json)
ARM_SUBSCRIPTION_ID=$(jq -r .subscriptionId auth.json)
ARM_TENANT_ID=$(jq -r .tenantId auth.json)
echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$ARM_CLIENT_ID"
echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]$ARM_CLIENT_SECRET"
echo "##vso[task.setvariable variable=ARM_SUBSCRIPTION_ID]$ARM_SUBSCRIPTION_ID"
echo "##vso[task.setvariable variable=ARM_TENANT_ID]$ARM_TENANT_ID"
- task: UsePythonVersion@0
inputs:
versionSpec: '3.x'
addToPath: true
- script: |
pip install -r requirements.txt
displayName: 'Install dependencies'
- script: |
terraform init
displayName: 'Terraform Init'
- script: |
terraform plan
displayName: 'Terraform Plan'
- script: |
terraform apply -auto-approve
displayName: 'Terraform Apply'
Detailed Explanation
- trigger: Automatically triggers the pipeline on changes to the
main
branch. - pool: Specifies the agent pool (
ubuntu-latest
). - variables: Sets environment variables for the Azure service principal and Terraform backend configuration.
- steps: Defines the steps to be executed in the pipeline.
Step 1: Authenticate with Azure
- task: AzureCLI@2
inputs:
azureSubscription: '$(azureServiceConnection)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az ad sp create-for-rbac --name terraform-pipeline-sp --role Contributor --scopes /subscriptions/$(ARM_SUBSCRIPTION_ID) --sdk-auth > auth.json
ARM_CLIENT_ID=$(jq -r .clientId auth.json)
ARM_CLIENT_SECRET=$(jq -r .clientSecret auth.json)
ARM_SUBSCRIPTION_ID=$(jq -r .subscriptionId auth.json)
ARM_TENANT_ID=$(jq -r .tenantId auth.json)
echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$ARM_CLIENT_ID"
echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]$ARM_CLIENT_SECRET"
echo "##vso[task.setvariable variable=ARM_SUBSCRIPTION_ID]$ARM_SUBSCRIPTION_ID"
echo "##vso[task.setvariable variable=ARM_TENANT_ID]$ARM_TENANT_ID"
- AzureCLI@2: Uses the Azure CLI task to authenticate and set environment variables for the service principal.
Step 2: Set Up Python
- task: UsePythonVersion@0
inputs:
versionSpec: '3.x'
addToPath: true
- UsePythonVersion@0: Ensures the pipeline uses Python 3.x.
Step 3: Install Dependencies
- script: |
pip install -r requirements.txt
displayName: 'Install dependencies'
- script: Installs Python dependencies (if any).
Step 4: Terraform Init
- script: |
terraform init
displayName: 'Terraform Init'
- script: Initializes Terraform configuration.
Step 5: Terraform Plan
- script: |
terraform plan
displayName: 'Terraform Plan'
- script: Generates and displays the Terraform execution plan.
Step 6: Terraform Apply
- script: |
terraform apply -auto-approve
displayName: 'Terraform Apply'
- script: Applies the Terraform configuration.
Summary
By following these steps, you can set up an Azure Pipeline to automate the deployment of your infrastructure using Terraform. This integration ensures that your infrastructure changes are applied consistently and automatically, leveraging the power of CI/CD pipelines to manage your cloud resources effectively.
Create a CI/CD pipeline to validate Terraform configurations by using AWS CodePipeline
https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/create-a-ci-cd-pipeline-to-validate-terraform-configurations-by-using-aws-codepipeline.html