GitOps with Terraform, Terragrunt and GitHub Workflows
In this post, we'll define GitOps, review a sample Terraform app module repo and an associated GitHub Workflow to lint and version the module repo, and review a sample Terragrunt live repo and a GitHub Workflow to apply infrastructure changes.
At DEPT, we love pushing the envelope of new technologies and tooling to deliver best-in-class products and solutions that delight our clients.
We use Terraform and Terragrunt to build and provision as much as we possibly can, and recently we've begun leveraging GitHub Workflows for CI/CD. In this post, we'll take a look at how we combine these tools to create a GitOps centric workflow for managing cloud infrastructure.
Specifically we'll:
- Define GitOps
- Review a sample Terraform app module repo and an associated GitHub Workflow to lint and version the module repo
- Review a sample Terragrunt live repo and a GitHub Workflow to apply infrastructure changes
GitOps
So what's GitOps?
The fundamentals are pretty straight forward:
- Git as the single source of truth of a system
- Git as the single place where we create, change and destroy all environments
- All changes are observable / verifiable
As we'll see below, combining Terraform, Terragrunt and GitHub Workflows is as GitOps as it gets.
Terraform
Terraform is a declarative, cloud agnostic tool for provisioning immutable infrastructure.
Terraform modules are a fundamental component. Any set of Terraform configuration files in a folder is a module. That's it.
Within a module, you leverage providers (AWS is a provider) to create resources (EC2 is a resource). Dynamic configuration data is defined in variables (EC2 instance class is a variable) and provisioners can be used to execute specific actions (like installing and configuring software) on hosts to prepare them for service. Each of these are present in the sample repo.
Here's our sample module repo:
https://github.com/rocketinsights/terraform-blog-sample-module
This module leverages the AWS provider to create the following resources:
- VPC
- Load Balancer
- Security Groups
- EC2 instance
- Installs/starts nginx on the EC2 instance (via a provisioner)
Linting and Versioning with a GitHub Workflow
In addition to the Terraform module code, we also have two GitHub Workflows defined; one for pull requests and one for merges to master.
The pull request workflow lints and validates formatting of the Terraform code:
| name: Lint and Validate Terraform Code | |
| on: | |
| pull_request: | |
| branches: | |
| - master | |
| jobs: | |
| build: | |
| runs-on: ubuntu-latest | |
| env: | |
| AWS_DEFAULT_REGION: us-east-1 | |
| steps: | |
| - uses: actions/checkout@v1 | |
| - name: Install Terraform and Terragrunt | |
| run: | | |
| brew tap rocketinsights/tgenv | |
| brew install tfenv tgenv | |
| tfenv install | |
| tgenv install | |
| - name: Get Versions | |
| run: | | |
| terragrunt --version | |
| terraform --version | |
| - name: Terraform Init | |
| run: find . -type f -name "*.tf" -exec dirname {} \;|sort -u | while read m; do (cd "$m" && terraform init -input=false -backend=false) || exit 1; done | |
| - name: Validate Terraform configs | |
| run: find . -name ".terraform" -prune -o -type f -name "*.tf" -exec dirname {} \;|sort -u | while read m; do (cd "$m" && terraform validate && echo "√ $m") || exit 1 ; done | |
| - name: Check Terraform config formatting | |
| run: terraform fmt -write=false -recursive |
view rawgithub-workflow-lint-terraform.yml hosted with ❤ by GitHub
The master workflow uses a special action to create and apply an auto-incremented SemVer version tag:
| name: Apply/Increment Tag | |
| on: | |
| push: | |
| branches: | |
| - master | |
| jobs: | |
| build: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@master | |
| with: | |
| fetch-depth: '0' | |
| - name: Bump version and push tag | |
| uses: anothrNick/github-tag-action@1.19.0 | |
| env: | |
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| WITH_V: true |
view rawgithub-workflow-apply-tag.yml hosted with ❤ by GitHub
Why do we care about versioning? Enter Terragrunt...
Terragrunt
Terragrunt is thin wrapper around Terraform.
It makes writing DRY Terraform modules easy and facilitates targeted control over when and where module updates are deployed.
As discussed above, our sample Terraform module is in its own repo and every time we merge to master, we automatically get a SemVer tag thanks to our fancy GitHub Workflow.
Now let's look at our Terragrunt live sample repo:
https://github.com/rocketinsights/terraform-blog-sample-live
This repo contains our Terragrunt configuration files for each environment; dev, staging and prod. Each environment specific Terragrunt file references our module at a specific version and sets any module variables required as inputs.
Because we are pinning each environment to a specific version of the module, we can make changes without affecting any running environment.
When we're ready to deploy a module change to a given environment, we simply increment the version tag in the source reference defined in the environment specific Terragrunt file. (This is usually the "ah-ha!" moment.)
Ok, great, but how do we continuously deploy this across different environments in AWS?
Applying infrastructure changes with a Github Workflow
Just as we saw in our module repo, our Terragrunt live repo also has two GitHub Workflows defined - one for pull requests and another for merges to master.
The pull request workflow outputs the plan (think dry run) of what Terragrunt is going to do (what AWS resources Terraform intends to create/update/delete):
| name: Terragrunt Plan All (dry run) | |
| on: | |
| pull_request: | |
| branches: | |
| - master | |
| jobs: | |
| build: | |
| runs-on: ubuntu-latest | |
| env: | |
| AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
| AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
| AWS_DEFAULT_REGION: us-east-1 | |
| steps: | |
| - uses: actions/checkout@v1 | |
| - name: Install Terraform and Terragrunt | |
| run: | | |
| brew tap rocketinsights/tgenv | |
| brew install tfenv tgenv | |
| tfenv install | |
| tgenv install | |
| - name: Get Versions | |
| run: | | |
| terragrunt --version | |
| terraform --version | |
| - name: Setup infra modules deploy key | |
| run: | | |
| mkdir ~/.ssh | |
| echo "${{ secrets.INFRA_MODULES_DEPLOY_KEY }}" > ~/.ssh/id_rsa | |
| chmod 600 ~/.ssh/id_rsa | |
| ssh-keyscan -t rsa github.com | |
| - name: Terragrunt plan-all | |
| run: terragrunt plan-all |
view rawgithub-workflow-terragrunt-plan-all.yml hosted with ❤ by GitHub
The master workflow takes that plan and applies it (actually creates/updates/deletes the resources in AWS based on the plan):
| name: Terragrunt Apply All (deploy) | |
| on: | |
| push: | |
| branches: | |
| - main | |
| jobs: | |
| build: | |
| runs-on: ubuntu-latest | |
| env: | |
| AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} | |
| AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | |
| AWS_DEFAULT_REGION: us-east-1 | |
| steps: | |
| - uses: actions/checkout@v1 | |
| - name: Install Terraform and Terragrunt | |
| run: | | |
| brew tap rocketinsights/tgenv | |
| brew install tfenv tgenv | |
| tfenv install | |
| tgenv install | |
| - name: Get Versions | |
| run: | | |
| terragrunt --version | |
| terraform --version | |
| - name: Setup infra modules deploy key | |
| run: | | |
| mkdir ~/.ssh | |
| echo "${{ secrets.INFRA_MODULES_DEPLOY_KEY }}" > ~/.ssh/id_rsa | |
| chmod 600 ~/.ssh/id_rsa | |
| ssh-keyscan -t rsa github.com | |
| - name: Terragrunt plan-all | |
| run: terragrunt plan-all | |
| - name: Terragrunt apply-all | |
| run: terragrunt apply-all --terragrunt-non-interactive |
view rawgithub-workflow-terragrunt-apply-all.yml hosted with ❤ by GitHub
This downloads the module from the source reference in the terraform block of the Terragrunt configuration file, sets the inputs (which correlate to the module variables) and creates the resources in AWS.
Secrets
Something to take note of regarding these workflows is their use of secrets.
From the GitHub:
libsodium sealed box
Once you create a secret, it can only be decrypted in the workflow (you can't decrypt it in the UI or via the GitHub API) and GitHub automatically redacts secrets printed to the log.
Let's look at each secret we leverage:
INFRA_MODULES_DEPLOY_KEY:
Terragrunt requires the source reference for a private GitHub repo to use the ssh:// format.
To accommodate this, we create a GitHub deploy key under the module repo and add the private key as a secret in the Terragrunt live repo. We then render the private key so it can be used when Terragrunt is called. (Our example repos are public so this is would ONLY be required if you were using private GitHub repos.)
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY:
These are the keys for the IAM user used by Terragrunt/Terraform to provision resources in AWS.
Final thoughts
Terraform, Terragrunt and GitHub Workflows are incredibly powerful tools that work extremely well together to facilitate a GitOps continuous delivery model.
At DEPT, we've used patterns like this to great effect, resulting in resilient and reliable workflows for efficiently replicating cloud infrastructure and application delivery pipelines on client projects.
We are always looking to work with both existing and new clients to apply these patterns to help evolve and adopt modern, robust and reliable patterns to their application delivery and cloud infrastructure management processes.
If you are interested in learning more about how we can help your organization or you'd like to work with us to apply these patterns to client projects, please let us know!
