Skip to content

The .gitlab-ci.yml File

Filename Location Group Project/Repository
.gitlab-ci.yml ./.gitlab-ci.yml infrastructure terraform

Why?

We need to instruct GitLab CI on how our pipeline is to be composed. This file will specifically address the requirements needed by our Terraform code base.

In this file you'll notice a lot of references to terraform-gitlab. This is a "wrapper" that augment's the Terraform implementation GitLab providers so that all of the configuration that Terraform needs is provided in the environment. Just know that it's there to help us operate Terraform inside of GitLab CI pipelines.

Breakdown

Here is a breakdown of the Terraform pipeline configuration file .gitlab-ci.yml. The file is a YAML file and is pretty straight forward. We'll break it down into keywords and explain what their purpose is.

Note

This section is going to contain some repetition from the previous section. This is because we've broken down this part of the configuration already to explain how a pipeline is constructed. I've repeated the content for clarity.

Configuration

Let's look at the very first few lines and checkout what's happening:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
variables:
  TF_ROOT: ${CI_PROJECT_DIR}
  TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}

cache:
  key: httpcats-beta
  paths:
    - ${TF_ROOT}/.terraform

before_script:
  - cd ${TF_ROOT}

All of this is configuring GitLab CI to behave in a particular way and do some tasks for use ahead of each stage. I think we should go over each item (above)...

image:

This configures the entire pipeline to run all script: configurations (explained below) in a Docker container using a specific image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest.

This particular image is perfect for our needs not just because it provides Terraform but because it's suitable for us inside of GitLab CI pipelines due to some bootstrapping that's being done around Terraform. This will become more clear later on.

variables:

This configuration keyword allows us to define variables that are available for use across the entire pipeline, in all stages, and can be used for all kinds of things.

cache:

Using the cache: keyword we can have the pipeline cache certain files and or directories between stages/jobs, and even across pipelines themselves. For us this is important because after we call terraform init we need to copy the .terraform/ to the other stages in the pipeline. If we didn't we would have to call terraform init for every job.

before_script:

In our stages we use the script: keyword to define the functionality of each stage and actually get our work done. The before_script: configuration is used to have a script execute before the script inside of each of our script: blocks. We're using the GitLab CI provided Terraform Docker image, so we need to use this feature to move into the TF_ROOT location.

Stages

Our pipeline's stages are as follows:

1
2
3
4
5
stages:
  - validate
  - plan
  - apply
  - destroy

These stages are stepped through, one by one, in the order shown. We have four stages:

  1. validate
  2. plan
  3. apply
  4. destroy

Let's explore each one.

Validate

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
validate:
  stage: validate
  rules:
    - if: $RUN_ANYWAY == "YES"
    - exists:
        - .destroy
      when: never
    - changes:
        - "*.tf"
  script:
    - gitlab-terraform init
Rules

What rules do we have in our validate job?

1
2
3
    - if: $RUN_ANYWAY == "YES"
    - exists:
        - .destroy

We're using an exists: keyword to determine if a file (.destroy) exists or not. If it does then the when: keyword determines what should happen, and in this case never means this stage should never be included in the pipeline.

1
2
      when: never
    - changes:

Finally we're asking GitLab CI to check for changes to a list of pattern matches. In our case we're looking for changes to any files that match *.tf, or Terraform configuration files. In the event such changes do exist then this rule evaluates to true and the stage is included in the pipeline.

Script

In our script we init the Terraform installation. Then we validate that the syntax of the code is valid. If not then the stage will fail and the pipeline will come to a halt.

In the above script we're using the gitlab-terraform

Plan

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
    - gitlab-terraform validate

plan:
  stage: plan
  artifacts:
    name: plan
    paths:
      - ${TF_ROOT}/plan.cache

    # This is a piece of magic that pushes the plan into the Terraform
    # backend of GitLab (CI)
    reports:
      terraform: ${TF_ROOT}/plan.json
  rules:
    - if: $RUN_ANYWAY == "YES"
    - exists:
        - .destroy
      when: never
    - changes:
        - "*.tf"
  script:
artefacts

With the artefacts: keyword we're telling GitLab CI to create two artefacts: the plan file for Terraform to use at later stages, and the JSON version that gets pushed into the back end of the GitLab CI Terraform solution.

As we need to generate a Terraform plan so that our apply can do its job, we use the artefacts: keyword to store it for later recovery.

Script

We produce a normal Terraform plan, a file that is used by apply to action changes after the plan has been approved.

We also produce a JSON version of the plan so that the Terraform back end built into GitLab CI and work its magic. This is background magic for GitLab internal workings. We don't have to worry about this part of the script all that much.

Apply

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
    - gitlab-terraform plan-json

apply:
  stage: apply
  environment:
    name: production
  dependencies:
    - plan
  rules:
    - if: $RUN_ANYWAY == "YES"
    - exists:
        - .destroy
      when: never

    # This is a nested condition:
    # - Include the stage if the commit branch is the project's default branch (such as master)
    # - Include the stage if there are changes to .tf files in the commit
    # - (attribute) Include the stage but make sure it's a manual run
    # - (attribute) Allow the stage to fail
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
Dependencies

The plan.cache file is downloaded into this current job from the plan job, and is used during the execution of terraform apply.

Rules

We're seeing something slightly new here:

1
    # This is a nested condition:

This is an if: statement just like we've seen before. This time, however, we checking a different variable: CI_COMMIT_BRANCH. And we're checking to see if it equal to the contents of another variable, CI_DEFAULT_BRANCH. What are these variables?

The CI_COMMIT_BRANCH pre-defined variable tells us what Git branch this job is running against. When a developer pushes code into the GitLab repository they will do so against a particular branch. This variables contains that branch name.

The CI_DEFAULT_BRANCH variable contains the default branch name for the repository. In the past this would have been master, and this is what it'll be called in Git repository that were created over a year ago. But these days this tends to be called main instead. Therefore the CI_DEFAULT_BRANCH variable will very likely be main or master.

Going back to our if:, we're asking GitLab CI to check of this particular job is being expected due to a push to the default branch. If true, then this job is included in the stage and thus the pipeline.

We're also doing something else that's interesting: when: manual. This is configuring the job to only execute based on manual intervention from a human. So this job isn't fully automated and requires us to press a button in the GitLab CI UI. you'll see this being referred to as a "manual gate" in the wild and it's good practice to put sensitive parts of your pipelines behind such gates.

Script

A simple terraform apply, but using the GitLab CI pipeline provided executable. This will execute our actual Terraform code, based on the plan.cache file (the Terraform plan) and build our network for us.

Destroy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
      when: manual
      allow_failure: true
  script:
    - gitlab-terraform apply

destroy:
  stage: destroy
  environment:
    name: production
  rules:
    - if: $RUN_ANYWAY == "YES"

    # Never include this stage if there are changes to .tf files in this commit
    # (because why would you destroy if you're making code changes?)
    - changes:
        - "*.tf"
      when: never

    # Nested condition, again...
    # - If we're on the default branch
    # - and the file .destroy exists
Rules

We're inverting the if: rule you'll find above in the apply: job. Instead of having this job execute if we detect changes to *.tf files in the current commit, we're instead saying we don't want to run this job if there are changes. We only want this job to run if there are no changes to *.tf files and a file called .destroy exists in the commit.

All the other rules we've seen before now, but also note how this job is also behind a manual gate.

Script

A simple terraform destroy with a -auto-approve flag to prevent Terraform asking us to confirm the command. This will destroy all of our network and so the requirement to create and commit a .destroy file helps us not do this accidentally.

The Solution

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
variables:
  TF_ROOT: ${CI_PROJECT_DIR}
  TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}

cache:
  key: httpcats-beta
  paths:
    - ${TF_ROOT}/.terraform

before_script:
  - cd ${TF_ROOT}

stages:
  - validate
  - plan
  - apply
  - destroy

validate:
  stage: validate
  rules:
    - if: $RUN_ANYWAY == "YES"
    - exists:
        - .destroy
      when: never
    - changes:
        - "*.tf"
  script:
    - gitlab-terraform init
    - gitlab-terraform validate

plan:
  stage: plan
  artifacts:
    name: plan
    paths:
      - ${TF_ROOT}/plan.cache

    # This is a piece of magic that pushes the plan into the Terraform
    # backend of GitLab (CI)
    reports:
      terraform: ${TF_ROOT}/plan.json
  rules:
    - if: $RUN_ANYWAY == "YES"
    - exists:
        - .destroy
      when: never
    - changes:
        - "*.tf"
  script:
    - gitlab-terraform plan
    - gitlab-terraform plan-json

apply:
  stage: apply
  environment:
    name: production
  dependencies:
    - plan
  rules:
    - if: $RUN_ANYWAY == "YES"
    - exists:
        - .destroy
      when: never

    # This is a nested condition:
    # - Include the stage if the commit branch is the project's default branch (such as master)
    # - Include the stage if there are changes to .tf files in the commit
    # - (attribute) Include the stage but make sure it's a manual run
    # - (attribute) Allow the stage to fail
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - "*.tf"
      when: manual
      allow_failure: true
  script:
    - gitlab-terraform apply

destroy:
  stage: destroy
  environment:
    name: production
  rules:
    - if: $RUN_ANYWAY == "YES"

    # Never include this stage if there are changes to .tf files in this commit
    # (because why would you destroy if you're making code changes?)
    - changes:
        - "*.tf"
      when: never

    # Nested condition, again...
    # - If we're on the default branch
    # - and the file .destroy exists
    # - (attribute) then add the stage but make it a manual step
    # - (attribute) and allow it to fail
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      exists:
        - .destroy
      when: manual
      allow_failure: true
  script:
    - gitlab-terraform destroy -auto-approve

Committing the Code

  1. Set your working directory to the infrastructure/terraform repository
  2. Save the file as .gitlab-ci.yml and use git add .gitlab-ci.yml to add it to the Git staging area
  3. Use git commit -am 'providing a CI pipeline for our IAC' to commit the file to our repository
  4. Push the code to GitLab.com: git push