After figuring out the project in Part 1 by clicking then using Terraform, here it goes a step further by adding multiple environments while reducing manual work.

GitOps philosophy

GitOps is a methodology rooted around git as single source of truth. Pushing to version control is then the only manual step to make the actual state and desired state converge.

Git as source of desired state

When everything is defined as code and code lives in version control, git can be the single source of truth. It then follows logically that all actions are performed as a response to pushing changes to git.

GitOps is the opposite of ClickOps

Clicking around is a good way to learn and figure things out but is not efficient in the long run. For example, if pushing to too many repositories at the same time, it's highly probable that someone forgets a step. GitOps removes this probability when operating important systems.

"You don't do GitOps"

Terraform performed correctly is GitOps. Correctly means it is declarative code (desired state) and is pushed to git, which triggers a process that converges the actual state with the desired state.

GitOps applied to the Cloud Resume Challenge

Workflow

1. Infrastructure changes pushed to GitHub

2. Cloud Build GitHub app triggers the build jobs

3. Pull request and merge to the dev branch → new infrastructure for the dev environment can be tested

4. Pull request and merge to the prod branch → new infrastructure for the prod environment

Terraform

To match GitOps methodology, the repository structure is modified to have one folder per environment.

terraform
├── environments
│   ├── dev
│   │   ├── .terraform
│   │   ├── .terraform.lock.hcl
│   │   ├── backend-config.tf
│   │   ├── main.tf
│   │   ├── terraform.tfvars
│   │   └── variables.tf
│   └── prod
│       ├── .terraform
│       ├── .terraform.lock.hcl
│       ├── backend-config.tf
│       ├── main.tf
│       ├── terraform.tfvars
│       └── variables.tf
├── modules
│   ├── gcs
│   │   ├── main.tf
│   │   └── outputs.tf
│   └── lb
│       ├── main.tf
│       └── outputs.tf
├── cloudbuild-assets.yaml
└── cloudbuild-infra.yaml

• One Terraform state per environment: having different states for different environments provides logical separation.

• Terraform state stored remotely: to allow multiple people to work on the project, Terraform state is stored in a Cloud Storage bucket defined in backend-config.tf.

• DRY principle applied to resources: since we have multiple environments using the same resources, we define them outside of the environments folders, in a modules folder. Resources can then be reused easily for new environments.

Cloud Build

Configuration file

The configuration file contains the logic to converge actual and desired state. It checks the branch name and the presence of an environment folder of the same name. If there is a match, terraform apply is executed to converge states.

Triggers

Triggers are linked to a repository, here front-end and back-end. I have a third trigger for the website files so that they are uploaded to Google Cloud Storage without triggering terraform.

Conclusion

As someone wanting to work in Data Engineering, learning about Cloud Engineering and DevOps gives me a better understanding of each stakeholder's role. This challenge has been the best learning experience before entering the professional world. I appreciate any feedback ! Thanks for reading 📚

Repositories:

Front-end

Back-end

Resources:

DevOps Paradox - Episode 74: Using GitOps in Your DevOps Workflow

Managing infrastructure as code with Terraform, Cloud Build, and GitOps