Advanced Terraform with AWS (Part 1)

Navigating the Complexities of Global Cloud Migration

In today's rapidly evolving cloud ecosystem, global enterprises face a unique set of challenges. As organisations strive to transition to the cloud, managing a multitude of multi-regional projects using tools like Terraform can become a intricate endeavour. Many organisations opt to spread their projects across multiple environments, not only to a minimum of three, to ensure redundancy and efficiency. Imagine orchestrating a cloud infrastructure spanning three regions: Asia, the Americas, and Europe. Now, multiply this by three environments - development (dev), integration (int), and production (prod). Factor in the myriad individual projects each organisation maintains, and this quickly becomes a vast, multi-dimensional cloud infrastructure maze.

Adding to this complexity, organisations must ensure their cloud setup adheres to corporate compliance and security best practices.

Overall, companies often result in deploying numerous identical resources across various contexts. To conquer this challenge in years past, DevOps teams frequently turned to Terragrunt to manage and reuse Terraform configurations and modules, minimising repetitive code when provisioning identical resources across a multitude of layered environments.

In this advanced Terraform series, comprising three insightful articles, I will delve into the common pitfalls faced by enterprises and highlight best practices to harness the power of Terraform effectively. This will provide a practical solution to these challenges. Throughout this series, this solutions will be anchored by three guiding principles: DRY (Don't Repeat Yourself), KISS (Keep It Simple, Stupid), and a robust separation of concerns.

It is assumed that readers are already familiar with Terraform, AWS, and central state management in the cloud. In this context, utilizing AWS services like S3 and DynamoDB for Terraform's remote state storage offers substantial advantages. AWS S3 serves as a secure and scalable storage solution for the Terraform state file, ensuring reliable and continuous accessibility. Meanwhile, DynamoDB is employed for state locking and consistency checks, safeguarding the state from concurrent modifications.

As a sneak peek into what's to come, one of our primary discussions will centre around high availability network environments, utilising different data centres. We'll explore how to avoid losing a clear sight of different availability zones without compromising efficiency.

Stay tuned as I embark on this enlightening journey, demystifying the complexities of Advanced Terraform with AWS.

Managing State in Terraform

A Deep Dive into Workspaces

In the dynamic world of cloud infrastructure, ensuring a clean separation by concern becomes imperative. And when we speak of separation, Terraform state management using workspaces, which is often still undervalued, provides a robust and flexible solution to apply one configuration source to different environments and establish clear boundaries between different environments and specification.

With workspaces, we can craft environments tailored to specific requirements, be it for staging, different projects, varying regions, or a combination of these. Let's look at one example, such as managing availability zones across different projects.

The extended source code can be referenced on my related Github repository.

Given file structure

.
├── backend.tf
├── environments
│   ├── dev-network-euc1-config.yaml
:   : # more environments
│   ├── test-network-use1-config.yaml
:   : # more regions
│   ├── dev-application-use1-config.yaml
:   : # more projects
│   └── prod-application-apse1-config.yaml
├── load_config.tf
├── network.tf
├── provider.tf
├── settings.tf
└── services.tf

A useful convention for workspace naming is: {env}-{project}-{region}-{suffix}. For instance, let's zoom in on a scenario configuring 3 different availability zones in the Frankfurt AWS region: Here is the related Terraform workspace name example: dev-network-euc1-config

This naming pattern not only provides clarity but also aids in mapping the workspace name to our configuration files.

Defining variables inside a YAML Config File

Manage a separate config file for each workspace in a custom subdirectory. Declaring different configuration for each workspace with in a YAML file can enhance readability. Below are examples for dev-network-euc1-config.yaml and prod-network-use1-config.yamlfile.

# dev-network-euc1-config.yaml
spec:
  env: dev
  project: network
  region: eu-central-1
  cidr: 10.10.0.0/24
  # more network config...

# prod-network-use1-config.yaml
spec:
  env: prod
  project: network
  region: us-east-1
  cidr: 10.20.0.2/24
  # more network config...

Handling the Workspace

Create a terraform workspace for each environment.

# run this command for each environment
terraform workspace new dev-network-euc1-config
Created and switched to workspace "dev-network-euc1-config"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.

Verify which workspace is currently selected.

terraform workspace list

* default
  dev-network-euc1-config
  # more environments
  test-network-use1-config
  # more regions
  dev-application-use1-config
  # more projects
  prod-application-apse1-config

To switch to a particular workspace, use the select command. Command example:

terraform workspace select dev-network-euc1-config

Dynamically reading individual configurations from a YAML file

Having selected the workspace, we can now dynamically reference the workspace name with terraform.workspace to map the corresponding config file. This forms the linchpin of our dynamic configuration management strategy and provides the desired flexibility automatically. Below is a snipped of the main.tf file in the workspace_config module utilised in the related Github repository.

locals {
  env = var.env == "" ? terraform.workspace : var.env
  return = var.ignore != local.env ? (lower(var.file_extention) == "json" ?
    jsondecode(file("${var.path}${var.prefix}${local.env}${var.suffix}.json"))
  : yamldecode(file("${var.path}${var.prefix}${local.env}${var.suffix}.${var.file_extention}"))) : {}
}

In this example, we can dynamically utilise the interpolaton functions jsondecode or yamldecode to convert any provided JSON or YAML configuration into hcl.

This setup provides various advantages and is designed to avoid common pitfalls as outlined below.

Modularity: Each workspace acts as a self-contained environment, allowing clear boundaries between different environments.
Flexibility: Easily manage variations in environments; in this example, we can define 9 separate YAML configurations.
Consistency: Using a standard naming convention reduces ambiguities and ensures everyone is on the same page.
Dynamic Configuration: The ability to map workspace names to config files means configurations can be read dynamically, making infrastructure setups more agile and responsive.
Prevention: Avoid accidentally mixing up environments with configurations by applying the wrong config to an incorrect region, etc.
Automation: Given the concise logic, the implementation of CI/CD becomes incredibly easy and provides support to run asynchronously.

Final Thoughts

In conclusion, the realm of cloud infrastructure demands precise and distinct delineations. Terraform's state management with workspaces rises to this challenge, offering a structured yet adaptable means to segregate various environments and setups. By adhering to conventions, and expressive naming pattern, clarity is enhanced, reducing potential confusion or errors. Utilising workspaces in Terraform ensures modularity, consistency, and dynamic configuration, streamlining infrastructure tasks and bolstering automation capabilities. This approach not only fortifies our infrastructure against inadvertent mishaps but also primes it for seamless integration with CI/CD processes, emphasising agility and responsiveness in a cloud-centric era.