Terraform for Platform Engineers: State, Modules, EKS and Pipelines

In a bank, nobody clicks in the console. If a VPC, an IAM role or an EKS cluster is not described in Terraform, reviewed in a pull request and applied by a pipeline with an approval record, then as far as the auditors are concerned it does not exist. You have used Terraform before, so this post does not teach you what a resource block is. It makes you interview-sharp on the four things a platform team actually gets grilled on: state, modules, the EKS and OpenShift specifics, and the pipeline that turns a PR into infrastructure. By the end you will be able to sketch a complete EKS build on a whiteboard, explain exactly where state lives and why, and describe a Terraform pipeline that would pass a change-management audit.

Why "not in Terraform, not real" is the rule

The JD bullets this post covers are "Develop Infrastructure-as-Code solutions using Terraform and cloud-native automation tooling" and "Support deployment patterns using ... Infrastructure as Code". Read them as a regulated-environment requirement, not a tooling preference. Infrastructure as Code (IaC) = describing infrastructure in versioned text files and letting a tool make the real thing match. For a bank that delivers the three things auditors care about: a reviewable history of every change (who, what, when, approved by whom), reproducibility (the disaster recovery plan is "re-apply from git"), and separation of duties (the person who writes a change is not the identity that applies it). A console click gives none of those, which is why the first thing a platform team does in any cloud account is remove console write access from humans.

Keep that framing for every section below. Bank interviewers are not testing whether you can write HCL; they are testing whether you understand why the process around the HCL exists.

The core workflow and the mental model

Vocabulary first so the rest is unambiguous. A provider is a plugin that talks to one API (aws, kubernetes, helm, vault, rhcs for Red Hat OpenShift on AWS). A resource is an object Terraform creates and owns (aws_eks_cluster). A data source is something Terraform reads but does not own (data "aws_vpc"). The workflow is five commands:

$ terraform init          # download providers/modules, connect to the backend, write .terraform.lock.hcl
$ terraform validate      # syntax and type check, no cloud calls
$ terraform plan -out=plan.tfplan   # refresh state, diff desired vs actual, save the diff
$ terraform apply plan.tfplan       # execute exactly that saved diff
$ terraform destroy       # plan and apply a diff that removes everything in this state

Under the hood Terraform builds a dependency graph: every reference like module.vpc.private_subnets becomes an edge, and Terraform walks the graph creating independent resources in parallel (10 at a time by default) and dependent ones in order. That graph is why you rarely need depends_on; if you are adding it everywhere, you are hiding a missing reference.

The mental model that matters: Terraform is desired-state reconciliation, the same idea as a Kubernetes controller, with one crucial difference. A Deployment controller runs continuously and fixes drift within seconds. Terraform reconciles only when you run plan and apply; between runs the real world can drift and Terraform will not notice. Idempotency = running the same apply twice produces the same result, with the second run changing nothing. A plan that shows changes every time you run it is a bug, and the troubleshooting section covers it.

One paragraph on licensing, because it comes up. In 2023 HashiCorp moved Terraform to the Business Source Licence, and the community forked the last open-source version as OpenTofu, now under the Linux Foundation. The tofu CLI is a drop-in replacement for Terraform 1.x with a few extras such as native state encryption. Most banks still run HashiCorp Terraform 1.x because the BSL permits end-user use and HCP Terraform (the renamed Terraform Cloud) is already in their vendor register; some standardised on OpenTofu after a legal review. In an interview, say "Terraform 1.x, and I know why OpenTofu exists" and move on.

HCL essentials you will be asked to write on a whiteboard

You know the syntax, so this section is about the choices behind it. Start with variables, because interviewers use them to check whether you write modules other people can safely use.

variable "environment" {
  type        = string
  description = "nonprod or prod; drives sizing and API endpoint exposure"
  validation {
    condition     = contains(["nonprod", "prod"], var.environment)
    error_message = "environment must be nonprod or prod."
  }
}

variable "node_groups" {
  type = map(object({
    instance_types = list(string)
    min_size       = number
    max_size       = number
    labels         = optional(map(string), {})
  }))
}

variable "argocd_admin_password" {
  type      = string
  sensitive = true   # redacted in plan/apply output; still stored in state
}

locals {
  name       = "bank-platform-${var.environment}"
  public_api = var.environment != "prod"   # conditional expression
  tags = {
    Environment = var.environment
    Owner       = "platform-eng"
    CostCentre  = "CC-4711"
    ManagedBy   = "terraform"
  }
}

output "cluster_endpoint" {
  value       = module.eks.cluster_endpoint
  description = "Private API endpoint; reachable from the bastion subnets only"
}

Three things to say out loud: typed variables with object and optional() make bad inputs fail at validate time rather than half-way through an apply; validation blocks encode platform standards (only two environments exist); and sensitive = true hides a value from terminal output but not from state, which is the most repeated point in this post.

for_each versus count

# count: addresses are positional -> aws_subnet.private[0], [1], [2]
resource "aws_subnet" "private" {
  count      = length(var.private_cidrs)
  vpc_id     = aws_vpc.this.id
  cidr_block = var.private_cidrs[count.index]
}

# for_each: addresses are keyed -> aws_subnet.private["a"], ["b"], ["c"]
resource "aws_subnet" "private" {
  for_each          = var.private_cidrs   # { a = "10.10.1.0/24", b = "10.10.2.0/24", c = ... }
  vpc_id            = aws_vpc.this.id
  cidr_block        = each.value
  availability_zone = "ca-central-1${each.key}"
}

With count, removing the middle element shifts every later element down one index, so Terraform plans to destroy and recreate subnets you never intended to touch. With for_each, removing key "b" destroys exactly ["b"]. The rule: for_each whenever the items have a natural identity (subnets, node groups, IAM roles, namespaces); count only for zero-or-one toggles like count = var.create_bastion ? 1 : 0. Both work on modules as well as resources.

dynamic blocks, lifecycle, moved and import

resource "aws_security_group" "bastion" {
  name   = "${local.name}-bastion"
  vpc_id = module.vpc.vpc_id

  dynamic "ingress" {              # one nested block per admin CIDR
    for_each = var.allowed_admin_cidrs
    content {
      from_port   = 22
      to_port     = 22
      protocol    = "tcp"
      cidr_blocks = [ingress.value]
    }
  }
}

resource "aws_kms_key" "eks" {
  description         = "EKS secrets envelope encryption"
  enable_key_rotation = true
  lifecycle { prevent_destroy = true }   # apply fails rather than deleting the key
}

resource "aws_launch_template" "nodes" {
  # ...
  lifecycle {
    create_before_destroy = true                 # build the replacement first
    ignore_changes        = [tags["LastScanned"]] # a scanner writes this tag; not ours
  }
}

moved {   # refactor without destroy: rename in state during the next apply
  from = aws_instance.bastion
  to   = aws_instance.bastion["a"]
}

import {  # adopt something created by hand; plan shows what will be imported
  to = aws_s3_bucket.registry
  id = "bank-platform-registry-ca-central-1"
}

dynamic blocks generate repeated nested blocks from a collection; use them sparingly because they make plans harder to read. The three lifecycle settings are the ones interviewers ask about: prevent_destroy is a seatbelt for KMS keys, state buckets and databases; create_before_destroy replaces a launch template or certificate without a gap; ignore_changes fixes "plan shows changes every time because something else manages that attribute". moved blocks (Terraform 1.1+) and import blocks (1.5+) put refactors and adoptions into the PR where they get reviewed, instead of someone running terraform state mv from a laptop. depends_on is for hidden dependencies the graph cannot see, such as an IAM policy attachment that must finish before a pod tries to assume the role; the EKS example uses it exactly once.

Sharing values between states

# Option 1: read another state's outputs directly
data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "bank-platform-tfstate-ca-central-1"
    key    = "nonprod/network/terraform.tfstate"
    region = "ca-central-1"
  }
}

# Option 2 (preferred): look the object up in the cloud API by tag
data "aws_vpc" "platform" {
  tags = { Name = "bank-platform-nonprod" }
}
data "aws_subnets" "private" {
  filter { name = "tag:Tier"; values = ["private"] }
  filter { name = "vpc-id";   values = [data.aws_vpc.platform.id] }
}

terraform_remote_state works, but the reader needs read access to the entire upstream state file, secrets included, and it couples the consumer to the producer's output names and state layout. The preferred pattern is explicit: the producer tags resources consistently or publishes a few values to SSM Parameter Store, and consumers use ordinary data sources. When asked "how does your EKS state find the VPC the network team built", answer with tags and data sources.

State: the interview favourite

State = the JSON file (terraform.tfstate) that maps every resource address in your configuration to the real object's ID and its last known attributes. Without it Terraform cannot diff desired against actual, because the cloud API does not know that vpc-0a1b2c3d is module.vpc.aws_vpc.this[0]. It contains resource IDs, every attribute Terraform read back (including database passwords, private keys from tls_private_key, any token you put in an output), dependency ordering and provider versions. Treat state as a secret. That one fact drives every decision below.

Remote backends, locking and encryption

terraform {
  required_version = "~> 1.10"
  backend "s3" {
    bucket       = "bank-platform-tfstate-ca-central-1"
    key          = "nonprod/eks/cluster/terraform.tfstate"
    region       = "ca-central-1"
    encrypt      = true
    kms_key_id   = "arn:aws:kms:ca-central-1:123456789012:key/2f1a…"
    use_lockfile = true   # S3-native locking (Terraform 1.10+); older: dynamodb_table = "tfstate-locks"
  }
}

A backend is where state is stored and how it is locked. Local state is disqualifying in a team, so every real setup uses a remote backend: S3 (versioning on, SSE-KMS, a bucket policy allowing only the pipeline roles, MFA delete or object lock for prod), GCS (backend "gcs" with bucket and prefix; locking and versioning are built in), Azure Blob (backend "azurerm"; blob leases provide locking), or HCP Terraform / Terraform Enterprise, which stores state, runs plans and keeps the audit trail. Locking stops two applies running at once: with S3 that historically meant a DynamoDB table, and since Terraform 1.10 the backend can lock with a .tflock object in the bucket itself, so DynamoDB is now optional and deprecated. Say "versioned, encrypted, locked, least-privilege bucket policy" and you have covered the checklist. Keep the backend block partial (-backend-config= at init) if one root module serves several accounts.

One state per environment and per component

The correct architecture is small states split along two axes: environment (nonprod, prod, often a DR region) and component (network, IAM and KMS, EKS cluster, cluster bootstrap, shared services). The reasons are blast radius (a bad apply in bootstrap cannot delete the VPC), speed (refreshing 3,000 resources takes ten minutes; 60 takes ten seconds), ownership (network state is applied by the network team's pipeline role, cluster state by the platform team's), and lock contention. The bucket key layout <env>/<component>/terraform.tfstate in the backend block above is exactly that.

The state commands you must know cold

$ terraform state list
module.eks.aws_eks_cluster.this[0]
module.eks.aws_iam_role.this[0]
module.eks.module.eks_managed_node_group["system"].aws_eks_node_group.this[0]
module.vpc.aws_vpc.this[0]
module.vpc.aws_subnet.private[0]

$ terraform state show 'module.eks.aws_eks_cluster.this[0]'
# module.eks.aws_eks_cluster.this[0]:
resource "aws_eks_cluster" "this" {
    arn      = "arn:aws:eks:ca-central-1:123456789012:cluster/bank-platform-nonprod"
    endpoint = "https://A1B2C3D4.gr7.ca-central-1.eks.amazonaws.com"
    version  = "1.33"
    ...
}

$ terraform state mv 'aws_instance.bastion' 'aws_instance.bastion["a"]'   # prefer a moved block in a PR
$ terraform state rm 'aws_s3_bucket.legacy'   # forget it; the bucket keeps existing
$ terraform state pull > state-backup-$(date +%F).json   # before any surgery
$ terraform import 'aws_security_group.bastion' sg-0123456789abcdef0   # prefer an import block

state rm is how you hand a resource to another state without destroying it: remove it here, import it there. The CLI forms of mv and import still work, but in a bank the reviewed equivalent is a moved or import block in a PR, so the change to state has the same audit trail as any other change. Refreshing = updating recorded attributes from the real API; plan does it automatically, terraform apply -refresh-only updates state without touching infrastructure (the replacement for the deprecated terraform refresh), and -refresh=false speeds up a plan when you know nothing drifted.

Drift detection, recovery, and the two controlled weapons

$ terraform plan -input=false -lock=false -detailed-exitcode -no-color
...
No changes. Your infrastructure matches the configuration.
$ echo $?
0     # 0 = no drift, 1 = error, 2 = plan has changes (drift or unapplied commits)

A nightly job runs that plan for every state and opens a ticket on exit code 2. The output is your compliance evidence that reality matches reviewed code, and it catches the console click someone made "just to fix prod quickly". State corruption recovery: the bucket is versioned, so you download the last good version, verify it with terraform show, terraform state push it back, and run a plan to confirm zero changes. Never hand-edit the JSON; a wrong serial or lineage makes Terraform refuse it, and a subtle typo makes it plan a destroy.

Two flags are legitimate but deliberate. terraform apply -replace='aws_instance.bastion["a"]' forces one resource to be recreated (the modern taint), right for a node with a corrupted disk. terraform apply -target=module.vpc applies part of the graph, right for recovering from a partial apply and wrong as a daily workflow, because state is inconsistent until a full apply runs. Either one in prod goes in the change ticket.

Modules: how you stop copy-pasting

A module is a directory of .tf files called from elsewhere with inputs and outputs. A root module is the directory you run apply in; it owns a state. A child module is reusable and owns nothing until a root module calls it. The standard layout:

modules/eks-cluster/
├── main.tf          # resources and module calls
├── variables.tf     # typed inputs with descriptions and validation
├── outputs.tf       # what callers are allowed to depend on
├── versions.tf      # required_version + required_providers (never a provider block)
├── README.md        # generated by terraform-docs
├── examples/
│   └── nonprod/     # a runnable root module used by terraform test and by humans
└── tests/
    └── eks.tftest.hcl

nonprod/eks/cluster/    # root module: backend.tf, providers.tf, main.tf, nonprod.auto.tfvars
prod/eks/cluster/       # same shape, different variables and a different pipeline role

Child modules never contain provider blocks; the root configures providers and passes them down. Modules are published by Git tag or to a private registry (HCP Terraform, Artifactory, GitLab), and callers pin with semantic versioning: version = "~> 2.3" accepts 2.3.x through 2.x but not 3.0, where breaking changes live. Bumping the pin is a PR whose plan shows exactly what changes.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"           # public registry; pin the major, review the CHANGELOG on bumps
  # ...
}

module "cluster_baseline" {
  source = "git::ssh://git@git.bank.internal/platform/terraform-modules.git//eks-baseline?ref=v2.3.1"
  # ...
}

On public modules: terraform-aws-modules/vpc/aws and terraform-aws-modules/eks/aws are excellent and you should say so. Then the caveats a bank cares about: they are big (the EKS module creates dozens of IAM, security group and KMS resources with defaults you must read), their majors carry breaking changes (v20 replaced aws-auth ConfigMap management with access entries; v21 dropped the cluster_ prefix from many inputs and requires AWS provider 6.x), and a regulated environment usually cannot pull from the public registry at apply time, so you mirror them and pin exact versions. Most platform teams wrap the public module in a thin internal module that hard-codes the bank's standards (private endpoint, KMS, logging on, approved instance types) and exposes a handful of inputs. That wrapper is the "reusable pattern" in the JD.

Layout patterns compared

A complete EKS example, condensed

This is the whiteboard answer to "how would you build an EKS cluster with Terraform", cut to what fits in an interview but structurally real. It builds on Post 27 (networking and IAM) and Post 28 (operations). The root module is nonprod/eks/cluster/ with the S3 backend shown earlier.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"

  name = local.name
  cidr = "10.10.0.0/16"
  azs  = ["ca-central-1a", "ca-central-1b", "ca-central-1d"]

  private_subnets = ["10.10.0.0/20", "10.10.16.0/20", "10.10.32.0/20"]
  public_subnets  = ["10.10.100.0/24", "10.10.101.0/24", "10.10.102.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = var.environment != "prod"   # cost in nonprod, HA in prod

  # The AWS Load Balancer Controller discovers subnets by these tags
  public_subnet_tags  = { "kubernetes.io/role/elb" = 1 }
  private_subnet_tags = { "kubernetes.io/role/internal-elb" = 1 }

  tags = local.tags
}

In a real bank the VPC usually comes from the network team's state via the data-source lookup from earlier; it is inlined here to keep the example self-contained. Now the cluster: private endpoint, KMS envelope encryption of Secrets, all five control plane log types, access entries instead of the old aws-auth ConfigMap, and a managed node group for system workloads (application capacity comes from Karpenter, covered in Post 28).

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = local.name
  kubernetes_version = "1.33"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.private_subnets

  endpoint_private_access = true
  endpoint_public_access  = local.public_api        # false in prod
  enabled_log_types       = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  create_kms_key = true                            # envelope-encrypts Secrets in etcd
  enable_irsa    = true                            # creates the OIDC provider for IRSA

  authentication_mode                      = "API"  # access entries only, no aws-auth
  enable_cluster_creator_admin_permissions = false  # the pipeline role is NOT a standing admin
  access_entries = {
    platform_admins = {
      principal_arn = "arn:aws:iam::123456789012:role/platform-eks-admin"
      policy_associations = {
        admin = {
          policy_arn   = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
          access_scope = { type = "cluster" }
        }
      }
    }
  }
}

Two choices there deserve a sentence each. enable_cluster_creator_admin_permissions = false means the pipeline role that created the cluster is not a standing cluster-admin; admin rights go to an explicit access entry for a role humans assume through a break-glass process, which is what an auditor wants to see. And endpoint_public_access is derived from the environment local rather than a variable, so nobody can make prod public by editing a tfvars file. The node group and add-ons continue inside the same module call; vpc-cni and the Pod Identity agent install before compute so nodes join with networking ready, and the EBS CSI driver gets its AWS permissions through EKS Pod Identity (the IRSA alternative is an OIDC-trusted role passed as service_account_role_arn).

  # still inside module "eks" { ... }
  eks_managed_node_groups = {
    system = {
      ami_type       = "AL2023_x86_64_STANDARD"
      instance_types = ["m6i.large"]
      min_size       = 2
      max_size       = 4
      desired_size   = 2
      labels         = { "node-role.bank.internal/system" = "true" }
    }
  }

  addons = {
    coredns                = {}
    kube-proxy             = {}
    vpc-cni                = { before_compute = true }
    eks-pod-identity-agent = { before_compute = true }
    aws-ebs-csi-driver     = {}
  }

  tags = local.tags

# Pod Identity: the CSI controller's ServiceAccount assumes this role, no OIDC annotation needed
resource "aws_iam_role" "ebs_csi" {
  name               = "${local.name}-ebs-csi"
  assume_role_policy = data.aws_iam_policy_document.pod_identity_trust.json  # trusts pods.eks.amazonaws.com
}
resource "aws_iam_role_policy_attachment" "ebs_csi" {
  role       = aws_iam_role.ebs_csi.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}
resource "aws_eks_pod_identity_association" "ebs_csi" {
  cluster_name    = module.eks.cluster_name
  namespace       = "kube-system"
  service_account = "ebs-csi-controller-sa"
  role_arn        = aws_iam_role.ebs_csi.arn
  depends_on      = [aws_iam_role_policy_attachment.ebs_csi]   # attach before pods try to assume
}

Run the plan. The excerpt below is a healthy first plan; what you look for is 0 to destroy and that every (known after apply) is something that genuinely cannot be known yet, like an endpoint URL.

$ terraform plan -input=false -out=plan.tfplan
module.vpc.aws_vpc.this[0]: Refreshing state... [id=vpc-0a1b2c3d4e5f67890]
data.aws_iam_policy_document.pod_identity_trust: Reading...

Terraform used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.eks.aws_eks_cluster.this[0] will be created
  + resource "aws_eks_cluster" "this" {
      + name     = "bank-platform-nonprod"
      + version  = "1.33"
      + endpoint = (known after apply)
      + access_config {
          + authentication_mode                         = "API"
          + bootstrap_cluster_creator_admin_permissions = false
        }
      + encryption_config {
          + resources = ["secrets"]
        }
      + vpc_config {
          + endpoint_private_access = true
          + endpoint_public_access  = true
        }
    }

  # module.eks.module.eks_managed_node_group["system"].aws_eks_node_group.this[0] will be created
  ...

Plan: 58 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + cluster_endpoint = (known after apply)

Saved the plan to: plan.tfplan

Bootstrapping with the kubernetes and helm providers, and the classic pitfall

Once the cluster exists you want the AWS Load Balancer Controller and Argo CD on it, and the tempting move is to add the kubernetes and helm providers to the same root module. Here is what that looks like, then why most teams split it out.

data "aws_eks_cluster_auth" "this" {   # short-lived token from your AWS identity
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {                       # helm provider 3.x uses attribute syntax (=), 2.x used blocks
  kubernetes = {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    exec = {                            # re-mints a token on every call; survives long applies
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
    }
  }
}

resource "helm_release" "aws_lb_controller" {
  name       = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"   # mirrored internally in a bank
  chart      = "aws-load-balancer-controller"
  namespace  = "kube-system"
  version    = "1.13.4"                              # pin the chart, always

  set = [                                            # helm 3.x: list of objects; 2.x used set {} blocks
    { name = "clusterName",         value = module.eks.cluster_name },
    { name = "serviceAccount.name", value = "aws-load-balancer-controller" },
    { name = "vpcId",               value = module.vpc.vpc_id },
    { name = "region",              value = var.region },
  ]
  depends_on = [aws_eks_pod_identity_association.lb_controller]
}

resource "helm_release" "argocd" {
  name             = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  namespace        = "argocd"
  create_namespace = true
  version          = "8.2.0"
  values           = [file("${path.module}/values/argocd.yaml")]  # SSO, RBAC, HA settings
}

The pitfall: provider configuration is evaluated at plan time, and on the first run module.eks.cluster_endpoint is unknown until the cluster exists. Terraform usually copes on the first apply by deferring, but it gets fragile fast: any plan that refreshes a helm_release must reach the API server, so a private endpoint with the runner outside the VPC, a cluster mid-upgrade, or a token that expired during a 20-minute apply fails the whole plan, including the parts unrelated to Kubernetes. Worse, terraform destroy must talk to the cluster to remove the Helm releases before deleting the cluster, and if the cluster is already gone the destroy wedges. You also end up sprinkling depends_on = [module.eks] on every Kubernetes object.

So the standard pattern is two states: eks/cluster creates the cluster and publishes its name, and eks/bootstrap reads it with data "aws_eks_cluster", installs the few things a cluster needs before GitOps can take over (load balancer controller, Argo CD, perhaps a secrets operator), and stops. Everything after that is Argo CD's job, the subject of Post 30. The phrase to use in the interview: "Terraform builds the cluster and installs Argo CD; Argo CD installs everything else."

Terraform and OpenShift

OpenShift moves the Terraform boundary, because the installer and the Machine API already own most of what Terraform manages on EKS. Post 19 and Post 20 cover those components; here is how they meet Terraform.

For self-managed OpenShift (IPI or UPI on AWS, Azure, GCP or vSphere), Terraform builds the landing zone: VPC and subnets, the Route 53 zones the installer needs, the bastion, the S3 bucket for the internal image registry, VPC endpoints, KMS keys, and the IAM pieces for the Cloud Credential Operator in STS or manual mode (the OIDC provider and per-component roles that ccoctl would otherwise create). In UPI mode Terraform also builds the API and ingress load balancers and their DNS records. What Terraform must not manage is the machines: the installer creates the control plane, and workers are MachineSet objects reconciled by the Machine API operator, so an EC2 instance created by Terraform is one OpenShift cannot replace, upgrade or drain. A fact worth having ready: the installer itself embedded Terraform to build cloud infrastructure until 4.15, and from 4.16 uses Cluster API providers instead, partly because of the licence change.

For managed OpenShift the boundary moves up. ROSA (Red Hat OpenShift Service on AWS) is provisioned with the rhcs provider (terraform-redhat/rhcs) and Red Hat's modules, which create the account roles, operator roles, OIDC config and the cluster; you still bring the VPC.

terraform {
  required_providers {
    rhcs = { source = "terraform-redhat/rhcs", version = "~> 1.6" }
  }
}
provider "rhcs" {}   # authenticates with RHCS_TOKEN (OCM service account) from the pipeline's secret store

module "rosa_hcp" {
  source  = "terraform-redhat/rosa-hcp/rhcs"
  version = "~> 1.6"

  cluster_name           = "bank-rosa-nonprod"
  openshift_version      = "4.18.12"
  aws_subnet_ids         = module.vpc.private_subnets
  aws_availability_zones = module.vpc.azs
  private                = true          # PrivateLink API; no public ingress
  replicas               = 3
  compute_machine_type   = "m6i.xlarge"

  create_account_roles  = true           # ROSA HCP account/operator IAM roles
  create_operator_roles = true
  create_oidc           = true
  tags                  = local.tags
}

ARO (Azure Red Hat OpenShift) has a native resource, azurerm_redhat_openshift_cluster, where you set private API and ingress visibility, VM sizes and subnets. On GCP, OpenShift Dedicated is created through OCM and the same rhcs provider rather than a Google-native resource. In every managed case Terraform's job is the landing zone plus the cluster object; Operators, MachineConfig, SCCs and RBAC are OpenShift objects delivered through GitOps.

Ansible versus Terraform in one paragraph, since Post 32 goes deeper: Terraform is for provisioning (create, change and destroy cloud objects, with a state that knows what exists) and Ansible is for configuration (run steps against existing machines or APIs, idempotent per task but stateless overall). On an OpenShift estate Terraform builds the landing zone, the installer or ROSA builds the cluster, and Ansible or Python handles procedural day-2 work such as patching a bastion or orchestrating a maintenance window. Anything that is a Kubernetes object goes through Argo CD, which answers "would you use the kubernetes provider for namespaces and quotas": only during bootstrap, and only until Argo CD is up.

Terraform in CI/CD: the automation-first bullet

The pipeline is where "not in Terraform, not real" becomes enforceable. The stages, in order: format and static checks (terraform fmt -check, terraform validate, tflint for provider-specific lint such as invalid instance types, and a security scanner such as trivy config, the successor to tfsec, or checkov); plan on pull request, posted as a PR comment so reviewers approve a concrete diff; a manual approval gate before prod, which is the evidence attached to the change ticket; apply on merge using the exact saved plan file; and a scheduled drift job. Cloud credentials come from OIDC federation: the CI system presents a signed identity token, the cloud exchanges it for a short-lived role session, and no static access key exists anywhere. For GitHub Actions that is permissions: id-token: write plus an IAM role whose trust policy restricts token.actions.githubusercontent.com to your org, repo and branch.

name: terraform-eks-nonprod
on:
  pull_request: { paths: ["nonprod/eks/**", "modules/**"] }
  push: { branches: [main], paths: ["nonprod/eks/**", "modules/**"] }

permissions:
  id-token: write        # OIDC token for AWS; no static keys
  contents: read
  pull-requests: write   # to post the plan as a comment

jobs:
  plan:
    runs-on: ubuntu-latest
    defaults: { run: { working-directory: nonprod/eks/cluster } }
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-terraform-nonprod-plan
          aws-region: ca-central-1
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: 1.12.2 }
      - run: terraform fmt -check -recursive
      - run: terraform init -input=false
      - run: terraform validate
      - uses: aquasecurity/trivy-action@0.28.0
        with: { scan-type: config, scan-ref: nonprod/eks/cluster, exit-code: "1", severity: HIGH,CRITICAL }
      - run: terraform plan -input=false -out=plan.tfplan
      - run: terraform show -no-color plan.tfplan > plan.txt
      - uses: actions/upload-artifact@v4
        with: { name: plan-${{ github.sha }}, path: nonprod/eks/cluster/plan.tfplan }
      - uses: actions/github-script@v7   # post plan.txt as a PR comment
        if: github.event_name == 'pull_request'
        with: { script: "/* read plan.txt, github.rest.issues.createComment(...) */" }

  apply:
    needs: plan
    if: github.ref == 'refs/heads/main'
    environment: nonprod            # required reviewers and secrets are attached to the environment
    runs-on: ubuntu-latest
    defaults: { run: { working-directory: nonprod/eks/cluster } }
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-terraform-nonprod-apply   # broader than plan
          aws-region: ca-central-1
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: 1.12.2 }
      - uses: actions/download-artifact@v4
        with: { name: plan-${{ github.sha }}, path: nonprod/eks/cluster }
      - run: terraform init -input=false
      - run: terraform apply -input=false plan.tfplan   # the same plan reviewers approved

Three details make this bank-grade. Plan file promotion: the apply job applies plan.tfplan from the plan job, not a fresh plan, so what was reviewed is what runs, and if anything changed in between Terraform refuses with a stale-plan error. Two roles: the plan role is read-only plus state access, the apply role can write, and only the main branch job assumes it. And the environment: key carries required reviewers, a wait timer and branch restrictions, so prod cannot be applied without a named second person, and GitHub's deployment log becomes CAB (change advisory board) evidence.

The same shape in Jenkins, which many banks still run on-premises. No OIDC handshake is needed because the agent runs with an instance profile (or a Pod Identity role if agents run on EKS), and the approval is an input step restricted to a group.

pipeline {
  agent { label 'terraform' }                       // agent has an IAM role; no static keys
  options { disableConcurrentBuilds() }
  environment { TF_IN_AUTOMATION = 'true'; AWS_REGION = 'ca-central-1' }
  stages {
    stage('Validate') { steps { dir('prod/eks/cluster') {
      sh 'terraform fmt -check -recursive && terraform init -input=false && terraform validate'
      sh 'tflint --recursive && checkov -d . --quiet --framework terraform'
    } } }
    stage('Plan') { steps { dir('prod/eks/cluster') {
      sh 'set -o pipefail; terraform plan -input=false -out=plan.tfplan | tee plan.txt'
      archiveArtifacts artifacts: 'prod/eks/cluster/plan.txt'
    } } }
    stage('Approve') { when { branch 'main' } steps {
      timeout(time: 4, unit: 'HOURS') {
        input message: 'Apply this plan to PROD?', submitter: 'platform-leads',
              parameters: [string(name: 'CHANGE_TICKET', description: 'CHG number')]
      }
    } }
    stage('Apply') { when { branch 'main' } steps { dir('prod/eks/cluster') {
      sh 'terraform apply -input=false plan.tfplan'
    } } }
  }
}

Alternatives worth naming: Atlantis is a self-hosted bot that plans on every PR and applies on a comment; HCP Terraform and Terraform Enterprise run plans remotely, store state, enforce Sentinel policies and keep the audit trail, with Enterprise being the on-premises choice; Spacelift and env0 are commercial equivalents with stack dependencies and OPA policies built in. Whatever the tool, the interviewer wants the same five stages, plan file promotion, OIDC credentials and a human gate before prod.

Policy as code and testing

Policy as code = rules about what infrastructure is allowed, evaluated automatically against the plan before apply. Checkov and Trivy ship hundreds of built-in rules that match a bank's baseline: no public S3 buckets, EBS volumes encrypted, no 0.0.0.0/0 on port 22, EKS control plane logging on, KMS keys rotated. For rules specific to your organisation you write your own with OPA (Open Policy Agent) and evaluate them with conftest against the JSON form of the plan, or with Sentinel on HCP Terraform or Enterprise.

$ terraform show -json plan.tfplan > plan.json
$ conftest test plan.json -p policy/
FAIL - plan.json - main - EKS cluster "bank-platform-prod" must not expose a public API endpoint

1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions

# policy/eks.rego
package main

deny contains msg if {
  rc := input.resource_changes[_]
  rc.type == "aws_eks_cluster"
  rc.change.after.vpc_config[0].endpoint_public_access == true
  endswith(rc.change.after.name, "-prod")
  msg := sprintf("EKS cluster %q must not expose a public API endpoint", [rc.change.after.name])
}

Testing is a separate concern: policy asks "is this change allowed", tests ask "does this module do what it promises". Terraform 1.6 added native terraform test, which runs .tftest.hcl files that plan (or apply) the module with given variables and assert on the result. It replaces most uses of Terratest, the Go library that applies real infrastructure and checks it with SDK calls, which you still reach for when the assertion needs a live API.

# modules/eks-cluster/tests/eks.tftest.hcl
variables {
  name        = "tftest"
  environment = "prod"
  vpc_id      = "vpc-00000000000000000"
  subnet_ids  = ["subnet-0000000000000000a", "subnet-0000000000000000b"]
}

run "prod_has_private_endpoint_only" {
  command = plan
  assert {
    condition     = output.endpoint_public_access == false
    error_message = "prod clusters must not have a public API endpoint"
  }
}

run "all_control_plane_logs_enabled" {
  command = plan
  assert {
    condition     = length(output.enabled_log_types) == 5
    error_message = "all five control plane log types must be enabled"
  }
}

Round it out with pre-commit hooks (the antonbabenko/pre-commit-terraform collection runs fmt, validate, tflint, trivy and terraform-docs on every commit) and terraform-docs, which regenerates each module's README inputs and outputs tables so documentation cannot drift from code. When the JD says "platform standards, docs, reusable patterns", this tooling is the concrete answer.

Secrets and sensitive values

The rules, in the order an interviewer expects them. Never commit a .tfvars file containing a secret; .gitignore *.tfvars except the non-secret *.auto.tfvars you deliberately check in. Read secrets at plan time from the bank's secret store with a data source, so the value flows from Vault or Secrets Manager straight into the resource that needs it. Mark variables and outputs sensitive = true so they are redacted in logs. And repeat the state warning: any value a resource stores, including the secret you fetched, lands in state in plain text, so the state bucket is protected like a secret store and read access to state is itself a privileged permission.

data "aws_secretsmanager_secret_version" "argocd_oidc" {
  secret_id = "platform/argocd/oidc-client-secret"   # still lands in state as an attribute
}

data "vault_kv_secret_v2" "registry" {
  mount = "platform"
  name  = "quay/pull-credentials"
}

# Terraform 1.10+: an ephemeral value is used during the run and never written to plan or state
ephemeral "aws_secretsmanager_secret_version" "db_master" {
  secret_id = aws_secretsmanager_secret.db_master.id
}

resource "aws_db_instance" "audit" {
  # ...
  password_wo         = ephemeral.aws_secretsmanager_secret_version.db_master.secret_string
  password_wo_version = 1    # bump to rotate; write-only args (1.11+) are never stored
}

Present the last part carefully. Ephemeral values (1.10) and write-only arguments (1.11, the _wo suffix) finally keep some secrets out of state, but only where the provider and attribute support them, so say "newer versions add ephemeral resources and write-only arguments that keep the secret out of state where supported, and I treat state as sensitive regardless". For secrets that live inside the cluster, Terraform should not be the delivery path at all; Post 6 covered the objects and Post 31 covers External Secrets Operator and Vault integration.

Operating Terraform at scale in a bank

Everything above is per-repository mechanics. A platform interview also probes how you run Terraform across an estate.

Repository strategy. A platform-infra monorepo with modules/, one directory per environment per component, and a pipeline that only plans the directories a PR touched is the common shape; some banks keep modules in their own repo so they release independently. Ownership lives in CODEOWNERS, and branch protection turns that file into a hard gate.

# .github/CODEOWNERS
/modules/eks-cluster/     @bank/platform-eng-maintainers
/modules/vpc-baseline/    @bank/network-eng
/nonprod/                 @bank/platform-eng
/prod/                    @bank/platform-eng-leads @bank/cloud-security-controls

Versions. Pin the binary with required_version and a .terraform-version file read by tfenv (or tenv, which also handles OpenTofu) so laptops and runners agree. Provider versions are locked in the committed .terraform.lock.hcl; run terraform providers lock -platform=linux_amd64 -platform=darwin_arm64 so it carries hashes for both runners and developer machines. Upgrades are a PR: terraform init -upgrade updates the lock file, the plan shows whether the new provider changes anything, and the reviewer reads the changelog. Major bumps of the EKS or VPC module get their own PR, applied in nonprod first, with the drift job confirming a clean plan before prod. The rule for breaking changes matches Kubernetes upgrades in Post 28: read the changelog, one major at a time, nonprod first, keep the plan file as evidence.

Module release process. A module change is a PR with tests; merging tags a semantic version; consumers bump the pin when ready, which lets prod stay on v2.3 while nonprod proves v3.0. Monthly drift review turns recurring drift into a code fix (add ignore_changes, or move the attribute into code) or a conversation with whoever is clicking. Disaster recovery is "apply from git into the DR account": network and IAM states first, then clusters, then bootstrap, then Argo CD reconciles the workloads, rehearsed regularly because a DR plan that has never been applied is a document, not a plan.

Migrating console-created resources is a project every bank has run. import blocks make it bulk-friendly: write the blocks (or generate them from an inventory script), run terraform plan -generate-config-out=generated.tf to draft configuration, tidy it into modules, and apply with the plan showing only imports. Infracost adds a "this PR changes monthly spend by $412" comment, which is not a security control but is what makes finance trust the platform team.

$ cat import.tf
import { to = aws_security_group.bastion,  id = "sg-0123456789abcdef0" }
import { to = aws_s3_bucket.registry,      id = "bank-platform-registry-ca-central-1" }
import { to = aws_kms_key.registry,        id = "2f1a7c4e-9b3d-4f6a-8e2b-1c5d7a9f0b3e" }

$ terraform plan -generate-config-out=generated.tf
...
Plan: 3 to import, 0 to add, 0 to change, 0 to destroy.

$ infracost breakdown --path . --format table
Project: nonprod/eks/cluster
 Name                                          Monthly Qty  Unit    Monthly Cost
 module.eks.aws_eks_cluster.this[0]                    730  hours         $73.00
 module.eks...aws_eks_node_group.this[0]  (2x m6i.large)  1,460  hours        $140.16
 module.vpc.aws_nat_gateway.this[0]                    730  hours         $32.85
 OVERALL TOTAL                                                             $296.21

Troubleshooting Terraform

Same habit as every troubleshooting post in this series: read the error, identify the layer (backend, provider auth, graph, state, API), then apply the smallest safe fix. These are the failures that appear in scenario questions.

$ terraform apply -input=false plan.tfplan
Error: Error acquiring the state lock

Error message: operation error S3: PutObject, https response error StatusCode: 412,
  api error PreconditionFailed: At least one of the pre-conditions you specified did not hold
Lock Info:
  ID:        6f0f2a1e-3c8b-4d7a-9e21-5b0c4f8d2a11
  Path:      bank-platform-tfstate-ca-central-1/nonprod/eks/cluster/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@gh-runner-7f9c
  Created:   2026-09-08 14:02:11 +0000 UTC

$ terraform force-unlock 6f0f2a1e-3c8b-4d7a-9e21-5b0c4f8d2a11   # only after confirming that run is dead

• State lock stuck. A runner died mid-apply and the lock stayed. Confirm the owning job is really dead (Who and Created tell you), then force-unlock with the exact ID. If the job is still running, you are about to corrupt state; wait.
• "resource already exists" on create. Someone created it by hand, or a previous apply created it and died before writing state. Do not delete it in the console; add an import block and re-plan.
• Provider authentication errors (ExpiredToken, AccessDenied on sts:AssumeRoleWithWebIdentity). Check the OIDC trust policy's sub condition matches repo and branch, that the session duration covers a long apply, and that the plan role is not being used for an apply.
• Cyclic dependency (Error: Cycle: aws_security_group.a, aws_security_group.b). Two security groups reference each other inline. Break the cycle with standalone aws_vpc_security_group_ingress_rule resources.
• count index shift planning destroys you did not ask for. Stop, add moved blocks or convert to for_each, and never apply a plan whose destroy count surprises you.
• "Kubernetes cluster unreachable" or "dial tcp: connection refused" from the kubernetes or helm provider. The runner cannot reach a private endpoint, the cluster is upgrading, or the token expired. The structural fix is the separate bootstrap state and a runner inside the VPC.
• Plan shows changes on every run. Find the attribute in the diff. If another system writes it (a scanner, AWS Backup, a cost tool), add it to ignore_changes. If the provider is normalising input (case, a reordered JSON policy), fix the input to match the canonical form.
• Timeouts. EKS clusters take 10 to 15 minutes and node groups a few more; some resources need a timeouts { create = "30m" } block, and the CI job's own timeout must be longer than Terraform's.
• Partial apply. An apply that fails at resource 40 of 58 has recorded 1 to 39 in state. Fix the cause and re-run a full plan and apply; Terraform continues from where it stopped. Use -target only if one broken resource blocks the rest, and follow it with a full apply.

Likely interview questions

Practise these out loud in the 30 to 60 second form. Each answer is the shape of a strong response, not a script.

Next up: Terraform has handed you a cluster with Argo CD on it, so Post 30 covers CI/CD for containers and GitOps with Argo CD, from image build and scan through ApplicationSets, sync waves and the promotion model a bank will accept.