Designing a Terraform execution system

A project factory can turn a harmless-looking file rename into a plan to destroy a production project. If the filename supplies the for_each key, Terraform treats the new name as a new resource identity. A reviewer sees a rename in Git; the plan shows a deletion and a replacement, potentially including the bucket holding the project's state.

I want infrastructure review to make that consequence visible before anyone merges. I also want to know which identity will execute the change, who authorised it and whether it ever reached the cloud. Those decisions need to be part of the system, so they don't depend on whoever happens to run Terraform that day.

This is the execution system I'd design around Terraform on GitHub Actions, with Google Cloud as the target. It's an architecture proposal, rather than a complete workflow to copy. The cloud authentication would differ on AWS or Azure, but the questions about review, approval and execution would remain.

What running Terraform centrally should solve#

Moving terraform apply from a laptop into CI gives you a consistent place to run it. That helps, but the same job can still hold an administrator key, apply an old commit or execute changes nobody reviewed.

The design has to solve the operational problems that made people depend on those laptops in the first place:

The result should let an engineer propose and ship infrastructure through the normal review process, without borrowing someone's credentials. It should also let an operator answer a less comfortable question: what failed to apply, and what does the system intend to do next?

The stack is the unit of execution#

A stack is one Terraform root directory with its own state and execution policy. It gets a stable identifier, a planning identity, an applying identity and an approval rule. These need to agree: separate directories are little protection if they write the same state or authenticate as the same administrator.

I'd keep that metadata in a small registry. This is an illustrative schema, with placeholder accounts and approvers; it isn't a configuration format supplied by Terraform:

stacks:
  networking/prod:
    inputs:
      - stacks/networking/prod/**
      - modules/networking/**
    plan_identity: tf-networking-prod-plan@<project>.iam.gserviceaccount.com
    apply_identity: tf-networking-prod-apply@<project>.iam.gserviceaccount.com
    approval:
      required: true
      approvers: [alice, bob]
  cache/dev:
    inputs:
      - stacks/cache/dev/**
      - modules/cache/**
    plan_identity: tf-cache-dev-plan@<project>.iam.gserviceaccount.com
    apply_identity: tf-cache-dev-apply@<project>.iam.gserviceaccount.com
    approval:
      required: false

Changed paths select affected stacks, which become independent jobs. The important detail is inputs: a shared module can affect several roots without changing a file inside any of them. Variables, provider lock files and execution policy also need to participate in change detection. If I can't establish the dependency set reliably, I'd plan more stacks rather than silently leave one out.

A malformed registry aborts the run. An unknown stack or a production approval rule with no approvers is an error, and a failed diff must never become an empty job matrix. Those cases otherwise produce a reassuring green check without doing the work.

State boundaries also don't make cross-stack changes atomic. If a stack needs outputs from a network stack, their ordering has to be declared or the changes delivered separately. I'd keep the first implementation to independently executable stacks; adding dependency scheduling would be a deliberate extension.

Follow a change from review to execution#

On a pull request, the system plans each affected stack and puts the result beside the review. Those plans are previews: they help reviewers decide whether the code should merge.

After merge, it records the desired revision for each affected stack and produces a plan from that revision. A development stack can execute through its configured automatic path. A production stack creates an approval request containing the plan summary and a reference to the saved plan. Execution waits for an authorised decision.

I'd distinguish these states explicitly:

A pull request can affect several stacks and leave them in different states. One may apply successfully while another fails or waits for approval. A single green workflow badge can't describe that outcome, so each stack needs its own status and a link to its execution record.

The durable record is what makes this an execution system. Workflows come and go; the desired revision, approval and result have to remain understandable after a runner exits.

Plans are evidence for review#

I'd publish one plan comment per stack, updating it on each push with a stable key. tfcmt supports patching plan comments, so a reviewer can find the current result without scrolling through a history of obsolete ones. The comment needs to say which revision was planned and whether planning completed.

In the project-factory example, the plan is where the rename becomes visibly destructive. I'd expect the reviewer to account for every destroy and replacement. I'd also fix the resource addressing: a stable project ID should supply the for_each key, so changing a filename doesn't change the resource's identity. Review is useful, but a design that prevents an accidental deletion is better than repeatedly asking people to spot it.

A plan still has limits. It can include existing drift, and a successfully calculated change can fail against quotas or API validation during apply. A pure address refactor should leave no infrastructure changes once its moved blocks are correct; an unexpected replacement needs an explanation before merge.

Planning also crosses a trust boundary. Providers execute, and an external data source can run a program named by the configuration. I'd give planning jobs only the cloud reads they need and the backend permissions necessary for state access and locking. That's a narrower identity than the one that can create or delete infrastructure, although state itself can contain secrets.

I'd use the pull_request event for the unmerged code. Running a fork's Terraform under pull_request_target would execute it in the privileged context of the base repository. Fork contributions need a separate, explicit policy for credential-free validation and trusted planning; enabling cloud access just to make their checks green would defeat the boundary.

Temporary credentials need a narrow trust policy#

The Google Cloud design assumes the organisation policy blocking service-account key creation is enabled. Jobs authenticate through Workload Identity Federation: GitHub issues an OIDC token, Google verifies its claims, and an authorised principal impersonates the stack's service account. The job obtains temporary credentials without a stored service-account key.

There are two separate permissions to get right. The service account's roles determine what it can change. The federation and impersonation policy determine who can become that service account.

That second part is easy to miss. A registry entry saying “use the development account” is only an instruction to the workflow. If the same caller is allowed to impersonate the production account, changing that entry can select production instead. Account separation only limits access when the cloud trust policy enforces the caller restrictions as well.

I'd restrict the federation provider to the expected repository and owner IDs, then constrain production impersonation to the intended protected execution context. Workflow definitions, the registry and the helper that validates approvals are consequently privileged code and need protected review. An environment name in YAML doesn't create a human approval requirement unless that environment actually has protection configured.

This adds setup work. Each stack needs its accounts, permissions and trust bindings before its first cloud plan succeeds. A failed authentication step may be a missing binding rather than a cloud outage, so stack onboarding should check those prerequisites and explain the failure. In return, I can inspect both what an apply may change and which jobs may authorise that access.

Decide what production approval authorises#

The pull-request plan and the production plan serve different decisions. The first supports code review. The second is the concrete change an operator is being asked to execute against the current infrastructure.

For production, I'd save the post-merge plan and bind approval to that artifact, its stack and its full commit SHA. The execution worker would run terraform apply with the approved plan file, rather than calculate a different set of operations after approval.

I'd commit provider lock files, pin module versions and record the variable inputs and toolchain used to produce each plan. That is stricter than approving code and planning again on resume. Replanning can be a reasonable policy for development, but it authorises whatever the later calculation produces. Even with identical code, changes to variable values, provider behaviour or data sources can change the result. Drift is only one possible cause.

A saved plan brings its own costs. Plan files can contain sensitive values, so the artifact store needs restricted access and a defined retention period. The approval should identify an immutable artifact and its digest, rather than a filename someone can overwrite. If the state has advanced or the artifact expires, I'd generate a new plan and request new approval.

Nor does saving a plan freeze the cloud while an approver is away. An old plan can describe operations that are no longer desirable even if nobody has written new Terraform state. I'd set an approval lifetime and return to planning when an incident or known external change invalidates its assumptions. The promise is that the worker executes the authorised plan, not that every operation will succeed.

An environment gate or an approval issue#

GitHub environments are the simpler starting point where the repository's plan supports the required protection rules. Approval happens before the job runs; it doesn't require a runner to sit polling for a person. I'd use that gate if its reviewer policy and deployment record meet the team's needs.

An issue-based gate is worth building when the approval needs to live beside a longer discussion or follow rules the environment gate can't express. In that design, the merge workflow plans, creates the request, notifies approvers and exits. A later issue_comment event wakes an execution workflow.

The issue is a useful interface, but its editable body can't be the authoritative execution request. I'd have the workflow create a separate record tying the request ID to the stack, revision and plan artifact. A comment selects that existing record; it can't supply an arbitrary account, directory or command.

Before acquiring write credentials, the worker checks that the request exists, remains open and current, and that the commenter is an allowed approver under the trusted policy. Validation and claiming happen atomically under the stack's execution control. That claim marks the start of execution and prevents duplicate comments from applying the request twice. A rejection closes the request's executable state; a notification failure must not make a rejected request runnable again.

This is enough logic to belong in a tested helper package. YAML should connect events to that package, rather than hide the approval rules inside shell expressions.

A rerun must not restore an old revision#

GitHub reruns use the original run's SHA and ref. Suppose revision A applied, then B added a resource. Rerunning A and creating a fresh plan from its old configuration can delete B's resource while reporting success.

I'd check freshness after the job has entered the stack's execution slot and before it obtains write credentials. The request uses a full commit SHA, the checkout must match it, and any failure to establish the current revision refuses execution.

Comparing with the tip of main is a conservative first implementation, but it also invalidates approvals when an unrelated stack changes. A per-stack revision is more useful once change detection covers all relevant inputs, including shared modules and execution policy. Looking only at the most recent edit inside the stack directory would miss those dependencies.

The check needs a defined moment. If a new revision arrives while an authorised apply is already running, I'd let that apply finish and schedule the newer request next. Canceling a writer midway is a poor substitute for ordering. The worker must refuse a request that was superseded before the worker validated and claimed it; it can't promise that the repository will stop changing during the apply.

Older workflows also need attention when introducing the guard. A rerun of a historical workflow doesn't acquire safety logic merely because that logic was added later. I'd retire rerunnable jobs that predate the control as part of the rollout.

Scheduling and state locking solve different problems#

Every workflow that writes a stack should use the same concurrency group. That includes automatic applies, approved applies and recovery attempts. I'd avoid canceling an in-progress apply.

GitHub's concurrency configuration now supports queue: max, allowing up to 100 pending jobs in a group. The default keeps one pending job and replaces it when another arrives. Neither option establishes which revision remains authorised when a waiting job eventually starts, so the freshness check still belongs inside the execution slot.

More importantly, a canceled event must not erase pending work. If merge A changes networking and merge B changes cache, scheduling only B's diff doesn't recover A's canceled networking job. I'd record desired and successfully applied revisions per stack, then reconcile unfinished work independently of the latest event. Events tell the system to look; they aren't the only record that something needs doing.

The remote backend provides the other control. The GCS backend supports state locking, which protects Terraform state against concurrent writers, including writers outside GitHub. It doesn't validate an approval or choose the next revision. I'd enable object versioning for state recovery, while recognising that restoring a state object doesn't restore the infrastructure it describes.

An apply can also fail after changing some resources. Terraform doesn't automatically roll back a partial apply. The next attempt needs a fresh view of the state and, for production, a newly reviewed recovery plan. Force-unlocking is a recovery action after confirming no writer remains, rather than an automatic response to contention.

Drift and failed reporting need their own outcomes#

A scheduled job plans every stack using the planning identity. With -detailed-exitcode, Terraform returns 0 for no changes, 1 for an error and 2 for a non-empty diff.

I'd report 2 as a difference to investigate. It might be drift, a merged change waiting to apply or an input that isn't stable. The desired/applied revision record helps distinguish those cases. An authentication error is a failed check, not evidence that a stack is in sync.

The scheduled job reports rather than repairs. An operator may have changed infrastructure to contain an incident; reverting that automatically can undo the fix. Investigation should lead to a reviewed change, an authorised recovery apply or an explicit decision to preserve the live behaviour.

Reporting after execution needs similar care. Terraform can succeed while the GitHub status update fails. I'd preserve the execution result first and retry the report separately, so a missing comment doesn't cause the worker to apply again. The record should retain the stack, revision, artifact, approver where required, timestamps and run outcome. That's enough to trace a change and identify missing evidence, without claiming it satisfies every audit requirement.

Prove the controls before giving them production access#

The first implementation I'd build is one fixture stack and a helper package that handles registry validation, affected-stack selection, approval validation and freshness. The fixture should manage disposable resources so the execution path can be rehearsed without risking production.

I'd test the refusals as carefully as the successful path: missing metadata, a shared-module edit, an unauthorised commenter, an edited issue, duplicate approval, an expired artifact and a superseded revision. Mutation testing is useful here because flipping an approver comparison should make a test fail.

GitHub's issue_comment workflow must exist on the default branch, so a branch test isn't a full rehearsal of the event path. Before using it for production, I'd merge and exercise the gate against that fixture, including a partial failure and a failed status update.

I'd consider a managed Terraform execution service before maintaining this much orchestration myself. GitHub Actions makes sense when the team wants this review surface and can own the scheduler and approval logic. The point at which cancellation, dependency scheduling or artifact lifecycle becomes a second platform to operate is a reason to reconsider that choice.

My acceptance check would be a deliberately hostile sequence: approve revision A, supersede it with B, then try to execute A again. A must refuse before acquiring write credentials, B must remain visible as unfinished work, and the operator must be able to tell why neither approval nor a green merge meant the change had applied.