An agent can write an infrastructure change and then produce a confident report that it worked. The report may cite a green deployment or a healthy controller without showing that the intended change reached a caller.
I measured a workflow for infrastructure changes against nine real merged pull requests. Without it, agents produced a usable validation plan for one of the nine. With it, they produced one for all nine. That answered whether the workflow could make agents write checks. It didn't answer whether those checks could catch a bad change.
Most of the pass criteria agents wrote could still appear satisfied when the underlying change was deliberately broken. I added an adversary agent to test each plan against a constructed failure. The useful idea is simple: make a check face the failure it claims to detect before trusting its green result.
A healthy system can still be running the wrong change#
Deployment systems report useful facts: a controller reconciled, resources look healthy, a job completed. Those facts don't necessarily answer the question a reviewer cares about: did this change produce the intended behavior?
A controller can be healthy while an application still serves an earlier version. A route can be accepted while the backend has no ready endpoints. A plan can render successfully even if a value in the change has no effect. These are different failure modes, but they share a problem: the signal is adjacent to the claim rather than evidence for it.
A useful validation plan starts by stating the claim in terms someone can observe. “The deployment completed” describes a process. “A request to the new route reaches the intended backend” describes a result. The first may be necessary; it does not establish the second.
The same distinction applies to infrastructure state. A successful apply tells you that the tool completed its work according to its inputs. It doesn't prove that the resulting service is reachable, that an access boundary rejects the wrong caller, or that an unrelated resource stayed unchanged. Each claim needs evidence suited to that claim.
Write the pass condition before running the check#
A validation plan should make each claim, check, expected result and responsible person visible. I use three kinds of result:
- Convergence: the automation finished and the system settled.
- Validation: the changed behavior works from a caller's point of view.
- Verification: live state matches what was declared, including the things the change was meant to leave alone.
A result should be pass, fail or unclear. “Unclear” matters. A missing metric, failed query or empty response isn't evidence of success. If a bounded retry can't resolve the uncertainty, the check has failed to establish its claim.
Here is a small illustrative plan for adding a route. The baseline response and expected statuses are examples for the scenario, not results from a live service.
| Claim | When | Check | Pass condition | Owner |
|---|---|---|---|---|
| The new route reaches its backend | After deployment | Request the route's health endpoint | The expected success status | Agent |
| Requests without a token are refused | After deployment | Call the protected endpoint without a token | The expected refusal status | Agent |
| The existing route still works | After deployment | Request the existing health endpoint | Same response as the recorded baseline | Agent |
| The render contains only the intended route change | Before merge | Compare the rendered baseline with the candidate | One added route and no unexplained changes | Executor |
| Merge and promote the change | At the gate | Human review and approval | Human decision recorded | Human |
The expected result belongs in the plan before anyone runs the check. Otherwise, it's easy to reinterpret an inconvenient result after seeing it. A status code, digest, exact string or diff can often be scored mechanically. If a row requires judgment, make the judgment explicit or rewrite the row so another person can evaluate it.
The first functional check should be sensitive to the change. If the new route already passes before the change, it cannot show that the change made it work. Likewise, a render command that exits successfully only shows that the renderer accepted its inputs. Comparing baseline and candidate output can show whether the intended resource appeared, or whether a refactor left the output unchanged.
For access changes, check both sides of the boundary. Confirm that the intended caller succeeds and that an unintended caller is refused. A successful authorized request alone says nothing about the fence.
It also helps to label where evidence came from. A value declared in source control, a current observation from the control plane and a historical observation are not interchangeable. A current health check cannot prove that an earlier change worked, and an old observation cannot establish today's state. Recording the source and time makes the limits visible to the next person who reads the result.
Challenge the plan with a broken version#
A check can look precise and still miss the defect. The best way I found to expose that weakness was to imagine a concrete world in which the change is wrong, then ask whether each proposed check would fail there.
For a route change, suppose the deployment finishes but traffic still reaches the old backend. Would the plan detect that? For a secret update, suppose the application continues to read the previous value. Would any check notice? For an alert, suppose the rule parses but can never fire. Does the plan test its behavior or only its syntax?
If the answer is no, the check needs repair before it can support a pass. A generic “controller is healthy” row doesn't become stronger because it is repeated in several places. The check must observe the property in the claim.
I added an adversary agent to do this systematically. It takes a proposed plan, builds a concrete failure scenario and attacks each row with one question: would this check fail in that scenario? If not, the row returns for revision. The adversary reviews the repaired version too. The purpose is not to invent increasingly elaborate failure stories; it's to test whether the evidence distinguishes the intended result from a plausible wrong one.
The method is useful beyond infrastructure. A test for a data migration should face a case where records are lost or misclassified. A permission check should face an unauthorized caller. A monitoring change should face the condition it is meant to detect. The broken case should come from the actual claim and its risks, not from a generic checklist.
Match the effort to the risk#
Not every change needs the same depth of review. A small reversible change with no traffic or data impact can use a short plan. Changes involving identity, persistent data, shared dependencies or quiet failure need more evidence and a human decision before mutation.
I find it useful to ask one question about reversibility: if this change is wrong, does undoing it restore the behavior, or only the source file? A configuration rollback may not restore deleted data. A route rollback may not restore a caller that has already switched endpoints. If the answer is hard to explain, the change deserves a more careful plan.
The plan should also say what must not change. A change that adds a route may accidentally remove a different one; a migration can alter records outside its intended scope. Those are separate claims with separate checks. A tidy diff isn't evidence that unrelated behavior stayed intact.
This is where human review remains valuable. The agent can propose checks and run bounded observations, but a reviewer should decide whether the plan covers the likely failure modes and whether the change is acceptable. For high-impact work, that review should happen before the action that changes production state.
Separate planning, execution and scoring#
An agent that proposes a change should not be the sole judge of whether the change worked. Planning, execution and scoring ask for different kinds of judgment.
The planner describes the intended result and writes the checks. An executor runs those checks and returns the command, status and raw output. A reviewer scores the evidence against the pass conditions. Keeping those jobs distinct makes it harder for an explanation to substitute for a result.
The executor should not be able to turn a failed command into a pass by interpreting its output. Nor should a checker that only has to inspect evidence be able to modify the system it is checking. These boundaries belong in the tools and permissions available to each role, not only in instructions asking an agent to behave well.
A read-only role is a useful starting point for observation. It should have only the access needed to inspect the relevant state. A separate role can make changes when the workflow and human approval allow it. This limits the consequences of a mistaken command and makes the division of responsibility easier to inspect.
The principle applies even when people run the tools. A command that reports “no difference” is only useful if the query completed and covered every target. A check that cannot see a resource should report that it could not inspect it, rather than silently treating absence of data as absence of a problem.
Keep the result useful after the session ends#
Infrastructure changes often outlast the session that proposed them. A reviewer may approve a plan, a deployment may finish later, and another person may need to determine what happened. The record should connect the claim, the check, the observed result and the revision that was evaluated.
Durable evidence can be reused for the same recorded revision. A live response or current permission check can go stale, so it should be rerun when another session resumes the work. If a comparison window was missed, the honest result is that the change cannot be attributed from the available evidence. Current health is not proof of historical success.
A single green workflow badge can hide partial outcomes when a change touches several independent parts. Keep status close to the unit being changed, and make a failed or pending result visible instead of letting another successful job cover it. The record is part of the safety mechanism: it gives the next person a place to continue without guessing what was already checked.
The diagram below shows an illustrative sequence for a higher-risk change: challenge the proposal, write and adversarially review the plan, get human approval, run the checks, and have read-only roles inspect the result. It describes a prospective method, not a map of a particular production workflow.
Illustrative design: adversarial planning, human gates, raw evidence and separate read-only checks.
Full diagram description
Illustrative sequence for a higher-risk change: an author writes a plan, an adversary challenges it, a human approves it, an executor runs checks, and read-only reviewers inspect the result.
I evaluated the workflow against nine real merged infrastructure pull requests. It raised usable validation plans from one of nine unaided runs to nine of nine with the workflow. Most agent-written pass criteria could still appear to pass when the underlying change was deliberately broken, so I added an adversary agent to test each plan against constructed failure scenarios. Humans still perform every merge, apply and promotion. The shared toolkit contains seven skills and four evaluation suites.
The lesson I keep is narrower than “agents can validate infrastructure.” They can help make a plan and gather evidence. The plan still needs to face a plausible failure, the evidence needs an owner, and a human needs to decide when the risk warrants approval. A check earns trust when it would notice the failure it claims to prevent.