An AI coding agent has finished a change, and a test command is red. The next move is not automatically “fix the test” or “run it again.” First find out whether the same test passed before the change, whether the command ran in the same environment, and whether the failure is connected to the edited code.
This article was prepared with AI writing assistance. I checked the product statement against Orke’s current sessions guide (https://orke.bomtobe.com/en/docs/orke/sessions/). BOMTOBE makes Orke, a desktop workspace for AI agents such as Claude Code and Codex. Orke brings terminal work, a web window, and Git into one workspace. The baseline-first method below is a proposed general workflow, not a measured experiment or a live Orke demonstration.
Section — Why one red test is not a diagnosis
A failing test is evidence that a particular command produced a particular result in a particular environment. By itself, it does not establish when the failure began or which change caused it.
The test may have failed on the starting revision. A dependency, browser, service, or environment variable may differ from the expected setup. The failure may be intermittent. The agent may have changed the relevant behavior, or it may have changed an unrelated file while a separate test was already broken. And a test that passes once after a retry does not prove the code is correct.
The useful comparison is between a known starting state and the proposed change under as similar conditions as you can manage. If there is no baseline, say so. Do not turn uncertainty into a claim that the agent introduced the failure.
Section — Preserve the first result
Before asking the agent to make another edit, keep a short record of what happened:
• The exact command, including any test filter or options.
• The repository revision or starting point, if you know it.
• The failing test name and the first relevant error output.
• Whether the command completed, timed out, or was interrupted.
• The runtime and dependency information that is easy to inspect.
• Which files the agent changed and whether the working tree contains other edits.
Do not paste credentials, customer records, or private data into a report. Redact secrets while preserving enough structure to understand the error. Save a concise log or excerpt rather than treating a wall of output as a diagnosis.
If a process is still running, establish that state before launching another copy. A duplicate test server, migration, or browser test can change the conditions you are trying to observe. The relevant controls depend on your agent and workspace. Orke’s sessions guide (https://orke.bomtobe.com/en/docs/orke/sessions/) explains the distinction between an execution session and its window; check the current product documentation for the controls available in your setup.
Section — Compare the change with a baseline
Use the clean starting revision where practical. Keep the same test command, runtime, dependency lockfile, and required services. If the repository has uncommitted work from before the agent started, record that too: the baseline is the state the agent actually received, not an imagined clean checkout.
There are a few reasonable ways to compare, depending on how risky the command is:
1. If you already have a recorded test result from the starting revision, compare its command and environment with the failing run.
2. If it is safe, run the same focused test on the starting revision in a separate clean checkout or worktree.
3. If neither is available, mark the baseline as unknown and investigate the failure without assigning a cause yet.
Do not reset or discard work just to manufacture a baseline. Preserve unrelated local changes. Avoid rerunning commands that modify shared, production, or customer data. A baseline comparison is useful only when the comparison itself is safe and reasonably equivalent.
Section — Sort the failure into a useful category
Case — The same failure exists on the starting revision
The failure is not evidence that the agent’s patch caused it. Record it as a pre-existing failure, then ask whether the task requires resolving it. If the task is unrelated, keep the original scope and report the baseline limitation clearly.
An agent may still have made a change that worsens the failure or introduces a second problem. Compare the full output, not only the final nonzero exit status.
Case — The test passes before and fails after
That comparison makes a regression more plausible, but it still does not identify the exact cause. Check whether the failing test covers the changed behavior. Inspect the diff, the assertion, and the first meaningful error. Then ask for the smallest explanation tied to specific changed lines and test evidence.
Keep the repair narrow. If the agent proposes changing the assertion, deleting the test, broadening a timeout, or weakening validation, ask what behavior the test protects and what independent evidence supports the change. A green result is not enough if the expected behavior has been changed to match a bug.
Case — The same test fails sometimes and passes sometimes
That pattern suggests instability, but a small number of runs cannot establish a reliable failure rate. Note each run and its conditions. Look for shared resources, timing assumptions, test ordering, external services, random seeds, or unclean state. Do not let repeated retries silently replace a reliable test with an unexplained “eventually green.”
If you compare revisions, run the same bounded procedure on both. Stop when the planned comparison is complete; do not keep running a costly or state-changing test merely to get a preferred answer.
Case — The test command did not run in a comparable environment
Check the visible facts first: runtime version, installed dependencies, lockfile changes, required service availability, environment flags, and whether the command ran locally or in a container. Record unknowns as unknowns. Do not claim that two runs are equivalent merely because they used the same command text.
If the environment differs, make that difference explicit and either restore a comparable setup or narrow the conclusion. A test result from one environment can still be useful, but it answers a narrower question.
Case — A different test fails after the patch
Read the test name and assertion before assuming it is unrelated. The failure may expose a dependency between components that was not obvious from the task. Trace whether the edited path can reach the behavior under test. If there is no evidence of a connection, report it separately rather than expanding the patch by guesswork.
Section — A fictional example
Imagine a small settings page where a blank display name should leave the saved name unchanged. Before an agent change, the focused test passes on the starting revision. The requested change is to show an explanation beside the field. After the patch, that test fails because submitting the blank value clears the saved name.
The comparison makes a regression plausible: the same focused test passed before, failed after, and the changed form handler is on the path named in the assertion. The next step is still review, not an instant diagnosis. Inspect the form handler diff and confirm that the test represents the intended behavior. Keep the explanation change separate from the save behavior if the two concerns can be handled independently.
If the same test also failed on the starting revision, this comparison would not support blaming the patch. If the starting result was never recorded and cannot be reproduced safely, the baseline stays unknown. In either case, report that limitation and avoid editing the test merely to obtain a green result. This example is invented to explain the method; it is not a report from a real application or a measured Orke workflow.
Section — Review the diff and the test evidence together
The diff answers what changed. The test output answers what one command observed. Neither replaces the other.
For each changed file, ask what task requirement it serves. For each failing assertion, ask which changed path could affect it. Look for generated files, configuration edits, lockfile changes, test modifications, and unrelated cleanup. Make sure the agent distinguishes facts it inspected from explanations it inferred.
If the agent cannot point to the assertion and changed behavior that connect the failure to the patch, ask it to investigate without editing. This read-only step often produces a more reviewable next action than another speculative fix. For a broader method to map one behavior before coding, see and its .
Section — A compact handoff prompt
You can give an agent this bounded request: Do not edit files yet. Compare the failing test with the starting revision using the same command and a comparable environment if safe. Report the command, revision, environment differences, exact failing assertion, and whether the same failure exists before the patch. Separate observations from hypotheses. Do not reset or discard existing work, change production data, weaken tests, or rerun a state-changing command without explaining why it is safe. If a comparable baseline is unavailable, say so and recommend the smallest next check.
After reviewing that report, decide whether to authorize a code change. If the failure is clearly tied to the patch, state the expected behavior and ask for the smallest correction. If the baseline was already failing, decide whether that issue belongs in the current task. If the environment is unknown, resolve or document that uncertainty before treating a test result as a verdict.
Section — Keep a small comparison record
Use this worksheet in a pull request, task note, or local review:
• Starting revision or state
• Changed revision or state
• Command and options
• Environment facts checked
• Baseline result
• Changed-state result
• Failing test and assertion
• Relevant diff paths
• Observed facts
• Open hypotheses
• Next safe check
The record does not need to be elaborate. Its purpose is to make the claim reviewable later: what was compared, what happened, and what remains uncertain. If you cannot fill in a field, mark it unknown rather than guessing.
Section — Decide what “done” means
A trustworthy handoff says more than “tests pass” or “tests fail.” It identifies the exact command, the observed result, the comparison point, and any limitation that affects confidence. It also says whether the agent changed tests or configuration and why.
For a successful change, review that the relevant test checks the intended behavior and that the patch did not make the test easier by removing the meaningful assertion. For an unsuccessful or blocked change, keep the failing output and baseline distinction visible. A careful “not yet known” is more useful than a confident diagnosis built on a single run.
Section — References
• Orke sessions and terminals (https://orke.bomtobe.com/en/docs/orke/sessions/)
Responses (0)
Join the conversation
Sign in to share your thoughts and talk with the author.
Sign in to respondNo responses yet. Be the first to share your thoughts!