BACK
2026-10-30
Proof before fixing AI review findings
You shouldn't change your code just because an AI reviewer says it has an issue.
When several reviewers look at a pull request, their comments can become a bottleneck. Some point to the same problem. Others describe a plausible failure that doesn't happen in the code. Changing the implementation as soon as I read a comment can add work without making the PR better.
I use /babysit to work through that feedback. It groups related comments, investigates each finding, and asks a separate agent to reproduce a plausible failure before proposing a fix. The LinkedIn post gives the short version. Here I show how evidence moves through the workflow and what happens when a finding can't be confirmed.
Install the workflow
Both commands install /babysit and its two companion skills, /verify and /commit-message, from my dotfiles repository.
Choose the command for your platform. The installer detects your coding agents and leaves existing agent instructions and terminal settings unchanged.
macOS or Linux:
curl -fsSL https://r1cco.com/skills/babysit | bash
Windows with PowerShell 7:
irm https://r1cco.com/skills/babysit.ps1 | iex
Then ask your coding agent to use /babysit on an open GitHub pull request.
Group the feedback
The workflow collects the PR's intent, changed files, unresolved review threads, review submissions, and relevant PR comments. It also includes the constraints I gave for that PR, such as work I explicitly asked it not to change.
It groups comments by the problem they describe. Two reviewers might point to different lines while talking about the same invariant. If the cause is shared, those comments become one finding.
Grouping makes each problem one unit of work, regardless of how many comments describe it. One fix can address several reports, and the workflow can keep their sources together when it records the decision and replies.
Investigate before attempting proof
Each group goes to a read-only agent with the same PR context and the relevant comments. Its job is to explain the claimed failure and point to the code that supports it.
The agent returns one of three initial results:
| Result | What happens next |
|---|---|
| Candidate | A plausible functional problem goes to a separate proof step. |
| Skip | The finding is a hypothetical issue, a style comment, or a problem outside the approved scope. |
| Unsure | The agent cannot tell whether the behavior is a bug or an intentional product decision. |
A reviewer’s priority label does not decide the outcome. The investigation needs to explain what could fail and where.
A finding that might already be fixed still goes to the proof step. A code-reading agent saying “this no longer happens” is useful, but I want to check that claim too.
Reproduce the claimed failure
A different agent takes each candidate and tries to make the failure happen on the PR's current code. It records the command, its inputs, and the result. The goal is to check the specific hypothesis rather than produce a general test failure.
The proof step can return three outcomes:
- Reproduced: the claimed failure occurred. The finding can move into the fix plan.
- Not reproduced: the attempt ran, but the claimed failure did not occur. The finding can be skipped or marked already fixed if the code or history independently shows that a later change addressed it.
- Unable to check: the attempt could not be run. The finding remains unresolved.
A failed attempt does not prove that a bug is impossible. It tells me what happened under the tested conditions.
The proof agent works outside the PR worktree. For trusted code, the skill can use a disposable worktree; for untrusted code, it requires isolation without developer credentials. A worktree by itself is not a security boundary.
One finding from a real PR
In dotfiles PR #75, a reviewer reported a problem with the shell lifecycle helper's signal handling. The concern was that a termination signal could release a lock while the script continued running.
One agent investigated the finding. A separate proof agent tested it against the PR at commit 643d97a. In a disposable environment, it acquired the lock, sent SIGTERM, and attempted a subsequent state write.
The captured result included:
locked=1
continued_after_term lock_held=0
state_written_after_term=yes
FAIL: mutation continued after SIGTERM with no lock
The script continued writing after the lock was released. The result did not demonstrate concurrent data corruption, so I don't claim that corruption occurred.
The fix kept lock cleanup on exit and changed the signal handlers to exit when they receive HUP, INT, or TERM.
I later ran the committed regression test against both versions of the helper. The original failed the TERM and HUP cases because execution continued; the fixed version passed both. That was a targeted corroborating check, not a rerun of the full suite or the PR's remote CI.
The review identified a plausible failure, the proof attempt showed a specific behavior, and the regression check tested the correction. Each result supports a different part of the decision.
Decide what to fix
After validation, /babysit presents a scope table when it needs authorization. It lists the findings, their outcomes, the evidence, the proposed commits, and the verification to run. If I've already authorized the same work in the session, it can reuse that grant.
A reproduced finding becomes an address item. An untestable finding remains blocked while it waits for a decision. The rest of the approved work can continue.
The workflow also distinguishes a skip from a disagreement. If an attempted reproduction does not fail, the skill can record that result and skip the finding. If it disagrees with a reviewer about an out-of-scope issue, it leaves the thread open for a person to decide.
CI failures follow a separate path. A failed remote check is an observed result, not a reviewer’s hypothesis. The skill reads the job log and investigates it without requiring the same proof step as a review finding.
Verify, reply, and watch the PR
For approved fixes, /babysit makes one commit per finding group. It runs verification after the fixes in a push batch, then pushes and refreshes the remote checks.
A local pass does not mean the PR's CI is green. The workflow waits for current checks and review automation, and it reports when expected results are missing. New feedback or failures can start another pass through the cycle.
After addressing feedback, it can reply to known bot threads with the relevant commit or the reason a finding was skipped. Replies to human reviewers require authorization for the exact text. Disagreements and findings that could not be checked stay open.
The skill handles one cycle on one open PR. It doesn't merge. I still decide when the evidence and remaining review state are enough to merge.
What I value most is the separation between a plausible claim and an observed failure. An AI review can point me toward something worth investigating. The proof step gives me a concrete result to use when deciding whether the code needs to change.