A Repair Can Save a Trajectory Without Teaching the Agent

A repaired trajectory can be correct, causally useful, and still be poor training data.

That distinction matters for a post-training recipe that sounds almost obviously right:

1. let a small agent attempt a task;
2. ask a stronger model to repair the small agent’s failed trajectory;
3. optionally let the small model rewrite the successful teacher trajectory in its own style;
4. fine-tune the small model on those corrected traces;
5. hope that supervised fine-tuning recovers most of what online reinforcement learning would have learned.

The recipe is plausible. Teacher-generated trajectories can improve smaller agents. Correcting student failures moves supervision toward states the student actually visits. Re-executing the repair can prevent a teacher from inventing tool observations. A student-side rewrite may also reduce the gap between the teacher’s language distribution and the student’s.

But one question is usually left implicit:

If I have several verified repairs, which one is the most valuable training example?

I tested one tempting answer: prioritize the repair with the largest measured increase in the current agent’s probability of success. I call that quantity the Causal Value-of-Repair, or CVoR. The implemented selector used student likelihood to break the many ties in that coarse repair-value estimate, a detail that matters later.

The experiment produced a sharp negative result:

• CVoR reliably found one-action repairs that rescued the current failure;
• its independent repair rescue rate was 79.1%;
• the candidate ranking generalized to held-out suffix randomness;
• CVoR-selected supervised data improved the base policy;
• but it did not beat simpler earliest-repair or random-repair selection after SFT;
• and it recovered only 25.4% of the gain from interaction-bracketed online RL.

The central lesson is:

immediate causal rescue value
≠ marginal learning utility

This post explains the hypothesis, the controlled test, why the mechanism passed while the training claim failed, and how I would change a real LLM-agent trajectory pipeline.

Why the idea was plausible

The high-level recipe is not new. Several lines of work make it credible.

FireAct showed that successful GPT-4 agent trajectories can substantially improve smaller language agents through fine-tuning. In its default HotpotQA setting, it used 500 successful GPT-4 trajectories, and the Llama-2-7B result improved by 77% relative to its prompted baseline.

STeP moved closer to failure repair. A large teacher detected erroneous actions, inserted reflection and correction, kept only successful corrected trajectories, and masked incorrect steps from the supervised loss. Its results support the idea that error-correction traces can be more useful than only imitating clean expert demonstrations.

Revisiting DAgger in the Era of LLM-Agents addresses the state-distribution problem directly. The student interacts with the environment, the teacher labels student-relevant states, and the dataset is aggregated iteratively. On SWE-bench Verified, the reported DAgger-style method improved over the strongest post-training baseline by +3.9 points at 4B and +3.6 points at 8B.

At the same time, WebAgent-R1 shows why supervised warm-up and online optimization should not be treated as interchangeable. Behavior cloning helps make sparse-reward learning reachable, but online multi-turn RL changes the policy using states generated by the current policy.

The missing piece is not whether corrected trajectories can help. They can. The narrower question is whether counterfactual rescue value is the right objective for selecting among corrected examples.

Two quantities that look similar but are not

Suppose a frozen student policy takes an erroneous action $a_t$ at state $s_t$. A teacher proposes a replacement action $a_t^*$. Restore the exact pre-action state, execute each action in a separate branch, and then return control to the same frozen student policy.

The local repair value is

where $u$ indexes matched suffix randomness and $R$ is terminal task success.

This is an action-intervention estimand. It asks:

If I replace this one action now,
how often does the current policy
succeed?

Training utility is different. For a labeled repair example $z$, the relevant quantity is closer to

It asks:

If I update the model on this example,
how much does held-out capability
move?

The first quantity depends on how decisive an action is in the current state. The second also depends on coverage, redundancy, gradient alignment, model capacity, regularization, optimizer dynamics, and how the update changes future state visitation.

There is no reason for the two rankings to be identical.

A deliberately small, exactly branchable experiment

Before paying for LLM rollouts and GPU RL, I built a controlled mechanism test. The environment is a deterministic, exactly branchable key-then-goal grid with walls, traps, tight deadlines, sparse terminal reward in {0, 1}, and immutable state snapshots. The policy receives a fixed 36-feature local observation. The evaluator and BFS oracle—not the policy—have access to the full task state.

The agent must pick up a key and reach the goal. An exact BFS oracle enumerates the complete set of shortest-path-preserving safe actions at each visited state. An action counts as an error only when it has positive regret relative to that set. This avoids calling an alternative but equally short path an error. Near a deadline, however, even a safe replacement action need not lead to terminal success once control returns to the student.

The same small two-layer policy is used everywhere:

• one 48-unit hidden layer;
• four environment actions;
• supervised cross-entropy for SFT arms;
• terminal-reward REINFORCE for the RL arms.

This is intentionally not a language-model benchmark. It isolates the causal selection mechanism while holding policy parameterization fixed across SFT and RL.

It also tests a minimal repair, not a full teacher-written natural-language trajectory. Each selected repair contributes a repaired state-action label, plus matched anchor labels. A positive result would justify moving to a bash-only code-agent environment. A negative result tells us not to scale the selection rule yet.

How repairs were generated and audited

For every failed frozen-policy rollout:

1. identify positive-regret steps using the BFS-optimal action set;
2. propose at most the earliest and latest error states;
3. freeze the transcript-equivalent state before the candidate action;
4. execute the original action and the repaired action in separate branches;
5. return control immediately to the frozen student;
6. reuse the same suffix random numbers for both branches;
7. score terminal success with the environment, not a learned judge.

The selection estimate used four suffix seeds. All 7,960 candidates—not only the winners—were then replayed with eight disjoint audit suffix seeds that could not affect selection. This audit was a research measurement layer, not part of the deployable selector.

No observation was copied from the original branch after the action changed. Every downstream state came from real re-execution. This is the branch-and- compare version of the counterfactual credit idea described in Training the Critic Without Crashing the Reward.

Matched training arms

The confirmatory run used eight paired checkpoint seeds. Per seed, it generated:

• 250 behavior-cloning pretraining tasks;
• 900 repair-collection tasks;
• 600 held-out in-distribution tasks;
• 300 size-9 out-of-distribution tasks.

Every supervised selection arm started from the same base checkpoint and used:

• the same candidate pool;
• 96 repair labels, balanced as 32 per task family;
• 96 anchor labels;
• 36 optimizer steps;
• the same learning rate, batch size, and number of epochs.

The main arms were:

The preregistered primary comparison was CVoR-SFT minus Earliest-Repair SFT. The downstream gate required a gain of at least +5 percentage points, a 95% paired-bootstrap lower bound above zero, and positive differences in at least seven of eight checkpoint seeds.

One protocol detail was not specified tightly enough in the preregistration. Four-seed repair values tie frequently, so the implementation sorts first by estimated delta, then by lower repair-action NLL, then by earlier step and a stable candidate ID. 1,382 of 4,359 task groups tied at their maximum delta, and changing the NLL tie-break to candidate ID changed about 13.3% of the selected examples. The test should therefore be read as a delta-dominant selector with an NLL tie-break, not a mathematically pure delta-only arm.

The mechanism passed

CVoR did what it was designed to do at the branch level.

Across the selected repairs:

CVoR’s rescue advantage over Random was +51.3 pp, with a 95% paired bootstrap interval of [+48.0, +54.6] pp.

The selection score also generalized to independent replay. Across each seed’s full candidate pool, the mean Spearman correlation between selection delta and audit delta was 0.734.

This is strong mechanism evidence. The delta-dominant ranking can identify actions that genuinely rescue the current policy from the current state.

Then the causal ranking failed to become a learning ranking.

CVoR-SFT helped, but simple repair selection helped more

Held-out in-distribution terminal success was:

CVoR-SFT improved over Base by +4.08 pp. The 95% interval was [+2.44, +5.83] pp, and all eight seed-level contrasts were positive.

So the package of CVoR-selected repair labels plus shared anchor labels improved over no update. This design does not isolate verification itself.

The problem was selection:

• CVoR averaged 1.46 pp below Earliest, with interval [-3.58, +0.98] pp, and failed the preregistered superiority gate;
• only 2/8 seed-level contrasts favored CVoR;
• in a secondary comparison without multiplicity correction, CVoR minus Random was -1.75 pp, with interval [-3.19, -0.23] pp;
• only 1/8 seeds favored CVoR over Random.

The preregistered downstream gate failed clearly.

The controls make the result more informative. Sham-CVoR trained on the same high-value states but reinforced the original wrong action. It fell 8.67 pp below Base. Correct labels matter; merely repeating important states is not enough. But there was no anchor-only or unverified-repair control, so this run cannot attribute the entire Base-to-CVoR gain to verified repair labels alone.

The exploratory student-likelihood gate also hurt. Filtering CVoR repairs using a q90 negative-log-likelihood threshold reduced ID success by 1.83 pp relative to plain CVoR, with all eight seeds moving in the wrong direction. “Easy for the student to imitate” was not a free proxy for “useful for the student to learn.”

The full preregistered decision table is stricter than any one contrast:

Online RL remained substantially stronger

The RL arms used the same base policy and only terminal {0, 1} reward.

The ID success rates were:

Base-to-CVoR gained 4.08 pp. Base-to-matched-RL gained 16.06 pp. The fraction of RL improvement closed by CVoR was therefore

CVoR remained 11.98 pp below matched RL. Even the quarter-budget RL checkpoint was 3.15 pp higher.

On the environment-transition axis, CVoR selection used 62,449.9 deployable transitions per seed. The reported 1x RL checkpoint was the first batch at or above that target and used 62,830.1, so “matched” means tightly bracketed, not exactly equal. The quarter-budget RL checkpoint used about 15.9k.

That does not establish complete economic dominance. The v1 cost ledger cannot make a full Pareto claim:

• BFS error screening is local computation, not a measured LLM-teacher query;
• clean teacher-path generation was not instrumented on the same cost axis;
• the Earliest, Random, and Uncertainty arms were selected from the shared candidate pool, but their native oracle-screening pipelines were not executed and costed separately;
• SFT and RL wall-clock timers cover different evaluation work;
• the independent audit used another 104.7k shared research transitions per seed, which should not be charged to a deployable CVoR arm.

The honest decision is:

mechanism:                 PASS
training / RL closure:     NO-GO
full economic Pareto:
    NOT EVALUABLE

Why high rescue value did not produce high learning value

The experiment does not identify one unique cause, but several mechanisms are consistent with the result.

1. Decisive states can be redundant training data

A repair can have enormous local value because it prevents an immediate fatal error. If many high-CVoR candidates teach nearly the same decision boundary, top-k selection spends labels on redundant gradients.

Random or earliest selection may cover more distinct layouts, error depths, and recovery contexts even when each example has lower immediate rescue value.

The local policy observation also aliases some remote layouts. This is useful for testing branch-level interventions but may make superficially similar labels require different long-horizon behavior, weakening static generalization.

2. Local value is policy-specific

CVoR was measured under the frozen base policy. After supervised updates, the policy changes. A state that was decisive for $\pi_0$ may become rare under the updated policy, while an apparently modest repair may prevent a broad family of future errors.

Online RL repeatedly refreshes this distribution. Static SFT does not.

3. One action can rescue an episode without teaching the suffix

The repair intervention replaced one action and returned control to the frozen student. That is ideal for causal identification. It is not automatically the best pedagogical unit.

The model may need a sequence of decisions: avoid the error, recognize the new observation, update the plan, recover if the repair was late, and stop at the right time. A one-turn label can identify the hinge action while omitting the behavior that makes the skill reusable.

4. Student likelihood can favor familiarity over information

The failed NLL gate is a warning for student-side trajectory rewriting. A trace that looks more probable under the student may be easier to imitate because it contains less new information.

Student compatibility matters, but raw likelihood is not the same as expected training gain.

5. RL optimizes the deployed state distribution

REINFORCE receives a sparse signal, but it receives it on trajectories sampled from the changing policy. It can improve action choices, path length, recovery, and state coverage together. CVoR-SFT only updates on a fixed labeled slice of the original failure distribution.

This is why SFT is often a good bootstrap without being an RL substitute.

What I would do for a real code or web agent

I would keep the repair pipeline, but change the selection objective.

1. Start from student-visited states

Collect failures on-policy or with a DAgger-style mixture. Pure teacher trajectories miss the states created by student mistakes.

2. Let the teacher propose minimal interventions

The teacher may edit reasoning or propose a replacement action, but it should not invent observations. Restore the real environment, execute the new action, and regenerate every downstream observation.

For code agents, run the command and tests. For web agents, replay from a valid snapshot. If the environment cannot branch, explicitly separate rollback repair from no-reset recovery.

3. Keep only executable, verified branches

A polished trajectory that cannot be replayed is not agent-training data. Mask incorrect turns from the supervised target, and preserve the real error observation when training recovery behavior.

4. Use CVoR as a feature, not the target

Counterfactual rescue is useful for fault attribution and candidate triage. I would combine it with:

• task- and state-family coverage;
• gradient similarity and redundancy penalties;
• error depth and horizon;
• recovery versus avoidance balance;
• teacher-query and environment-replay cost;
• a cross-fit estimate of one-step validation utility.

The target should be closer to

expected held-out capability gain
/ acquisition cost

than to immediate episode rescue alone.

5. Treat student rewriting as projection, not verification

Letting the small model rewrite a verified teacher trajectory may improve style and distributional compatibility. But every action change must be re-executed. The student cannot safely rewrite tool outputs or environmental observations.

The important ablation is not “teacher prose versus student prose.” It is:

direct teacher target
vs. student-projected target
vs. student-likelihood selection
vs. coverage and update-utility
    selection

under the same trained-token and optimizer budget.

6. Use SFT to make online learning reachable

Verified repair SFT can teach format, tools, basic recovery, and a nonzero success rate. Then online RL or iterative DAgger can train on the state distribution produced by the improved policy.

The practical stack is not “SFT or RL.” It is usually:

student rollouts
    → teacher repair proposals
    → exact replay + verification
    → coverage-aware SFT bootstrap
    → refreshed on-policy rollouts
    → conservative RL / iterative
      expert correction

What the experiment does and does not establish

The run contains 7,960 repair candidates, 4,608 selection records, and 79,200 paired evaluation outcomes. The repository contains seven unit tests; the frozen artifact validation passed 21/21 checks. The compact data behind the figures is available as JSON.

The result supports three narrow claims:

1. exact branch replay can identify one-action repairs with real causal rescue value;
2. the package of CVoR-selected repair labels and shared anchors can improve a held-out policy;
3. immediate repair delta was not sufficient as the primary ranking signal, under the implemented NLL tie-break, to select the best supervised data in this controlled agent.

It does not establish that CVoR will fail for LLM code agents. It does not rank PPO, GRPO, or DPPO against LLM SFT. The online comparator here is a small-policy terminal-reward REINFORCE implementation. It also does not prove that full corrected trajectories are worse than one-action labels.

Those are the next experiments, not conclusions hidden inside this one.

There are additional boundaries worth keeping visible. Collection and repair replay were stochastic, while held-out evaluation was greedy. Initial behavior cloning used only the open and walls families; repair collection and evaluation also included traps and tight-deadline families. Base-to-CVoR therefore mixes correct repair supervision with exposure to new task families. The SFT arms also used method-specific minibatch shuffle seeds, so small 1–2 pp arm gaps include optimizer-order noise. Eight checkpoint replicates are adequate for a directional screen, not for a 5 pp non-inferiority claim.

Final takeaway

The original intuition was half right.

A strong teacher should correct errors made by the student. The corrected branch should be executed in the real environment. The model should not train on fabricated observations or unmasked wrong actions. This produces useful supervised data.

The failed leap was assuming that prioritizing the repair most valuable inside the current episode would also identify the example most valuable for changing the model.

It was not.

The next version should preserve counterfactual replay as the causal measurement layer, then optimize data selection for coverage and measured learning utility. That is a smaller claim than “trajectory repair can replace RL,” but it is a much better foundation for testing whether it eventually can.

Related reading

• Training the Critic Without Crashing the Reward: A Practical Guide to Agentic RL
• From GRPO Outcome Rewards to Token-Level Advantage
• Reproducing CompactRL: What Worked, What Failed, and Why We Did Not Scale
• FireAct: Toward Language Agent Fine-tuning
• Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Masking
• Revisiting DAgger in the Era of LLM-Agents
• WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning