Reproducing CompactRL: What Worked, What Failed, and Why We Did Not Scale

Long-horizon agents eventually face a basic systems problem: their interaction history can exceed the model’s context window before the task is complete. Summarizing old context keeps the agent running, but an inference-time summary is not automatically a good reinforcement-learning action. If it drops the one fact needed 50 turns later, the policy may never learn that the summary caused the eventual failure.

CompactionRL makes a sharper proposal:

Treat context compaction as an action sampled by the trainable policy, and give the summary tokens credit from the final task reward.

I built an open reproduction to test that idea from three directions:

1. implement and independently test the public equations;
2. test whether learned summaries solve a controlled 96-step task with three context resets;
3. integrate the method with public THUDM/slime and run real Qwen actor–critic training on Modal.

The complete code, configs, reports, and decision artifacts are public in ajing/compactionrl-repro.

The short conclusion is:

• the algorithmic path is correct;
• the controlled long-horizon experiment works;
• the real LLM training path works;
• stable small-model capability improvement is not closed;
• the final experiments did not justify further scale-up.

This post explains both the positive result and the evidence that stopped us from turning it into a larger claim.

What “the GLM-5.2 technical report” actually refers to

The public evidence is split across three sources:

1. The GLM-5.2 model card describes the model and training pipeline.
2. The GLM-5 technical report describes asynchronous agent RL and the slime infrastructure in more detail.
3. The CompactionRL paper specifies the compaction rollout, loss, and cross-trajectory credit equations reproduced here.

There is no single public “GLM-5.2 technical report” containing a directly executable CompactRL recipe. This project implements the intersection of the three public sources and explicitly separates it from undisclosed prompts, dataset ordering, reward code, and large-model training settings.

It does not claim to reproduce the paper’s 30B or 106B benchmark scores.

The algorithm in one pass

Let the active history be

where each assistant action $a_i$ and environment observation $o_i$ form one atomic step. If the model’s context capacity is $C$ and the required remaining budget is $T_{\text{comp}}$, compaction triggers when

The same policy then samples a summary. Execution resumes from

One logical trajectory becomes several trainable segments:

execution_0
summary_0
execution_1
summary_1
execution_2

Four details are especially easy to get wrong:

• The summary is an action. It receives actor loss when the policy samples it.
• The copied summary is context. Its later appearance in the resume prompt must be masked so it does not receive loss twice.
• Tool calls and observations remain atomic. Splitting a pair corrupts environment state and token provenance.
• Credit does not reset at segment boundaries. Earlier summary tokens must remain connected to the eventual task reward.

Cross-trajectory GAE

The paper computes local GAE inside each segment, then discounts earlier segments by the number of optimized tokens that occur after them. For segment $s$:

The length-adaptive trace parameter is

where $L_s$ is the response length.

Token-normalized PPO

Variable segment counts and lengths create another trap. Averaging one loss per segment lets a short summary receive the same weight as a long execution sequence. The implementation instead reduces once over every enabled token in the global batch:

These are not cosmetic details. The paper’s 106B compacted ablation scores 66.8/24.5 with the full method, 60.0/21.3 without token-level loss normalization, and 63.0/22.5 without cross-trajectory GAE correction.

The full derivation and disclosure gaps are in the repository’s technical analysis.

What the paper reports

The public results motivate this reproduction; they are not directly comparable to our 0.5B synthetic-task experiments.

The narrow supported conclusion is that compaction-aware RL beats the tested baselines when evaluation also uses compaction under the same peak-context constraint. It does not establish universal superiority.

An exact reproduction is blocked by missing details, including literal summary and resume prompts, exact training rows and order, maximum summary length, verifier and timeout behavior, optimizer settings, critic initialization, and the internal GLM-5.2 configuration.

The central question: what should the value function learn?

This was the easiest place to obtain a training run that appeared healthy while delivering a corrupted actor signal.

CompactRL does not require a critic whose target is “summary quality.” The default critic still predicts expected task return:

Summary and execution tokens belong to the same terminal-reward chain. We compute task returns and GAE first, then apply reward-distance correction to summary advantages across segment boundaries. Compaction changes trajectory structure and credit distance; it does not change the critic’s semantic target.

The implementation maintains these invariants:

• execution and summary segments share the terminal task reward;
• the critic evaluates trainable summary states;
• copied context does not receive duplicate policy loss;
• critic warm-up may precede actor training;
• actor and critic checkpoints use separate paths;
• a missing scalar value head is zero-initialized, not randomized before PPO.

That last issue caused a concrete failure. On an all-zero-reward replay, a random value head predicted approximately 0.9747, produced critic gradient norms as high as 785.3, and even produced an actor gradient norm of 47.8. After zero initialization, the same audit yielded exactly zero values, returns, advantages, and gradients.

Why later summary-quality signals are actor-only

Phases 7–17 tested fidelity rewards, positive routing, token-local credit, coverage-stop pressure, and causal-prefix redistribution. The resulting rule is:

The task critic learns only the environment-defined task return. Handcrafted summary-quality signals enter as optional actor-only auxiliary advantages after task GAE has been computed.

A repeated field, missing milestone, or excess summary token is not itself an environment value. Mixing those heuristics into returns can produce a critic that accurately fits the wrong target.

The corrected exact-paired Phase 17 audit validates this boundary. All 48 task streams, outcomes, and token trajectories match Phase 16 exactly. Every critic/value metric is bit-identical. Only actor loss and gradient change, by the small amount predicted by the auxiliary intervention.

The current data flow is:

environment reward
    → task return and GAE
    → critic target
    → cross-trajectory correction
    → optional actor-only summary credit
    → PPO actor loss

A controlled 96-step long-horizon experiment

Before renting GPUs, I built a dependency-free memory chain:

• each episode lasts 96 steps;
• the context resets at steps 24, 48, and 72;
• only the newest two atomic steps survive outside the summary;
• the final answer depends on a fact observed at step 1;
• the joint arm trains execution and summary decisions;
• the control uses the same actor–critic but freezes the summary policy.

Across seeds 7, 17, and 29:

This is not a language-model benchmark. It answers a narrower causal question: can terminal reward traverse multiple compaction boundaries and teach an early summary action to preserve task-critical state? In this setting, yes.

The experiment and its readable trace are both public.

Integrating with public slime

The audited slime revision is fb42ae4. It already provides PPO, a critic, per-token loss, a custom advantage hook, list[Sample] fan-out, and agent adapters. At that revision, it does not provide a directly enabled “CompactionRL estimator plus rollout.”

The reproduction therefore uses a thin integration:

• framework-neutral equations live in src/compactionrl/;
• rollout and framework adapters live in integrations/slime/;
• custom advantages enter through a public extension point;
• a minimal patch is pinned to the audited critic loop;
• each live run records immutable identity, token trajectories, masks, advantage/return audits, actor/critic gradients, and role-specific checkpoints.

The concrete contract and launch path are documented in the slime integration guide.

Calibration before the numbered phases

The later experiments did not begin from an unverified GPU stack. Earlier controls progressively isolated sampling, checkpoint, grammar, and full-context learnability:

The reversal of the initial KL 0.05 result is particularly important. Without a fixed task stream and common sampling randomness, an attractive RL difference may simply represent two different training trajectories.

Complete Phase 1–17 result ledger

PASS below means that a mechanism or preregistered gate passed. It does not mean that the paper’s model capability was reproduced.

Every number traces to a phase report and a frozen decision artifact.

Failures that could have produced false conclusions

A random value head

The missing critic head was initially random, so even zero reward produced large advantages. Zero initialization and a zero-reward replay gate fixed it.

A shared actor/critic checkpoint directory

Critic shards saved later replaced actor shards, invalidating an early saved-checkpoint capability claim. Role-specific paths and checkpoint inventory are now publication gates.

Re-encoding text does not reproduce sampled tokens

Phase 14 initially retokenized decoded summary text to locate credit. A real tokenizer does not guarantee segment-wise decode–encode identity. The repair aligns against sampled-token prefix decoding and audits every offset and its total mass.

Logging made valid training look like a failed mechanism

A nested diagnostic could not be serialized by the slime logger. Rollout and critic work had completed, but the run produced no checkpoint. Phase 15 closed the mechanism only after repairing the logger.

The first Phase 17 run was not an exact pair

The launcher omitted the Phase 17 profile and routed it to synthetic, while Phase 16 used synthetic_phase10. The completed run was rejected as paired evidence. After freezing and testing dataset routing, the corrected run matched all 48 tasks, outcomes, and trajectories.

These failures show why “RL is unstable” is not a sufficient explanation. Dataset, sampling, value initialization, checkpoint, and logging errors must be eliminated before the remaining difference can be called optimization variance.

Why we did not scale

Several local results were strong:

• deterministic no-KL seed 23: 26/60 vs base 14/60, paired p=0.00754;
• independent same-seed rerun: 25/60;
• seed 17: 29/60 vs base 14/60, paired p=0.000729;
• substantial ordered-plan gains on two Phase 2 seeds;
• simultaneous exact, invention, and length improvement in Phase 11.

The stronger negative evidence matters just as much:

• the initial KL 0.05 gain reverses under common-random-number control;
• the first ordered-plan full-context control is 0/20 because of output grammar, not memory;
• once grammar SFT reaches 20/20 in full context, compaction reduces it to 8/20, isolating summary transfer as the real problem;
• the third Phase 2 seed fails;
• Phase 3 fails its absolute invention gate;
• most Phase 11–17 interventions improve one metric while damaging repetition, invention, or length;
• Phase 17 changes the real actor gradient by only about 0.059%.

Simultaneously satisfying all criteria is difficult because the current small model and reward design operate in a multi-objective conflict region. That is an experimental result, not a reason to lower the criteria. Adding GPUs would sample the same conflict more expensively.

Cost discipline

Later experiments used a staged authorization ladder:

1. local analytic or counterfactual gate;
2. tokenizer and dataset preflight;
3. one-update mechanism probe;
4. evaluation only after mechanism and quality gates pass;
5. multi-seed or scaled runs only after evaluation passes.

The conservative GPU cost ceilings for Phases 15, 16, and 17 were approximately $0.75, $0.69, and $1.07. Phase 17 includes both the rejected wrong-dataset run and the corrected pair. We did not purchase a Phase 17 evaluation because the exact-paired gradient audit already showed that the effect was too small.

A cheap run is not justified merely because it costs less than one dollar. It must answer one question that can change the next decision.

Reproduce the reproduction

The core algorithm and long-horizon simulation require no third-party package:

git clone https://github.com/ajing/compactionrl-repro.git
cd compactionrl-repro
PYTHONPATH=src python3 -m unittest discover -s tests -v
PYTHONPATH=src python3 examples/toy_joint_training.py
PYTHONPATH=src python3 experiments/long_horizon_memory_chain.py

Generate and inspect the deterministic synthetic data:

PYTHONPATH=src python3 scripts/generate_sim_data.py
PYTHONPATH=src python3 scripts/summarize_sim_data.py

Validate a pinned slime checkout:

python3 scripts/check_slime_compat.py --slime-root /path/to/slime

For a real Modal run, do not begin with the largest profile. Read the preflight, evidence-download, and publication-gate sections of the integration guide, then select a frozen configuration.

What should come next

The next step is not to scale Phase 17 unchanged. More promising changes are:

1. keep the task critic pure and train a separate, calibrated summary-quality model used only as a constraint or actor-side control variate;
2. represent summaries as structured memory slots so write, overwrite, and stop become explicit actions;
3. begin with tasks already proven solvable in full context, then increase compaction count through a horizon curriculum;
4. preregister a cross-seed Pareto gate over task success, milestone recall, invention, repetition, and summary length;
5. move to a larger model only after the small model exhibits a stable and sufficiently large gradient effect.

Bottom line

CompactRL’s core idea is credible and reproducible: make the summary a policy action and propagate terminal reward through context resets to summary tokens. In the controlled 96-step experiment, it nearly perfectly solves a problem that the frozen-summary control cannot solve. On public slime, the reproduction also demonstrates an auditable path through actor, critic, GAE, masks, checkpoints, and deterministic trajectories.

“The algorithm can work” and “this small-model recipe works robustly” are different claims. The evidence supports the first. The second likely requires a different summary representation or credit model, not a larger copy of Phase 17.

The most useful artifact is not only a successful experiment. It is the combination of successful and failed evidence that prevents the next researcher from paying to rediscover the same mistakes.