Tag: Evaluation

All the articles with the tag "Evaluation".

• Do Verifier Errors Grow Superlinearly with Horizon? A Six-Stage Experiment

  Updated: 17 Aug, 2026 · 17 min read

  A preregistered 224-artifact study confirms a 66% hybrid-verifier improvement, does not support the universal superlinear headline, and finds that task structure can reverse the apparent horizon effect.

• From Tool Calls to Policy Updates: A Reproducible Agent RL Stack

  14 Aug, 2026 · 45 min read

  A reproducible blueprint for Agent RL: environment snapshots, verifier contracts, credit assignment, three-policy semantics, asynchronous generation and training, partial rollouts, and runnable smoke tests.

• A Repair Can Save a Trajectory Without Teaching the Agent

  14 Aug, 2026 · 18 min read

  An eight-seed counterfactual-replay experiment found that high-value one-action repairs reliably rescue failed agent trajectories—but a repair-value-dominant SFT selector did not beat simpler rules and recovered only 25% of matched-RL improvement.

• Why Embeddings Cannot Solve Eval-Set Contamination

  29 Jun, 2026 · 11 min read

  A technical deep dive on why semantic embedding search is useful but insufficient for eval-set decontamination: leakage is about evaluation advantage, not just text similarity.

• Pretraining Contamination: Why Don't Train on the Test Set Became Hard

  29 Jun, 2026 · 14 min read

  A practical introduction to LLM pretraining contamination: why benchmark leakage is not ordinary deduplication, how public evals leak into web-scale corpora, and how layered decontamination pipelines reduce risk.

• How to Arbitrarily Increase the Difficulty of Agent Evaluation Sets

  28 May, 2026 · 18 min read

  A practical framework for making agent benchmarks harder in a controlled way: treat difficulty as trajectory-graph complexity, not prompt wording. Covers deterministic scoring, capability facets, harness effects, and systematic data generation.