Tag: Evaluation

All the articles with the tag "Evaluation".

• What Makes an Agent Swarm Work? Simple Rules, Information Boundaries, and the Value of Verification

  5 Sep, 2026 · 37 min read

  A theory-and-experiment study of agent swarm organization: operational roles, marginal-contribution incentives, selective information sharing, and why verification remains the limiting assumption.

• Do Verifier Errors Grow Superlinearly with Horizon? A Six-Stage Experiment

  Updated: 17 Aug, 2026 · 17 min read

  A preregistered 224-artifact study confirms a 66% hybrid-verifier improvement, does not support the universal superlinear headline, and finds that task structure can reverse the apparent horizon effect.

• From Tool Calls to Policy Updates: A Reproducible Agent RL Stack

  14 Aug, 2026 · 45 min read

  A reproducible blueprint for Agent RL: environment snapshots, verifier contracts, credit assignment, three-policy semantics, asynchronous generation and training, partial rollouts, and runnable smoke tests.

• A Repair Can Save a Trajectory Without Teaching the Agent

  14 Aug, 2026 · 18 min read

  An eight-seed counterfactual-replay experiment found that high-value one-action repairs reliably rescue failed agent trajectories—but a repair-value-dominant SFT selector did not beat simpler rules and recovered only 25% of matched-RL improvement.

• Why Embeddings Cannot Solve Eval-Set Contamination

  29 Jun, 2026 · 11 min read

  A technical deep dive on why semantic embedding search is useful but insufficient for eval-set decontamination: leakage is about evaluation advantage, not just text similarity.

• Pretraining Contamination: Why Don't Train on the Test Set Became Hard

  29 Jun, 2026 · 14 min read

  A practical introduction to LLM pretraining contamination: why benchmark leakage is not ordinary deduplication, how public evals leak into web-scale corpora, and how layered decontamination pipelines reduce risk.