Posts

All the articles I've posted.

• Pretraining Contamination: Why Don't Train on the Test Set Became Hard

  29 Jun, 2026 · 14 min read

  A practical introduction to LLM pretraining contamination: why benchmark leakage is not ordinary deduplication, how public evals leak into web-scale corpora, and how layered decontamination pipelines reduce risk.

• How to Arbitrarily Increase the Difficulty of Agent Evaluation Sets

  28 May, 2026 · 18 min read

  A practical framework for making agent benchmarks harder in a controlled way: treat difficulty as trajectory-graph complexity, not prompt wording. Covers deterministic scoring, capability facets, harness effects, and systematic data generation.

• The Mercor Breach: What 4TB of Stolen Data Reveals About How Frontier AI Labs Actually Train Models

  12 Apr, 2026 · 22 min read

  A $10B AI data vendor was breached, exposing 84 Airtable workspaces of training data for OpenAI, Anthropic, Apple, Amazon, and Meta. This post analyzes what the public reporting reveals about each lab's evaluation methodology — rubric design, RLHF pipelines, and quality control — and what it means for the industry.

• The Unverifiable Reward Problem: The Real Frontier of RL for LLMs

  7 Mar, 2026 · 12 min read

  Deep research on tasks with unverifiable rewards in RL — the key bottleneck for scaling RL beyond math and code. Covers JEPO, NRT, RLNVR, self-play methods, GenRM, Constitutional AI, reward hacking mitigation, and more.

• Instruction Following: What Models Get Wrong and How to Fix It with Better Post-Training Data

  4 Mar, 2026 · 37 min read

  LLMs can write poetry and solve math, but ask them to 'respond in exactly 3 bullet points using only lowercase' and they stumble. This post dissects the taxonomy of instruction-following failures and provides a practical playbook for building post-training data that actually fixes them.

• Experience-Augmented In-Context Learning: A Training-Free Complement to RL Post-Training

  28 Feb, 2026 · 24 min read

  RL post-training makes models smarter, but it can't cover the infinite long tail of real-world cases. Experience-augmented ICL retrieves successful reasoning traces at inference time, letting agents learn continuously from real usage — no retraining required.