Posts

All the articles I've posted.

• When a Looped Transformer Router Almost Works

  1 Jul, 2026 · 12 min read

  A controlled small-scale language-composition experiment comparing fixed and routed looped Transformers, showing why the first sequence-level router was competitive but collapsed toward a weak exit policy.

• Optimizing Inference for Router Looped Transformers

  Updated: 29 Jun, 2026 · 20 min read

  A research note on serving router looped transformers: why normal KV cache semantics break, what latency data says so far, and how vLLM or SGLang could be adapted with route-template batching and virtual-step KV cache.

• How to Test Pretraining Ideas at Small Scale Before Betting on a Large Model

  29 Jun, 2026 · 25 min read

  A practical guide to validating pretraining improvements with small proxy models, scaling ladders, isoFLOP budgets, loss curves, downstream evals, and rank-correlation checks before committing to an expensive large-model run.

• Why Embeddings Cannot Solve Eval-Set Contamination

  29 Jun, 2026 · 11 min read

  A technical deep dive on why semantic embedding search is useful but insufficient for eval-set decontamination: leakage is about evaluation advantage, not just text similarity.

• Pretraining Contamination: Why Don't Train on the Test Set Became Hard

  29 Jun, 2026 · 14 min read

  A practical introduction to LLM pretraining contamination: why benchmark leakage is not ordinary deduplication, how public evals leak into web-scale corpora, and how layered decontamination pipelines reduce risk.

• How to Arbitrarily Increase the Difficulty of Agent Evaluation Sets

  28 May, 2026 · 18 min read

  A practical framework for making agent benchmarks harder in a controlled way: treat difficulty as trajectory-graph complexity, not prompt wording. Covers deterministic scoring, capability facets, harness effects, and systematic data generation.