Tag: Scaling Laws

All the articles with the tag "Scaling Laws".

• A Looped Transformer Router Shows Its First Replicated Gain

  6 Jul, 2026 · 10 min read

  A small-budget BPE language-model experiment where a sparse late-final-loop token-feedback router becomes the first route-looped Transformer candidate to beat matched fixed-loop baselines across several controlled checks.

• When a Looped Transformer Router Almost Works

  1 Jul, 2026 · 12 min read

  A controlled small-scale language-composition experiment comparing fixed and routed looped Transformers, showing why the first sequence-level router was competitive but collapsed toward a weak exit policy.

• How to Test Pretraining Ideas at Small Scale Before Betting on a Large Model

  29 Jun, 2026 · 25 min read

  A practical guide to validating pretraining improvements with small proxy models, scaling ladders, isoFLOP budgets, loss curves, downstream evals, and rank-correlation checks before committing to an expensive large-model run.