Tag: Inference

All the articles with the tag "Inference".

• Optimizing Inference for Router Looped Transformers

  Updated: 29 Jun, 2026 · 20 min read

  A research note on serving router looped transformers: why normal KV cache semantics break, what latency data says so far, and how vLLM or SGLang could be adapted with route-template batching and virtual-step KV cache.

• How to Make LLM Inference Faster

  19 Aug, 2023 · 4 min read

  An overview of LLM inference optimization techniques including KV cache, FlashAttention, and memory management strategies.