Speaker: Lequn Chen Sources https://www.perplexity.ai/hub/blog/lower-latency-and-higher-throughput-with-multi-node-deepseek-deployment https://github.com/ppl-ai/pplx-kernels 1. SetupMultiple nodes t

IntroThis article is compiled from a livestream.The guest speaker is Fuzhao Xue, a Google Deepmind Senior Research Scientist and the author of OpenMoE Main research areas: Gemini Pretraining, Model A

Official V1 blog https://blog.vllm.ai/2025/01/27/v1-alpha-release.html Why V1? V0 is slow: CPU overhead is high V0 is hard to read and develop e.g., V0 scheduler is 2k LOC, V1 is 800 LOC V0 code decou

KV-cache-aware routing in multi-host servinghttps://github.com/vllm-project/production-stack/issues/59 https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/498 Solution 1 Use stri

Why Speculative Decoding (SD)? Decoding is memory-bound: loading KV cache and model takes a long time memory-bound cases: big matrix * small matrix; vector * matrix → O(n^2) compute-bound cases: lar

Why P/D disaggregation? Initial scheduler logic in vLLM: prioritize prefill for good throughput Problem: prefill may slow down other requests’ decode How to mix P and D together? Well, even thei