Why AWS, Kimi, and DeepSeek Are All Splitting Prefill from Decode
胡新宇
Published on 2026-07-11
Three vendors shipped LLM inference disaggregation (prefill/decode split) in four months. Here's the pattern, the economics, and when it actually pays off vs colocation.
Why AWS, Kimi, and DeepSeek Are All Splitting Prefill from Decode
On July 10, 2026, AWS published the engineering blueprint for splitting LLM inference into two GPU pools. Within four months, the same pattern showed up in Kimi's cross-region paper and DeepSeek's latest speculative-decoding work. The disaggregation revolution isn't coming. It's already deployed in production at three of the most-watched model providers on the planet.
This is the story of why every serious LLM vendor is converging on the same architectural pattern, what it actually costs to run, and where it falls apart.
The Two-Phase Problem That's Been Eating GPUs
Every LLM inference request goes through two phases. They behave nothing alike.
Prefill is the compute-bound phase. It reads the entire prompt — could be 500 tokens, could be 50,000 — and computes attention across the whole sequence in parallel. The GPU is busy. Every FLOP is working. Prefill builds the KV cache that the next phase will use.
Decode is the memory-bound phase. It generates one token at a time. Each new token needs to read the entire model weights and the growing KV cache from HBM. The arithmetic units sit idle. What matters is memory bandwidth.
On a single GPU handling both phases, a long prompt kills everyone else's latency. Drop a 10K-token document into a chat endpoint, and every concurrent user waits while the prefill kernel grinds through the input. This is , and it's haunted production inference stacks for two years.
LLM Inference Disaggregation: AWS, Kimi, DeepSeek Compared
head-of-line blocking
The fix is obvious in hindsight: don't share the GPU. Run prefill on one pool, decode on another, ship the KV cache between them.
The execution is harder.
Three Vendors, Three Scale-Out Strategies
What makes the current moment interesting is that three independent teams — AWS, Kimi/Moonshot AI, and the DeepSeek community — all shipped disaggregation work in a four-month window. They solved the same problem three different ways.
AWS: Same-AZ Disaggregation on SageMaker HyperPod
AWS's July 10 blog post lays out the cleanest reference architecture (source). The system has four components:
An intelligent router that tokenizes each prompt, applies a configurable threshold, and decides whether to disaggregate or send the request end-to-end to a decoder.
A prefiller pod running vLLM with LMCache as the KV connector. It computes KV cache layer-by-layer and pushes it to the decoder while overlapping compute with transfer. An L1 CPU cache short-circuits repeated prefixes (system prompts, multi-turn history).
A decoder pod running vLLM with LMCache as the receiver. It reserves a PD buffer in GPU memory and starts generation the moment the transfer completes.
A four-layer transfer stack: LMCache PD → NIXL → libfabric → EFA. NIXL picks the right RDMA operation, libfabric exposes EFA as kernel-bypass GPU-Direct RDMA.
The numbers: on ml.p5.48xlarge instances with 3,200 Gbps EFA, transferring the KV cache for an 8,000-token Llama 3.3 70B prompt takes single-digit milliseconds. Compared to the prefill compute itself, that's noise.
The key design choice: AWS keeps prefill and decode in the same Availability Zone because EFA's high-bandwidth mode requires it. Cross-AZ traffic falls back to TCP, which kills the win.
Kimi: Cross-Region Disaggregation via Commodity Ethernet
Kimi's April 2026 paper (source), co-authored with Tsinghua, takes the opposite bet. They call it Prefill-as-a-Service (PrFaaS).
The premise: RDMA is great, but it locks you to one data center. The real cost optimization is shipping prefill to wherever GPUs are cheap and decode to wherever GPUs are scarce. Kimi's hybrid attention model — a 1T-parameter beast — measured the following on internal benchmarks:
54% throughput gain vs colocated prefill-decode.
64% P90 latency reduction for long-context workloads.
32% throughput gain vs a naive cross-data-center scheme without smart scheduling.
The catch: cross-data-center KV cache transfer only consumed 13 Gbps of bandwidth. That's well below commodity Ethernet's 100 Gbps ceiling. Kimi proved you don't need RDMA for this to work — you need a scheduler smart enough to batch transfers intelligently.
This is the more interesting result. AWS's approach requires P5 or P6 instances and same-AZ placement. Kimi's approach runs on anything with a network cable.
DeepSeek + P-EAGLE: Speculative Decoding as a Decoder Multiplier
Speculative decoding doesn't disaggregate prefill from decode. It disaggregates draft generation from verification. A small draft model proposes K tokens; the large model verifies them in one pass. Net result: decode phase becomes 2-4x faster on average.
AWS's June 2026 post on P-EAGLE (source) reported 2.4x throughput on Llama-3.1-8B. DeepSeek's variant pushes this further on their own model family.
Speculative decoding is the complementary fix, not the primary one. It attacks decode-side memory bandwidth, which disaggregation alone doesn't help. The vendors who win the cost war will combine both: disaggregate prefill from decode, then run speculative decoding inside the decode pool.
Why Now: The Economics Finally Closed
The disaggregation pattern has been discussed in papers since 2024. None of it mattered for production until mid-2026. Three things changed:
RDMA silicon matured. AWS EFA, Mellanox RoCE, and Broadcom's Tomahawk 5 all crossed the 3,200 Gbps threshold. Transfers that took 50ms in 2024 take 3ms now.
The vLLM + LMCache + NIXL stack composes. Before LMCache, every vendor wrote their own KV transfer layer. LMCache standardized it. NIXL unified the memory abstraction across GPU, CPU, and remote peers.
Inference economics broke. Training compute is dominated by frontier model runs. Inference compute is dominated by latency-sensitive long-context workloads — agent loops, RAG over 50-page documents, coding assistants with 30K-token repositories. These workloads make prefill heavy and decode light, exactly the regime where colocation wastes the most GPU time.
The breakthrough isn't an algorithm. It's that the network, the open-source stack, and the workload mix all arrived at the same place in 2026.
When Disaggregation Wins, When It Loses
Not every workload should disaggregate. AWS's threshold heuristic is the cleanest rule of thumb: disaggregate when prompts regularly exceed 4,096 tokens and you have multiple concurrent users. Below that threshold, the fixed cost of transferring KV cache over EFA RDMA outweighs the benefit.
Translated to production:
Workload
Win disaggregation?
Long-context chat assistants (10K+ tokens)
Yes — prefill regularly stalls decode
RAG with large retrieved contexts
Yes — same shape
Agentic pipelines with 50-step traces
Yes — prefill dominates latency budget
Short-prompt chatbots (<1K tokens)
No — transfer overhead > prefill savings
Offline batch inference
No — throughput is what matters, not tail latency
Low-concurrency deployments (<10 streams)
No — GPU contention isn't the bottleneck
The hidden cost is the router. It tokenizes every prompt and applies a threshold check. That's a few hundred microseconds of overhead per request — negligible at scale, painful at low TPS.
There's also LMCache sizing. Each prefiller holds an L1 CPU cache for prefix hits. Get the size wrong and you either waste RAM or miss the cache and recompute prefill. AWS's reference values give starting points, but production tuning is workload-specific.
What This Means for Builders
If you're buying inference, the disaggregation wave is your leverage. AWS, Kimi, DeepSeek, and most likely OpenAI's upcoming Broadcom silicon are all converging here. By Q4 2026, expect every major provider to ship disaggregation as a configurable option. The right question to ask vendors isn't "do you support long context" — it's "what's your P90 latency on a 10K-token prompt under 50 concurrent users."
If you're building your own inference stack, the vLLM + LMCache + NIXL stack is now the lowest-risk starting point. Don't write your own KV transfer layer. Don't roll your own router. The open-source pieces work and compose.
If you're running inference on a single H100 or H200 node with low concurrency, do nothing. Disaggregation only earns its complexity at scale.
The Database Analogy
The pattern is the same one the database industry went through fifteen years ago.
In 2008, Oracle and MySQL ran compute and storage on the same node. By 2015, the entire industry split compute from storage — Snowflake, BigQuery, Aurora. The companies that made that transition owned the next decade of cloud infrastructure.
LLM inference is hitting the same wall. The compute nodes (GPU pools) and the state nodes (KV cache) want different hardware, different networks, and different scaling curves. Splitting them is inevitable. The only question is who ships the cleanest abstraction first.
AWS, Kimi, and DeepSeek all just answered that question with code. Everyone else is going to copy the pattern within twelve months.
The cost of intelligence is going to drop. The vendors who master disaggregation will be the ones who drop it the most.
