Mooncake Store: putting a price on repeated work

A coding agent that returns after running a tool may bring back most of the context it submitted before. Recomputing that history consumes capacity without adding new information. Mooncake Store turns previously computed attention state into a resource that multiple inference instances can reuse. For compute buyers, it raises a practical question: how much of the workload being priced is genuinely new work?

The project comes from Mooncake, the serving architecture developed for Moonshot AI's Kimi. Its research paper, first submitted on June 24, 2024 and revised on October 8, 2026, describes separate prompt-processing and token-generation clusters, with a shared cache drawing on CPU, DRAM and SSD resources. That puts memory placement and scheduling alongside accelerator performance in the serving design. S1

In vLLM, ordinary prefix caching reuses the attention keys and values associated with an already-processed prompt prefix. It saves prompt-processing work, called prefill; it does not directly accelerate generation of new output tokens, called decode. Long answers with little repeated input therefore have a different opportunity from repeated questions against a large shared document. S2

MooncakeStoreConnector extends reuse beyond one instance. Current vLLM documentation describes a shared external store, CPU and disk offloading, and both single-node and multi-node configurations. A separate MooncakeConnector transfers state directly between prefill and decode workers. The distinction matters when specifying a deployment: point-to-point transfer and retained, shared cache capacity solve different problems. The store can be embedded in workers or hosted separately; its documented transport choices include RDMA and a TCP fallback. S3

The dated performance evidence is striking but bounded. In a May 6, 2026 vLLM report, the authors tested Kimi-2.5 NVFP4 with separate prefill and decode on 12 GB200 GPUs using coding-agent traces. They reported 3.8 times the throughput, a 46-fold reduction in median time to first token, and an 8.6-fold reduction in end-to-end latency. Cache hits rose from 1.7% to 92.2%. These are the authors' results, not PCT measurements. Their separate scaling experiment used a synthetic workload and reached 60 GPUs; it should not be described as the same trace experiment. S4

PCT's economic reading is conditional. A faster first token does not establish an equivalent reduction in cost per completed task. Suppose a deployment doubles useful completions per hour while its total hourly infrastructure cost rises by 25%. Cost per completion becomes 1.25 divided by two, or 62.5% of the starting figure: a 37.5% reduction. This is illustrative arithmetic, not a Mooncake result. Storage, networking, CPU work and operations belong in the numerator. Completed requests meeting the required latency and quality targets belong in the denominator.

Reuse also has boundaries. vLLM's cache identifiers incorporate the preceding prefix and relevant distinctions such as adapters and multimodal inputs. Its design includes per-request salts to restrict sharing. Mooncake's current guide separately states that strict tenant isolation requires a multi-tenant master and registered tenant quota policy. A shared pool should not be assumed to provide the required isolation merely because it is reachable. S5S3

Before assigning monetary value to a cache hit, replay representative traffic with cold and warm caches, realistic tool delays and the intended routing policy. Measure skipped prefill, transfer time, evictions, tail latency and total cost together. Test restart and failure behavior too. PCT's conclusion: shared state can substitute for repeated accelerator work, but the trade only pays when retaining and retrieving it is cheaper than recomputing it under the service's actual constraints.