{ "id": "pct-2026-11-04-spotlight-mooncake-store", "title": "Mooncake Store: putting a price on repeated work", "section": "Product spotlight", "publisher": "Peterson Compute Trading", "as_of_date": "2026-11-04", "status": "desk_research", "subject": "Mooncake Store and vLLM shared KV caching", "editorial_word_count": 558, "facts": [ { "id": "F1", "claim": "Mooncake paper first submitted June 24, 2024; current cited revision dated October 8, 2026.", "source_ids": [ "S1" ], "evidence_type": "dated_primary_research" }, { "id": "F2", "claim": "Prefix reuse reduces prefill, not direct decode time.", "source_ids": [ "S2" ], "evidence_type": "primary_documentation" }, { "id": "F3", "claim": "MooncakeStoreConnector supports cross-instance caching and CPU/disk offload; strict isolation requires configured multi-tenancy.", "source_ids": [ "S3" ], "evidence_type": "primary_documentation" }, { "id": "F4", "claim": "May 6 authors report 3.8× throughput, 46× lower median TTFT and 8.6× lower E2E latency on 12 GB200s; separate synthetic scaling test reaches 60 GPUs.", "source_ids": [ "S4" ], "evidence_type": "author_reported_benchmark_not_independently_reproduced" }, { "id": "F5", "claim": "vLLM hashes cache identity with prefix and extra inputs; request salts restrict reuse.", "source_ids": [ "S5" ], "evidence_type": "primary_documentation" }, { "id": "F6", "claim": "Mooncake repository identifies Apache-2.0 licensing.", "source_ids": [ "S6" ], "evidence_type": "repository_license" } ], "inferences": [ { "id": "I1", "claim": "The procurement comparison should include shared-cache infrastructure and completed requests meeting service constraints, not GPU hourly rates or TTFT alone.", "basis_fact_ids": [ "F2", "F3", "F4" ], "label": "PCT analysis" }, { "id": "I2", "claim": "If useful hourly throughput doubles and total hourly infrastructure cost rises 25%, cost per completion falls 37.5%.", "basis_fact_ids": [], "label": "illustrative arithmetic, not observed benchmark", "assumptions": [ "Same task mix and required quality/latency constraints.", "Throughput multiplier is 2.0.", "All-in hourly infrastructure cost multiplier is 1.25.", "Ratio is 1.25 / 2.0 = 0.625; reduction is 1 - 0.625 = 0.375." ] }, { "id": "I3", "claim": "A distributed cache creates value only when avoided recomputation and service improvements justify retention, retrieval and operational costs.", "basis_fact_ids": [ "F2", "F3", "F4" ], "label": "PCT analysis" } ], "unknowns": [ "No independent benchmark reproduction, interviews, deployment or supplier quote was performed.", "No measured all-in dollar cost, power consumption or cost-per-task saving was established.", "The May benchmark does not establish equivalent gains for another model, GPU, cache configuration, request mix or latency target.", "Latest documentation is mutable and is not an immutable historical release snapshot.", "Source publication dates are null where no original publication date was established." ], "corrections": [ { "status": "prepublication_guardrail", "text": "Do not label 46× as throughput, GPU savings or cost savings; it is the authors' median TTFT result." }, { "status": "prepublication_guardrail", "text": "Do not conflate the 12-GPU real-trace experiment with the 60-GPU synthetic scaling experiment." }, { "status": "prepublication_guardrail", "text": "Do not repeat the May 6 roadmap as current feature status; November 4 documentation describes disk offload." }, { "status": "prepublication_guardrail", "text": "Original Mooncake research publication, May 6 vLLM integration benchmark and today's documentation observation are distinct dates." } ], "license": { "editorial_text": "Original PCT synthesis. Publisher controls the license assigned to this original editorial text; no new public license is asserted here.", "third_party_material": "No third-party image, chart, code or extended quotation reproduced. Primary facts are paraphrased and attributed.", "software": "Mooncake repository code is Apache-2.0; this does not establish the license for all linked blog and paper content.", "republication_note": "Keep source links and benchmark qualifications with any republication." }, "sources": [ { "id": "S1", "title": "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving", "publisher": "Ruoyu Qin et al., arXiv", "published_at": "2024-06-24", "updated_at": "2026-10-08", "retrieved_at": "2026-11-04", "url": "https://arxiv.org/abs/2407.00079v5", "primary": true, "license": "arXiv non-exclusive distribution license; no blanket downstream republication grant assumed" }, { "id": "S2", "title": "Automatic Prefix Caching: features and limits", "publisher": "vLLM", "published_at": null, "page_date": "2026-10-13", "page_date_label": "Displayed footer date; treated as page date, not inferred original publication", "retrieved_at": "2026-11-04", "url": "https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/", "primary": true }, { "id": "S3", "title": "MooncakeStoreConnector Usage Guide", "publisher": "vLLM", "published_at": null, "page_date": "2026-11-04", "page_date_label": "Displayed footer date; treated as page date, not inferred original publication", "retrieved_at": "2026-11-04", "url": "https://docs.vllm.ai/en/latest/features/mooncake_store_connector_usage/", "primary": true }, { "id": "S4", "title": "Serving Agentic Workloads at Scale with vLLM x Mooncake", "publisher": "vLLM", "authors": [ "Yifan Qiao", "Trong Dao Le", "Ao Shen", "Zhewen Li", "Bowen Wang" ], "published_at": "2026-05-06", "retrieved_at": "2026-11-04", "url": "https://vllm.ai/blog/2026-05-06-mooncake-store", "primary": true, "license": "Page footer states all rights reserved; only original summary and attributed facts used" }, { "id": "S5", "title": "Automatic Prefix Caching: design", "publisher": "vLLM", "published_at": null, "page_date": "2026-06-23", "page_date_label": "Displayed footer date; treated as page date, not inferred original publication", "retrieved_at": "2026-11-04", "url": "https://docs.vllm.ai/en/latest/design/prefix_caching/", "primary": true }, { "id": "S6", "title": "Mooncake Apache License", "publisher": "kvcache-ai / Mooncake contributors", "published_at": null, "retrieved_at": "2026-11-04", "url": "https://github.com/kvcache-ai/Mooncake/blob/main/LICENSE-APACHE", "primary": true, "license": "Apache-2.0 (repository code)" } ], "benchmark_scope": { "reported_by": "vLLM authors", "publication_date": "2026-05-06", "model": "Kimi-2.5 NVFP4", "hardware": "12 GB200 GPUs", "topology": "1 prefill instance and 1 decode instance; prefill TP4, decode DP8 plus EP", "workload": "Codex coding-agent traces described in source S4", "independent_reproduction": false, "reported_metrics": { "throughput_multiplier": 3.8, "median_ttft_speedup": 46, "end_to_end_latency_speedup": 8.6, "baseline_cache_hit_pct": 1.7, "store_cache_hit_pct": 92.2 }, "separate_scaling_experiment": "Synthetic workload based on traces, scaled from 12 to 60 GB200 GPUs." }, "agent_fields": [ "model_and_revision", "serving_engine_revision", "cache_connector_revision", "hardware_and_gpu_count", "prefill_decode_topology", "cache_capacity_by_tier", "tenant_isolation_policy", "request_mix", "routing_policy", "cache_warmth", "prefix_reuse_rate", "transfer_latency", "eviction_rate", "p50_ttft", "p99_ttft", "end_to_end_latency", "completed_requests_meeting_slo_per_hour", "total_infrastructure_cost_per_hour", "currency", "source_url", "retrieved_at" ], "body": "A coding agent that returns after running a tool may bring back most of the context it submitted before. Recomputing that history consumes capacity without adding new information. Mooncake Store turns previously computed attention state into a resource that multiple inference instances can reuse. For compute buyers, it raises a practical question: how much of the workload being priced is genuinely new work?\n\nThe project comes from Mooncake, the serving architecture developed for Moonshot AI's Kimi. Its research paper, first submitted on June 24, 2024 and revised on October 8, 2026, describes separate prompt-processing and token-generation clusters, with a shared cache drawing on CPU, DRAM and SSD resources. That puts memory placement and scheduling alongside accelerator performance in the serving design. [S1]\n\nIn vLLM, ordinary prefix caching reuses the attention keys and values associated with an already-processed prompt prefix. It saves prompt-processing work, called prefill; it does not directly accelerate generation of new output tokens, called decode. Long answers with little repeated input therefore have a different opportunity from repeated questions against a large shared document. [S2]\n\nMooncakeStoreConnector extends reuse beyond one instance. Current vLLM documentation describes a shared external store, CPU and disk offloading, and both single-node and multi-node configurations. A separate MooncakeConnector transfers state directly between prefill and decode workers. The distinction matters when specifying a deployment: point-to-point transfer and retained, shared cache capacity solve different problems. The store can be embedded in workers or hosted separately; its documented transport choices include RDMA and a TCP fallback. [S3]\n\nThe dated performance evidence is striking but bounded. In a May 6, 2026 vLLM report, the authors tested Kimi-2.5 NVFP4 with separate prefill and decode on 12 GB200 GPUs using coding-agent traces. They reported 3.8 times the throughput, a 46-fold reduction in median time to first token, and an 8.6-fold reduction in end-to-end latency. Cache hits rose from 1.7% to 92.2%. These are the authors' results, not PCT measurements. Their separate scaling experiment used a synthetic workload and reached 60 GPUs; it should not be described as the same trace experiment. [S4]\n\nPCT's economic reading is conditional. A faster first token does not establish an equivalent reduction in cost per completed task. Suppose a deployment doubles useful completions per hour while its total hourly infrastructure cost rises by 25%. Cost per completion becomes 1.25 divided by two, or 62.5% of the starting figure: a 37.5% reduction. This is illustrative arithmetic, not a Mooncake result. Storage, networking, CPU work and operations belong in the numerator. Completed requests meeting the required latency and quality targets belong in the denominator.\n\nReuse also has boundaries. vLLM's cache identifiers incorporate the preceding prefix and relevant distinctions such as adapters and multimodal inputs. Its design includes per-request salts to restrict sharing. Mooncake's current guide separately states that strict tenant isolation requires a multi-tenant master and registered tenant quota policy. A shared pool should not be assumed to provide the required isolation merely because it is reachable. [S5][S3]\n\nBefore assigning monetary value to a cache hit, replay representative traffic with cold and warm caches, realistic tool delays and the intended routing policy. Measure skipped prefill, transfer time, evictions, tail latency and total cost together. Test restart and failure behavior too. PCT's conclusion: shared state can substitute for repeated accelerator work, but the trade only pays when retaining and retrieving it is cheaper than recomputing it under the service's actual constraints.", "content_type": "product_spotlight", "publication_date": "2026-11-04", "observation_date": "2026-11-04", "editorial_voice": "Spotlight Desk | AI-assisted, buyer-focused", "license_status": "Private original PCT prose; public license undecided. External sources retain their own rights.", "public_url": null, "public_projection": { "version": "1.0.0", "date": "2026-11-10", "changes": [ "Internal workspace artifact paths omitted where present. Historical license/status fields retain original provenance; current public reuse policy is stated separately." ], "rights": "PCT reserves rights in original editorial content; third-party rights retained. No general source-data reuse license." } }