PQlet blog

[WIP] How exactly DeepSeek-V4 uses less VRAM for KV-Cache ? [SGLang]

GitHub repo | Issue in SGLang repo

TODO: run experimental measurement after Issue is closed.


Q1: What does this repo show?
A1: A theoretical KV-cache calculation matching DeepSeek-V4’s Figure 1. At 1,048,576 tokens, V4-Flash needs 3.43 GiB, versus 46.94 GiB for V3.2: 13.67× smaller. No experimental validation of memory usage.

Q2: Where do the savings come from?
A2: Mostly, fewer cache entries. Also, fewer bytes per entry. V4-Flash has 21 layers compressing ×4 (CSA), 20 compressing ×128 (HCA), and two using only sliding-window attention. All 43 retain a 128-token local window. Main entries use FP8 non-RoPE + BF16 RoPE + scales: 584 bytes. The theoretical indexer uses FP4: 68 bytes/entry.

Q3: What is the calculation?
A3: For context length N ≥ 128, bytes per cache copy:

V3.2’s released config supports 163,840 tokens. Its 1M result means “what its cache would cost if extended to 1M,” not that the released model supports that length.

For V4-Flash:

Q4: Per GPU or across all GPUs?
A4: Per GPU in this TP setup. V4 uses one shared KV head: “Shared Key-Value MQA. After selecting the sparse KV entries, CSA then performs core attention in a Multi-Query Attention (MQA) (Shazeer, 2019) manner.” Each rank stores the full cache to compute its heads locally. Sharding that cache would require extra communication and a different attention implementation. At TP=8, theoretical 3.43 GiB/GPU becomes 27.48 GiB total. Attention equations.

Q5: Did SGLang measurements confirm this?
A5: No. SGLang lacks functioning V4 KV-memory reporting. Its pools leave mem_usage=0, so the exported metric returns zero. nvidia-smi cannot isolate request KV usage inside preallocated pools. It does not mean the VRAM savings are not implemented in practice; only the reporting is not implemented. I reported this in #37852.

Q6: What about the experimental memory tables in your repo?
A6: They are not measured KV-memory results. They convert token counts into bytes using assumptions; earlier versions also used the wrong multiplier. They do not verify Figure 1. The repo’s result is the theoretical reproduction = calculation of mem usage.

Q7: Can any implementation expose exact KV-cache memory usage?
A7: Yes — the naive reference implementation on the official HF Hub page of DS-V4 exposes plain PyTorch cache tensors. The KV-Cache tensor is pre-allocated fully before the inference (see here). Their allocated bytes can be counted directly with numel() × element_size(), counting shared storage once. Its caches use BF16 with simulated FP8/FP4 quantization, so those measured bytes describe this implementation, not the paper’s packed-cache layout or SGLang’s allocation. Code.