What is KV Cache

Every time you send a message to an LLM, the server needs to process ALL previous tokens in the conversation. For a 10-turn chat, that's a lot of repeated work. KV Cache is the trick that makes this bearable.

Remember the usage block at the end of the last section? It said 4,736 of our 4,820 input tokens were "cache hits". In this section we'll find out what that actually means.

Analogy Think of it like watching a movie. Without cache, you'd rewind to the beginning every time you want to see the next scene. With cache, you just press play.

1. What Gets Cached

Inside the model, every token gets projected into three vectors: a Query, a Key, and a Value. Attention works by having each new token's Query look at the Keys and Values of every token before it.

Here's the useful part: a token's Key and Value depend only on that token and the tokens before it. They never change when new text is appended later. So they can be computed once and kept — that stored pile of K and V matrices is the KV Cache.

pseudocode

# Without cache — project every token, every request
for token in all_tokens:          # 4,820 tokens
    k, v = project(token)         # ← recomputed from scratch

# With cache — only the new tail does real work
for token in cached_prefix:       # 4,736 tokens
    k, v = cache.lookup(token)    # ← already on the GPU
for token in new_tail:            # 84 tokens
    k, v = project(token)         # ← the only real computation

The saving is not marginal. In that example the server did real work on 84 tokens instead of 4,820 — roughly 1.7% of the original cost.

2. How Prefix Matching Works

Now, how does the server know which tokens it has already seen? It keeps cached sequences in a tree, and it walks your new request against that tree from the very first token.

The rule is simple, and strict:

Start at token 0 and compare.
Keep walking while tokens match.
Stop at the first difference. Everything from there on is computed fresh.

This is called longest common prefix matching. The match has to be contiguous from the start — a chunk that matches in the middle of your request is worth nothing.

Let's watch it happen over three turns of a simple chat:

Turn 1 S₀ U₀

Turn 2 S₀ U₀ C₁ U₁

Turn 3 S₀ U₀ C₁ U₁ C₂ U₂

Served from cache Computed fresh

Turn 1 is a cold start: nothing to match, everything is computed. From then on, the green region grows and the yellow tail stays about the same size. That's the whole game.

Now here's what a bad request looks like. We changed one word in the system prompt:

Turn 3′ S₀′ U₀ C₁ U₁ C₂ U₂

Cache invalid — recomputed at full price

The mismatch happens at token 0, so the walk stops immediately and the entire request is recomputed. U₀, C₁, U₁ and the rest are byte-for-byte identical to last time, and it doesn't help them one bit — their position in the sequence changed the moment the prefix changed.

3. Block Alignment: Pages in a Book

One detail before we do arithmetic. The cache doesn't store tokens one at a time; it stores them in fixed-size blocks, commonly 64 tokens each. Think of pages in a book — you can bookmark a page, not a single word.

So a hit is always rounded down to a whole number of blocks:

python

BLOCK = 64
matching_prefix = 4820

usable_blocks = matching_prefix // BLOCK   # 75 blocks
cached_tokens = usable_blocks * BLOCK      # 4800 tokens
leftover      = matching_prefix % BLOCK    # 20 tokens, recomputed

Two practical consequences. First, a prefix shorter than 64 tokens caches nothing at all. Second, on long conversations the leftover is rounding noise — losing up to 63 tokens out of 4,800 doesn't change your bill.

4. Putting Numbers On It

Imagine a 3-turn conversation with an 800-token system prompt, 100-token user messages, and 200-token answers:

Turn	Input sequence	Total	Cached	Fresh	Hit rate
1	S₀ + U₀	900	0	900	0%
2	S₀ + U₀ + C₁ + U₁	1,200	900	300	75%
3	S₀ + U₀ + C₁ + U₁ + C₂ + U₂	1,500	1,200	300	80%
10	… seven more turns …	3,600	3,300	300	91.7%

Watch the two right-hand columns. The fresh work per turn is flat at 300 tokens, while the cached region keeps growing. Long conversations are where caching pays off most.

✏️ Think about it! If you change your system prompt between turns, what happens to the cache? (Hint: the entire prefix becomes invalid.) That's why "let me just append today's date to the system prompt" is a surprisingly expensive habit.

5. Why This Shows Up on Your Invoice

A cached token costs the provider almost nothing — the GPU work was done and paid for on an earlier request. Providers pass that through with a separate, much lower price.

Key insight This is why providers charge LESS for cached tokens — they cost less to compute.

Here is DeepSeek's off-peak pricing for deepseek-v4-flash, per million tokens:

Token type	Symbol	Price / M tokens	Relative
Input, cache hit	P_cr	0.05 ¥	1×
Input, cache miss	P_i	1.50 ¥	30× more
Output	P_o	4.50 ¥	90× more

Thirty times cheaper. Take turn 10 from the table above — 3,300 cached tokens. Across 1,000 requests, that prefix costs 0.165 ¥ at the cache price versus 4.95 ¥ at the full input price.

And you get latency for free: no computation means a faster first token, which is exactly the part users feel.

6. What Breaks the Cache

Anything that changes the token sequence at or before position k invalidates everything from k onward:

Editing the system prompt — even one character, even a timestamp.
Reordering or deleting earlier messages.
Context compression: summarizing old turns rewrites the prefix and resets you to a cold start.
Switching model or endpoint — different weights, different cache.
Eviction. GPU memory is finite and caches expire; under heavy traffic a long-idle conversation may simply be gone.

The pattern worth remembering: append-only conversations are cache-friendly; edited conversations are not.

7. Why Agent Mode Has an Advantage

This is where the last section connects. In Agent mode the history is strictly append-only — S₀, U₀, R₁, C₁, T₁, TR₁, U₁, R₂, … — and nothing already sent is ever rewritten. The prefix from the previous request is still a prefix of this one, so the match runs almost to the end.

Direct Chat is nearly append-only, but not quite, and the gap costs you. We'll see exactly where it leaks in the next section.

Recap

KV Cache stores the Key and Value matrices of past tokens on the GPU so they never have to be recomputed.
Matching is longest-common-prefix: it starts at token 0 and stops at the first difference.
Hits are rounded down to 64-token blocks, so very short prefixes gain nothing.
Cached input tokens cost about 1/30 of fresh ones, and arrive faster.
Anything that rewrites the prefix — especially context compression — resets you to a cold start.

So caching saves money. But how much exactly? That depends on the cache hit rate — which is what we'll calculate next.

什么是 KV Cache

你每发一条消息，服务端都得把这段对话里之前的所有 token 重新处理一遍。一个十轮对话，重复的活儿相当可观。KV Cache 就是让这件事变得可以接受的那个技巧。

还记得上一节结尾那个 usage 块吗？4820 个输入 token 里有 4736 个是「缓存命中」。这一节就来看看这句话到底是什么意思。

类比想象你在看电影。没有缓存的话，每想看下一个镜头都得先倒回片头重放一遍；有缓存，你直接按播放就行。

1. 被缓存的到底是什么

在模型内部，每个 token 都会被投影成三个向量：Query、Key 和 Value。注意力机制的工作方式，是让每个新 token 的 Query 去看它前面所有 token 的 Key 和 Value。

关键在这里：一个 token 的 Key 和 Value 只取决于它本身以及它前面的内容。后面再追加多少文字，它们都不会变。所以它们完全可以算一次然后留着 —— 存下来的这一堆 K、V 矩阵，就是 KV Cache。

伪代码

# 没有缓存 —— 每次请求都把所有 token 重新投影一遍
for token in all_tokens:          # 4820 个 token
    k, v = project(token)         # ← 从零重算

# 有缓存 —— 只有新增的尾巴需要真算
for token in cached_prefix:       # 4736 个 token
    k, v = cache.lookup(token)    # ← 已经在 GPU 上了
for token in new_tail:            # 84 个 token
    k, v = project(token)         # ← 唯一真正的计算

这个节省不是一点点。上面这个例子里，服务端真正干活的是 84 个 token，而不是 4820 个，大约是原来的 1.7%。

2. 前缀匹配是怎么工作的

那服务端怎么知道哪些 token 它见过？它把缓存过的序列组织成一棵树，然后拿你的新请求从第一个 token 开始，沿着这棵树往下走。

规则很简单，也很死板：

从第 0 个 token 开始比。
只要对得上就继续往下走。
在第一个不一样的地方停住。从那里往后全部重新计算。

这叫最长公共前缀匹配。匹配必须从头开始连续 —— 请求中间某一段恰好一样，一点用都没有。

拿一段三轮的普通对话看看它是怎么发生的：

第 1 轮 S₀ U₀

第 2 轮 S₀ U₀ C₁ U₁

第 3 轮 S₀ U₀ C₁ U₁ C₂ U₂

从缓存读取重新计算

第 1 轮是冷启动：没东西可匹配，全部都要算。从第 2 轮开始，绿色区域越来越长，而黄色的尾巴大小基本不变。整件事的窍门就在这儿。

再看一个糟糕的请求长什么样。我们只改了系统提示词里的一个词：

第 3′ 轮 S₀′ U₀ C₁ U₁ C₂ U₂

缓存失效，按全价重算

不匹配发生在第 0 个 token，所以匹配立刻中断，整个请求全部重算。U₀、C₁、U₁ 这些内容跟上一次一个字节都没差，也救不了它们 —— 前缀一变，它们在序列里的位置就全变了。

3. 块对齐：书里的页

算数之前还有个细节。缓存不是一个 token 一个 token 存的，而是按固定大小的块存，常见是 64 个 token 一块。可以想成书里的页：你能夹一个书签标到某一页，但标不到某一个字。

所以命中长度总是向下取整到整数个块：

python

BLOCK = 64
matching_prefix = 4820

usable_blocks = matching_prefix // BLOCK   # 75 块
cached_tokens = usable_blocks * BLOCK      # 4800 个 token
leftover      = matching_prefix % BLOCK    # 20 个 token，要重算

两个实际后果。第一，不到 64 个 token 的前缀，一点缓存都吃不到。第二，长对话里这个零头就是舍入噪声 —— 4800 里最多丢 63 个，账单上看不出来。

4. 把数字填进去

假设一段三轮对话：系统提示词 800 token，每条用户消息 100 token，每条回答 200 token：

轮次	输入序列	总量	命中	新算	命中率
1	S₀ + U₀	900	0	900	0%
2	S₀ + U₀ + C₁ + U₁	1,200	900	300	75%
3	S₀ + U₀ + C₁ + U₁ + C₂ + U₂	1,500	1,200	300	80%
10	…… 再走七轮 ……	3,600	3,300	300	91.7%

盯着右边两列看。每轮新算的量一直是 300 token，而命中的区域越来越大。对话越长，缓存的收益越明显。

✏️ 想一想！ 如果你在两轮之间改了系统提示词，缓存会怎样？（提示：整个前缀都失效了。）这就是为什么「顺手在系统提示词里塞一个今天的日期」是个出乎意料地贵的习惯。

5. 为什么这会出现在账单上

一个命中缓存的 token，对服务商来说几乎不花成本 —— GPU 的活在更早的某次请求里就已经干完并且收过钱了。这部分省下来的成本，服务商用一个单独的、低得多的价格传导给你。

关键结论 这就是为什么服务商对命中缓存的 token 收费更低 —— 它们的计算成本本来就低。

下面是 DeepSeek deepseek-v4-flash 空闲时段的价格，单位是每百万 token：

token 类型	符号	单价 / 百万 token	相对倍数
输入，缓存命中	P_cr	0.05 元	1×
输入，缓存未命中	P_i	1.50 元	贵 30 倍
输出	P_o	4.50 元	贵 90 倍

便宜三十倍。拿上面表里第 10 轮的 3300 个命中 token 来算：跑 1000 次请求，这段前缀按缓存价是 0.165 元，按全价输入是 4.95 元。

而且延迟是白送的：不用计算意味着首个 token 出来得更快，而这恰好是用户能直接感觉到的那部分。

6. 什么会打破缓存

任何在位置 k 及其之前改动了 token 序列的操作，都会让 k 往后的全部失效：

改系统提示词 —— 哪怕一个字符，哪怕只是个时间戳。
重排或删掉靠前的消息。
上下文压缩：对旧轮次做摘要就是在重写前缀，等于回到冷启动。
换模型或换端点 —— 权重不同，缓存也不是同一份。
被驱逐。GPU 显存有限，缓存也会过期；高流量时段，一个闲置太久的对话可能就是没了。

值得记住的规律：只追加的对话对缓存友好，被编辑过的对话则不然。

7. 为什么 Agent 模式占优

这里就接上了上一节。Agent 模式的历史是严格只追加的 —— S₀、U₀、R₁、C₁、T₁、TR₁、U₁、R₂、…… —— 已经发出去的东西永远不会被改写。上一次请求的整段内容仍然是这一次的前缀，所以匹配几乎能走到末尾。

直聊模式接近只追加，但差了那么一点，而这一点是要花钱的。漏在哪儿，下一节精确算给你看。

小结

KV Cache 把历史 token 的 Key、Value 矩阵存在 GPU 上，避免重算。
匹配方式是最长公共前缀：从第 0 个 token 开始，在第一个差异处停住。
命中长度向下取整到 64-token 的块，所以特别短的前缀吃不到收益。
命中缓存的输入 token 大约是全价的 1/30，而且返回更快。
任何重写前缀的操作 —— 尤其是上下文压缩 —— 都会把你打回冷启动。

所以缓存能省钱。但到底能省多少？这取决于缓存命中率，我们下一节就来算它。