Cache Hit Rate

Now we know the server tries to reuse previous computations. But how much does it actually reuse? That's the cache hit rate — and it's different for our two API modes.

Hit Rate in One Line

Let's start with the intuition. The hit rate is just a ratio of two token counts:

$$\text{HitRatio} = \frac{\text{tokens reused from cache}}{\text{total input tokens}} = 1 - \frac{\text{new tokens}}{\text{total input tokens}}$$

The second form is the useful one. Counting what's new is easy — it's whatever we appended since the last request. So the whole job is: figure out what got appended, and divide.

And "what got appended" is exactly where the two modes part ways.

Agent Mode: Only the Tail Is New

In Agent mode we keep everything, reasoning included. So the conversation grows by pure appending:

Turn n−1 S₀ U₀ … R₍ₙ₋₁₎ C₍ₙ₋₁₎ T₍ₙ₋₁₎ TR₍ₙ₋₁₎ U₍ₙ₋₁₎

Turn n S₀ U₀ … R₍ₙ₋₁₎ C₍ₙ₋₁₎ T₍ₙ₋₁₎ TR₍ₙ₋₁₎ U₍ₙ₋₁₎ Uₙ

Already in the cache New this turn

Notice: each new turn only adds Uₙ at the end. Everything before it is identical to last time, byte for byte, in the same positions.

So the hit rate is simply one minus new stuff over total stuff:

$$\text{HitRatio}_{\text{agent}}(n) = 1 - \frac{U_n}{S_0 + U_0 + \sum_{i=1}^{n-1}(R_i + C_i + T_i + TR_i + U_i) + U_n}$$

The numerator is a single user message and stays roughly constant. The denominator grows by a few thousand tokens every turn. That's why the ratio climbs — the miss doesn't grow, the total does.

Quick check: turn 10, agent mode

With S₀ = 800, U = 100, and R + C + T + TR = 930 per turn, the input at turn 10 is 900 + 9 × 1030 + 100 = 10,270 tokens. The miss is just the 100-token Uₙ:

Total input at turn 10	10,270
New (cache miss)	100
Hit rate	99.0%

Direct Chat: Where the Cache Leaks

In Direct Chat, we strip reasoning. But that creates a problem — and it's a subtle one, because it's not about the size of the history. It's about positions.

Think about what the GPU held at the end of turn n−1. The server processed our input, then generated its reasoning and then its answer. Generated tokens land in the cache too, in the order they were produced:

In cache S₀ … U₍ₙ₋₁₎ R₍ₙ₋₁₎ C₍ₙ₋₁₎

Now look at what we send for turn n. The reasoning is gone, so C₍ₙ₋₁₎ has slid forward to sit immediately after U₍ₙ₋₁₎:

We send S₀ … U₍ₙ₋₁₎ C₍ₙ₋₁₎ Uₙ

Matches the cache Stripped by the server Position shifted — no longer matches

Walk the prefix match by hand. S₀ matches, the middle matches, U₍ₙ₋₁₎ matches. Then the cache expects R₍ₙ₋₁₎ and gets C₍ₙ₋₁₎ instead. The server sees a mismatch at C₍ₙ₋₁₎'s position, and cache breaks there.

So the miss is two blocks, not one: the previous answer plus the new question.

$$\text{HitRatio}_{\text{chat}}(n) = 1 - \frac{C_{n-1} + U_n}{S_0 + U_0 + \sum_{i=1}^{n-1} C_i + \sum_{i=1}^n U_i}$$

Two things follow from comparing the formulas. Direct Chat has a bigger numerator (C₍ₙ₋₁₎ + Uₙ instead of just Uₙ) and a smaller denominator, since all the reasoning tokens are missing from the history. Both push the ratio down.

The irony Direct Chat sends fewer tokens per request, which sounds cheaper. But a larger share of those tokens is billed at the full input price — up to 30× the cache rate. Fewer tokens, worse mix.

See It Move

Formulas are easier to trust once you've poked at them. The simulator below evaluates both formulas live. Drag a slider and watch which curve reacts.

缓存命中率

我们已经知道服务端会尽量复用之前的计算。但它实际复用了多少？这就是缓存命中率 —— 而两种 API 模式的答案很不一样。

一行话说清命中率

先建立直觉。命中率就是两个 token 数量的比值：

$$\text{命中率} = \frac{\text{从缓存复用的 token}}{\text{总输入 token}} = 1 - \frac{\text{新增的 token}}{\text{总输入 token}}$$

第二种写法才好用。数新增的部分很容易 —— 就是上次请求之后我们追加的那些。所以整件事就剩：搞清楚追加了什么，然后做除法。

而「追加了什么」，正是两种模式分道扬镳的地方。

Agent 模式：只有尾巴是新的

Agent 模式什么都留着，推理也留着。所以对话是纯追加式增长的：

第 n−1 轮 S₀ U₀ … R₍ₙ₋₁₎ C₍ₙ₋₁₎ T₍ₙ₋₁₎ TR₍ₙ₋₁₎ U₍ₙ₋₁₎

第 n 轮 S₀ U₀ … R₍ₙ₋₁₎ C₍ₙ₋₁₎ T₍ₙ₋₁₎ TR₍ₙ₋₁₎ U₍ₙ₋₁₎ Uₙ

已经在缓存里这一轮新增

注意：每一轮新增的只有末尾的 Uₙ。它前面的一切都和上次一模一样，字节相同，位置也相同。

所以命中率就是一减去「新的」除以「全部」：

$$\text{HitRatio}_{\text{agent}}(n) = 1 - \frac{U_n}{S_0 + U_0 + \sum_{i=1}^{n-1}(R_i + C_i + T_i + TR_i + U_i) + U_n}$$

分子是一条用户消息，大致恒定；分母每轮增加几千个 token。这就是命中率会往上爬的原因 —— 未命中不长，总量在长。

算一下：Agent 模式第 10 轮

取 S₀ = 800、U = 100、每轮 R + C + T + TR = 930，第 10 轮的输入是 900 + 9 × 1030 + 100 = 10270 个 token。未命中只有那 100 个 Uₙ：

第 10 轮总输入	10,270
新增（未命中）	100
命中率	99.0%

直聊模式：缓存漏在哪里

直聊模式会把推理剥掉。但这带来一个问题，而且很隐蔽 —— 问题不在历史的大小，而在位置。

想一想第 n−1 轮结束时 GPU 上存的是什么。服务端处理完我们的输入，然后先生成推理、再生成回答。生成的 token 也会进缓存，而且是按生成顺序进的：

缓存里 S₀ … U₍ₙ₋₁₎ R₍ₙ₋₁₎ C₍ₙ₋₁₎

再看第 n 轮我们发出去的是什么。推理没了，于是 C₍ₙ₋₁₎ 往前挪，紧贴在 U₍ₙ₋₁₎ 后面：

我们发送 S₀ … U₍ₙ₋₁₎ C₍ₙ₋₁₎ Uₙ

与缓存一致被服务端剥掉位置挪动，对不上了

手动走一遍前缀匹配：S₀ 对得上，中间对得上，U₍ₙ₋₁₎ 也对得上。然后缓存期待的是 R₍ₙ₋₁₎，拿到的却是 C₍ₙ₋₁₎。服务端在 C₍ₙ₋₁₎ 这个位置发现不匹配，缓存就断在这里。

所以未命中是两块而不是一块：上一轮的回答，加上这一轮的新问题。

$$\text{HitRatio}_{\text{chat}}(n) = 1 - \frac{C_{n-1} + U_n}{S_0 + U_0 + \sum_{i=1}^{n-1} C_i + \sum_{i=1}^n U_i}$$

把两个公式对比一下能看出两件事。直聊模式的分子更大（是 C₍ₙ₋₁₎ + Uₙ 而不只是 Uₙ），分母更小（历史里所有推理 token 都没了）。这两个方向都在把比值往下压。

反直觉的地方 直聊模式每次请求发的 token 更少，听起来更省。但这些 token 里按全价计费的比例更高 —— 全价可以是缓存价的 30 倍。token 少了，结构却更差。

动手看它变化

公式自己拨一拨才信得过。下面这个模拟器会实时计算两个公式。拉一根滑块，看看哪条曲线在动。

Cache Hit Rate Explorer

缓存命中率探索器

The toggle picks which mode the bar and the metric cards describe. The chart always plots both, so you can see the gap.

这个开关决定下面的色条和指标卡描述哪种模式。折线图始终同时画两条，方便你看差距。

S₀ — system prompt S₀ —— 系统提示词 800

U — user message U —— 用户消息 100

R — reasoning R —— 推理 500

C — answer length C —— 回答长度 200

T — tool call T —— 工具调用 80

TR — tool result TR —— 工具返回 150

n — current turn n —— 当前轮次 10

What the GPU sees at turn n:

第 n 轮时 GPU 看到的东西：

Prefix hit — billed at cache price Miss — billed at full input price Reasoning thrown away by the server

前缀命中 —— 按缓存价计费未命中 —— 按全价输入计费被服务端丢掉的推理

9800

Total Input Tokens

总输入 token

94.2%

Hit Rate

命中率

100

Missed Tokens

未命中 token

Turn 1 is always 0% — a cold start has nothing to match against.

第 1 轮永远是 0% —— 冷启动没有任何东西可以匹配。

✏️ Try the simulator above! Set C (answer length) to 2000 and watch what happens to the Direct Chat hit rate. It sags hard, while the Agent curve barely notices. Then set R to 2000 and see the Agent curve climb even higher — long reasoning inflates the cached denominator.

Four Conclusions

Hit rate climbs toward 100% as the conversation grows. The denominator grows every turn while the numerator stays about the same size, so the ratio can only go up. Long sessions are cheap sessions.
Context compression resets the clock. Summarizing old turns rewrites the prefix, which invalidates the cache tree from that point on. On the chart, you fall back to turn 1 and start climbing again.
Verbose models hurt Direct Chat. C₍ₙ₋₁₎ sits in the numerator, so a bigger answer is literally a bigger miss. Asking for concise answers is a cache optimization, not just a UX preference.
Agent mode always beats Direct Chat. The miss is Uₙ instead of C₍ₙ₋₁₎ + Uₙ, over a larger total. Keeping reasoning in the history costs you nothing at cache prices and buys a cleaner prefix.

Recap

Hit rate is one minus new tokens over total input tokens.
In Agent mode the only new tokens are Uₙ, so the rate approaches 100%.
In Direct Chat, stripping reasoning shifts C₍ₙ₋₁₎ forward and breaks the match there, so the miss is C₍ₙ₋₁₎ + Uₙ.
Anything that rewrites the prefix drops you back to a cold start.

High hit rate means low cost. But exactly how low? Let's build the full cost formula.

✏️ 上面的模拟器动手试试！ 把 C（回答长度）拉到 2000，看看直聊模式的命中率会怎样。它掉得很明显，而 Agent 那条曲线几乎无感。然后把 R 拉到 2000，你会看到 Agent 曲线还往上走了 —— 长推理把作为分母的缓存部分撑大了。

四个结论

对话越长，命中率越接近 100%。分母每轮都在长，分子大小基本不变，比值只能往上走。长会话就是便宜的会话。
上下文压缩会把时钟归零。给旧轮次做摘要就是在重写前缀，从那个位置往后整棵缓存树都失效。在图上，你会掉回第 1 轮，然后重新往上爬。
话多的模型拖累直聊模式。C₍ₙ₋₁₎ 就在分子里，回答越长，未命中就越大。要求模型简洁不只是体验偏好，它本身就是一种缓存优化。
Agent 模式永远优于直聊模式。未命中是 Uₙ 而不是 C₍ₙ₋₁₎ + Uₙ，而且分母还更大。把推理留在历史里，按缓存价算几乎不花钱，换来的是一条干净的前缀。

小结

命中率等于 1 减去「新增 token / 总输入 token」。
Agent 模式下新增的只有 Uₙ，所以命中率趋近 100%。
直聊模式剥掉推理后，C₍ₙ₋₁₎ 的位置往前挪，匹配就断在那里，于是未命中是 C₍ₙ₋₁₎ + Uₙ。
任何重写前缀的操作都会把你打回冷启动。

命中率高就意味着成本低。可到底低多少？下一节我们把完整的成本公式搭起来。