🔬Cache Engineering Cuts Agent Costs 27.7% With KV Control
TL;DR
Tsinghua researchers let agents drop and reposition KV cache entries directly, avoiding re-prefilling. A rolling-drop policy beat summarization by 1.7 benchmark points while cutting cost 27.7%.
Tsinghua researchers let agents drop and reposition KV cache entries directly, avoiding re-prefilling. A rolling-drop policy beat summarization by 1.7 benchmark points while cutting cost 27.7%.
Key Points
Two operations: drop excludes entries from attention, repos adjusts position encodings
AKV serving engine exposes a stateless API over stateful cache reuse
Up to 1.2x output throughput and 1.5x faster P95 latency
Self-improved cache policies cut cost another 7.1% versus expert-designed ones
Code released at github.com/thu-nics/AKV
Why It Matters
Long agent runs are mostly a KV cache bill. Controlling the cache beats summarizing history, which invalidates cached prefixes.
Quick Facts
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,567 builders reading daily.