🛠️Shopify Cuts a 6,000-Token Prompt to 1,500 Gist Tokens
TL;DR
Shopify trained learned 'gist' tokens to stand in for a long system prompt, shrinking its Sidekick GraphQL agent prompt 4:1 with no measured quality loss. Median end-to-end latency fell from 6.8s to 4.2s, throughput rose 16%, and the team cut allocated GPUs.
Shopify trained learned 'gist' tokens to stand in for a long system prompt, shrinking its Sidekick GraphQL agent prompt 4:1 with no measured quality loss. Median end-to-end latency fell from 6.8s to 4.2s, throughput rose 16%, and the team cut allocated GPUs.
Key Points
Sidekick GraphQL agent system prompt went from ~6,000 tokens to 1,500 gist tokens, a 4:1 context reduction
At 350 requests per minute, median time to first token dropped from 438ms to 354ms and end-to-end latency from 6.8s to 4.2s
Throughput rose from 20.2 to 23.4 queries per second, letting the team cut allocated GPUs
Training is a teacher/student pass minimising KL divergence, then gist embeddings are written into the model's embedding matrix as special tokens
No custom attention mask, encoder or serving path is required, and it compounds with prefix caching rather than replacing it
Why It Matters
Long system prompts are a recurring tax on every request. Gisting shows the fix can live in the embedding matrix instead of the serving stack, which makes it adoptable without re-architecting inference.
Quick Facts
Frequently Asked Questions
Why does this matter?
Long system prompts are a recurring tax on every request. Gisting shows the fix can live in the embedding matrix instead of the serving stack, which makes it adoptable without re-architecting inference.
What happened?
Shopify trained learned 'gist' tokens to stand in for a long system prompt, shrinking its Sidekick GraphQL agent prompt 4:1 with no measured quality loss. Median end-to-end latency fell from 6.8s to 4.2s, throughput rose 16%, and the team cut allocated GPUs.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,467 builders reading daily.