⚡How OpenAI Cut ChatGPT Read Latency 12x for 1B Users
TL;DR
OpenAI's Habitat storage platform serves over 1 billion users across 40 regions and holds more than 500 PB. Global read p50 latency fell from about 120 ms to 10 ms through nearby replication, in-region caching and a Rust proxy rewrite.
OpenAI's Habitat storage platform serves over 1 billion users across 40 regions and holds more than 500 PB. Global read p50 latency fell from about 120 ms to 10 ms through nearby replication, in-region caching and a Rust proxy rewrite.
Key Points
Habitat runs on Azure Cosmos DB; the largest account serves over 20M QPS
Reads are more than 90% of traffic, so Valkey lookaside caching pays off
CDC streams millions of changes per second into more than 100 Kafka topics
The Python proxy layer was rewritten in Rust
A late-2024 regional outage showed failing over the largest accounts would take hours
Why It Matters
The lesson for builders is boring and useful: caching and multi-tenancy controls, not model tricks, keep a consumer AI product fast at scale.
Quick Facts
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,567 builders reading daily.