🚀Bluesky Backend Cuts Query Time with OHBF Bloom Filters
Bluesky slashes query time with a new bloom filter
TL;DR
Bluesky's backend uses a single-hash bloom filter called OHBF to drastically reduce query times. The OHBF implementation has been live for 9 months, handling 1 million items with a 1% false positive rate.
Bluesky's backend now uses a single-hash bloom filter called OHBF, cutting query times significantly. This new implementation reduces contention and speeds up deploys from 90 minutes to 15 minutes. OHBF uses xxh3 hash function and is optimized for cache line size, ensuring one cache miss per lookup. The false positive rate is 1% and it can handle up to 1 million items.
Key Points
OHBF uses a single xxh3 hash function for bloom filter implementation.
OHBF splits bloom filter into 64-byte blocks for optimal CPU cache line size.
OHBF guarantees one cache miss per lookup with unsafe block allocation.
False positive rate of 1% for OHBF with expected capacity of 1 million items.
OHBF is used in Bluesky backend and jetstream sync service for efficient key checks.
Why It Matters
If you're running a high-performance data system, OHBF can drastically reduce query times and contention. It's live in Bluesky's backend, cutting deploy times from 90 minutes to 15 minutes. The single-hash implementation with optimized cache line size ensures minimal cache misses and efficient memory usage.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,566 builders reading daily.