🏰Airbnb's CASTLE Trains Search Models With Synthetic Data, Hitting 0.08 KL Divergence
Your search cold-start problem might have a synthetic fix
TL;DR
Airbnb built CASTLE, a framework that uses LLMs and de-identified booking sessions to generate synthetic search queries and relevance labels. It fixes the cold-start problem for natural language search. If you're building search without query logs, this is a usable playbook.
Airbnb just published CASTLE, a way to train natural language search without waiting for real user queries. The trick: it generates synthetic queries and relevance labels together, using LLMs and de-identified booking sessions. Most teams fix one half of this problem and fake the other. CASTLE does both at once. If you're building search on a fresh product, you know the cold-start pain. No logs means no training data, and no training data means search that feels dumb. CASTLE builds contrastive listing pairs from booking history, then prompts an LLM to write a query that matches the positive listing but not the negative one. For pairs with thin booking history, a Virtual Judge, which is another LLM-based evaluator, fills the gap. Teams already in production can use CASTLE-R, which folds in real search traffic as it piles up. The numbers are the reason to pay attention. CASTLE hit a KL divergence of 1.01 against real-user query distributions, and 0.08 at the attribute-type level. Three subject matter experts reviewed 200 sampled triplets, and their agreement with CASTLE's relevance assignments landed between 91% and 93%. The approach mixes template, few-shot, and attribute-grounded prompts, plus a variety mechanism that pushes query length and phrasing around so the synthetic data doesn't all sound the same. The full paper is on arXiv if you want the details.

Key Points
CASTLE reached a KL divergence of 1.01 against real-user query distributions, and 0.08 at the attribute-type level, which is a strong signal the synthetic queries resemble how guests actually search.
Keep reading — it's free
Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.
Already a member? Sign in
Comments
Be the first to comment