🧪Datadog Adds Online Experiments to Test AI Agents on Live Traffic
Your agent tests just got a lot more honest
TL;DR
Datadog's new Online Experiments lets teams split live traffic between AI agent versions and compare quality, latency, token use, and cost side by side. Offline test suites miss real user behavior, so this is worth a look before your next prompt or model swap.
Datadog is extending its Agent Observability platform with Online Experiments, which lets you run two versions of an AI agent against real production traffic and see which one actually performs better. The pitch is simple: a change that wins on your curated test dataset can still fall over when users show up with weird requests and tools return unexpected responses. If you're shipping agents, this closes the gap between your eval suite and reality. You can compare a control and a variant by randomly splitting traffic with Feature Flags, then scoring quality with Datadog's evaluators alongside latency, token consumption, estimated cost, and workflow errors. That means a model swap, prompt tweak, retrieval change, or new tool can be judged on the same dashboard as your operational metrics. The example Datadog shares is revealing. An investigation agent was tested with GPT-5.6 Luna against GPT-6 Luna, traffic split evenly. The newer model cut estimated cost by 44.3% with quality scores of 3.84 and 3.86 on a 1-to-5 LLM-judged scale. But execution duration and workflow errors crept up, and trace inspection showed the newer model sent invalid tool parameters more often, usually recovering on retry. Cheaper and nearly as good on paper, but not free of operational cost.

Key Points
Online Experiments pairs Datadog's Agent Observability evaluators with Feature Flags, so teams can randomly split live traffic between a control agent and a modified variant.
Keep reading — it's free
Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.
Already a member? Sign in
Comments
Be the first to comment