Skip to content
Source: InfoQ

🧪Datadog Adds Online Experiments to Test AI Agents on Live Traffic

Your agent tests just got a lot more honest

TL;DR

Datadog's new Online Experiments lets teams split live traffic between AI agent versions and compare quality, latency, token use, and cost side by side. Offline test suites miss real user behavior, so this is worth a look before your next prompt or model swap.

Datadog is extending its Agent Observability platform with Online Experiments, which lets you run two versions of an AI agent against real production traffic and see which one actually performs better. The pitch is simple: a change that wins on your curated test dataset can still fall over when users show up with weird requests and tools return unexpected responses. If you're shipping agents, this closes the gap between your eval suite and reality. You can compare a control and a variant by randomly splitting traffic with Feature Flags, then scoring quality with Datadog's evaluators alongside latency, token consumption, estimated cost, and workflow errors. That means a model swap, prompt tweak, retrieval change, or new tool can be judged on the same dashboard as your operational metrics. The example Datadog shares is revealing. An investigation agent was tested with GPT-5.6 Luna against GPT-6 Luna, traffic split evenly. The newer model cut estimated cost by 44.3% with quality scores of 3.84 and 3.86 on a 1-to-5 LLM-judged scale. But execution duration and workflow errors crept up, and trace inspection showed the newer model sent invalid tool parameters more often, usually recovering on retry. Cheaper and nearly as good on paper, but not free of operational cost.

Datadog Adds Online Experiments to Test AI Agents on Live Traffic — InfoQ

Key Points

1

Online Experiments pairs Datadog's Agent Observability evaluators with Feature Flags, so teams can randomly split live traffic between a control agent and a modified variant.

Keep reading — it's free

Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.

Also get
or

Already a member? Sign in

ai-agentsobservabilitydatadogllm-evaluationfeature-flags

Comments

Subscribe to join the conversation...

Be the first to comment