🔍Google Open-Sources AQuA to Catch Silent AI Agent Failures in Production
Your agents might be failing quietly right now
TL;DR
Google open-sourced AQuA, a reference implementation that reviews up to 1,000 production agent sessions to catch silent quality regressions. It runs outside the request path, reading Cloud Trace, Cloud Logging, or BigQuery. Worth a look if your agents ship without anyone reading transcripts.
Your agents can be broken for days and your dashboards will still look green. Google just open-sourced AQuA, the Ambient Quality Agent, to catch exactly that kind of silent regression. It sits next to your agent in Google Cloud, outside the request path, and reads telemetry from Cloud Trace, Cloud Logging, or BigQuery. If you run multi-agent systems, this matters because infrastructure monitoring tells you the service is up, not that the answers got worse. AQuA samples recent sessions and runs them through five stages: sampling, reviewing, clustering findings, verifying candidate clusters, and tracking issues. You can define domain-specific criteria in a goal.md file and add deterministic custom metrics. It runs on a schedule, after deploys, or on demand. Google's test case was a multi-agent travel app. It reviewed 32 sessions, surfaced 42 findings, and grouped them into nine candidate clusters. The verifier rejected three unsupported clusters after reading up to three full transcripts per candidate, leaving six verified issues. Findings get tagged NEW, RECURRING, or auto-RESOLVED after 14 days unseen. The catch: sampling can miss rare failures, and replay may not reproduce external state. AQuA also won't apply fixes or open pull requests on its own. Treat it as a diagnosis tool, not an auto-remediator.

Key Points
AQuA samples up to 1,000 recent production sessions per run, reading telemetry from Cloud Trace, Cloud Logging, or BigQuery without sitting in the request path of your agent.
Keep reading — it's free
Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.
Already a member? Sign in
Comments
Be the first to comment