🤖ORCA-bench Tests Language Models for On-Call RCA
AI agents struggle with real-world on-call tasks
TL;DR
ORCA-bench evaluates language model agents for on-call RCA, revealing significant accuracy gaps. Best models hit only 25.3% on medium-difficulty tasks.
ORCA-bench was introduced to test language model agents in real-time systems for on-call root cause analysis (RCA). The benchmark pairs a live OpenTelemetry-instrumented microservice system with 1,079 RCA tasks, systematically varying report specificity and co-occurring fault scenarios. Ground-truth symptoms are curated by expert SREs. Despite this rigorous testing, the best model only achieves 25.3% accuracy on medium-difficulty tasks, while some models hallucinate implausible root causes in 40% of incident reports. Removing source-code access further degrades performance. This benchmark highlights the engineering investment required for production reliability.
Key Points
ORCA-bench evaluates language model agents on a curated testbed of 50 GB over six days
The benchmark includes 1,079 RCA tasks with varying levels of report specificity and fault scenarios
Ground-truth symptoms are signed off by expert SREs for accuracy
Best models achieve only 25.3% accuracy on medium-difficulty tasks; worst hallucinate in 40%
Removing source-code access degrades every metric, emphasizing the importance of context
Why It Matters
If you're using AI agents for on-call RCA, ORCA-bench reveals significant performance gaps. The best models only hit 25.3% accuracy on medium-difficulty tasks, and removing source-code access drops performance further. This benchmark underscores the need for substantial engineering investment to ensure production reliability.
Frequently Asked Questions
Why does this matter?
If you're using AI agents for on-call RCA, ORCA-bench reveals significant performance gaps. The best models only hit 25.3% accuracy on medium-difficulty tasks, and removing source-code access drops performance further. This benchmark underscores the need for substantial engineering investment to ensure production reliability.
What happened?
ORCA-bench evaluates language model agents for on-call RCA, revealing significant accuracy gaps. Best models hit only 25.3% on medium-difficulty tasks.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.