🚀Netflix Overhauls Conductor to Handle 200K Workflows
Netflix's Workflow Engine Gets a Major Upgrade
TL;DR
Netflix has overhauled its Conductor workflow engine to support 200,000 workflows across 150 apps, executing 420M workflows monthly. The redesign slashes p99 latency by 40%, crucial for Netflix's expanding content and gaming divisions.
Netflix just overhauled its Conductor workflow engine to handle massive scale, supporting 200,000 workflows across 150 applications and executing 420 million workflows monthly. This redesign slashes p99 latency by 40%, making it a must-watch for anyone managing complex workflows at scale. Conductor now supports workflows up to 30,000 tasks, up from 2,500, and moves task data to S3, improving performance and scalability.
Key Points
Conductor now supports 200,000 workflow definitions across 150 applications.
Conductor executes 420 million workflows monthly, up from earlier versions.
Redesign increases supported workflow size from 2,500 to 30,000 tasks.
P99 workflow evaluation latency reduced by 40%, improving performance.
Conductor moves task data to S3, decoupling indexing with Kafka and Elasticsearch.
Why It Matters
If you're managing complex workflows at Netflix, Conductor's overhaul is a game-changer. It supports 200K workflows, executes 420M monthly, and cuts p99 latency by 40%. This is crucial for teams in Content and Studio Engineering, Ads, and Games as Netflix expands into live content, games, and podcasts.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,476 builders reading daily.