🚨OVH Reboots 6K Hosts to Fix Critical Januscape Bug
Massive cloud reboot to fix critical bug, but at what cost?
TL;DR
To address a severe security flaw in Januscape, OVH initiated a large-scale host reboot affecting thousands of machines. Despite some outages and customer impact, the company considers it a 'remarkable feat'.
OVH took drastic action to fix a critical bug in its cloud infrastructure by rebooting over 6,000 hosts across multiple regions. This move was necessary due to the severe security implications of the Januscape vulnerability, which allowed attackers with root access to execute code as root on host machines or take control of other VMs. The decision to proceed with a mass reboot, despite potential service disruptions and customer impact, highlights OVH's commitment to maintaining isolation guarantees for its cloud customers. Key details: 6,000 hosts affected, approximately one million virtual machines impacted, and the operation took place in waves over several days.
Key Points
To address Januscape's guest-host escape vulnerability, OVH decided to backport a fix into its Debian production environment and initiate a large-scale reboot operation.
Approximately one million virtual machines were impacted by this security flaw, highlighting the need for swift action from cloud providers.
The company orchestrated the reboots in waves, with strict thresholds set to minimize simultaneous interruptions: 15 hosts in high-density regions or five boxes elsewhere.
Despite careful planning, some VMs experienced issues post-reboot, including data corruption and failure to restart due to hardware faults.
OVH's approach was deemed a 'remarkable feat' despite outages, emphasizing the need for improved processes in handling future kernel vulnerability disclosures.
Why It Matters
If you're running critical workloads on cloud infrastructure, this highlights the trade-offs between security and uptime. OVH’s decision to prioritize security over potential customer impact showcases a proactive approach to mitigating risks.
Frequently Asked Questions
Why does this matter?
If you're running critical workloads on cloud infrastructure, this highlights the trade-offs between security and uptime. OVH’s decision to prioritize security over potential customer impact showcases a proactive approach to mitigating risks.
What happened?
To address a severe security flaw in Januscape, OVH initiated a large-scale host reboot affecting thousands of machines. Despite some outages and customer impact, the company considers it a 'remarkable feat'.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 2,179 builders reading daily.