Impact:

Between approximately 10:40 AM and 1:25 PM BST on the 29th May, customers using our platform would have encountered intermittent “502” errors and noticeable slowness. At one stage, the service appeared to return to normal, but navigating within iSell would lead to “404” errors (indicating “page not found”).

Cause & Resolution:

The incident was caused by an issue with accessing Ubuntu’s software resource repositories (archive.ubuntu.com), which are essential for building and provisioning our servers. These repositories became temporarily unavailable or extremely slow to respond, leading to timeouts during package downloads. When new servers were launched during this window, they could not complete their build process and were only partially configured. As a result, they would be terminated automatically due to failed health checks. We attempted to work around this by rolling back to a previous snapshot of our servers, which allowed us to bypass some of the missing package downloads and bring new servers online faster. However, this snapshot lacked some required configurations, resulting in “404” errors when navigating around iSell. We identified the missing configurations and promptly deployed them to the new servers, restoring full service by 1:25 PM. Later that day, Ubuntu resolved the issue on their end, and we were able to provision servers normally again.

Prevention Measures:

 While the unavailability of Ubuntu’s software resources is outside our direct control and is a rare occurrence, we are taking proactive steps to mitigate the impact of similar future events:

• Failover Improvement: We plan to enhance our emergency fallback process. During this incident, rolling back to a previous snapshot helped to partially restore service, and we believe this process can be automated and improved to provide a more seamless failover path in the event of external repository outages.
• Automation of Emergency Rollback: We will explore automating the configuration deployment step when falling back to a snapshot, to ensure full service readiness even if external resources remain unavailable.
• Ongoing Monitoring and Collaboration: We will continue to monitor external resource availability and collaborate with upstream providers (such as Ubuntu) to stay informed about any ongoing or planned maintenance that could affect our platform’s build processes.