Impact
Customers on both the iSell London and iSell Dublin platforms experienced significant service degradation. This included general slowness, failed flight and hotel searches, failed bookings, and occasional 500 errors across several areas of the platform.
Normal service was restored once corrective actions were taken.
Cause & Resolution
The incident was traced to a newly created database replica introduced as part of ongoing performance improvements. A fault occurred during replication between the main database and its replica.
When replication fails, binary logs (temporary records of database changes used to keep replicas in sync) can continue to accumulate. In this case, the logs filled the available storage on the primary database, which led to service degradation across multiple platform components.
To restore service, storage capacity was increased on the affected database, which immediately resolved the issue and returned the system to normal operation.
Prevention Measures
Replication on the affected database has been temporarily disabled while alternative approaches are evaluated for synchronising data between databases.
Additional measures include:
These improvements will help prevent a recurrence of similar storage-related issues.