Degraded Performance Observed in London and Dublin environments

Incident Report for Traveltek

Postmortem

Impact:

During the incident, customers experienced slowness across both the iSell London and Dublin platforms due to database storage reaching critical capacity levels. The issue affected all users of cruise products on both platforms.

Cause & Resolution:

The incident was caused by the accumulation of MySQL binary logs and issues with the replication configuration, which prevented automatic log cleanup. As a result, storage filled up over time, leading to performance degradation.
To quickly restore service stability, manual storage expansion was performed, followed by corrective actions to address binary log retention and replication health.

Prevention Measures:

To prevent recurrence, the following measures are being implemented:

  • Configure automated binary log management with appropriate retention settings and monitoring alerts to avoid excessive accumulation.
  • Implement enhanced storage monitoring covering binary logs, table fragmentation, and replication health, with alerts at critical thresholds.
Posted Aug 02, 2025 - 03:28 BST

Resolved

The incident has been resolved. A report will follow and be attached to the incident in due course.
Posted Jul 29, 2025 - 09:59 BST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 29, 2025 - 09:52 BST

Identified

We've identified the issue and currently working on a fix. Further updates to be provided shortly.
Posted Jul 29, 2025 - 09:44 BST

Investigating

We are currently experiencing a degradation in performance on our London and Dublin environments. Our engineering team is actively investigating the issue to identify the root cause and restore full functionality as soon as possible.
We apologize for the inconvenience caused and appreciate your patience. Further updates will be provided as we make progress.
Posted Jul 29, 2025 - 09:38 BST
This incident affected: iSell London (Databases) and iSell Dublin (Databases).