Increased Pipeline Latencies due to Database Extractions Failing

This incident has been resolved - the majority of pipelines have caught up fully and the last few remaining pipelines show all extractions from database sources succeeding.

The issue was an upgrade to our Spark library which is used in creating the extraction jobs for database sources. A networking configuration option in the upgraded library caused failures when submitting the extraction jobs. We reverted the upgrade to the library and have confirmed that extraction job submission is now successful. We apologise for any inconvenience caused.

The fix has been deployed across all environments. We are starting to see successful database extractions and expect all failing pipelines to catch up soon. We are continuing to monitor.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.