Data Processing Delays - Events and Persons ingestion

We have multiple engineers working on remediation as a top priority. Data is being written to our new data store and we are carefully validating the data integrity before we start to switch services over to it which will begin to solve the core underlying issues.

We expect delays to continue to event and persons ingestion as we make these changes over the next 12-24 hours as we are prioritise data integrity and stability. Importantly no data has been lost.

There will be a full public post-mortem once we are recovered.

We are still working on remediation for the database issue and lag is still increasing, some events will see delays of up to 13 hours. We have identified the root cause but due to the volumes of data involved repairing the database will take some time

Our data processing infrastructure is lagging as we work to resolve an issue with our persons database. No data has been lost but the incident is ongoing and substantial lag should be expected between event submission and availability in the product during remediation.

We are experiencing issues with our database responsible for updating Persons from incoming events which is causing large delays in processing.
You may see delays ranging between 10-60 minutes depending on the customer. We are actively working on remedying the underlying issue although we expect to have continued delays over the coming hours as we carefully fix the issues whilst maintaining data integrity.