Customer Data Platform - Production Database Issues

This incident has been resolved.

Queue processing is progressing well without any issues. We will continue to monitor the situation closely to ensure optimal performance before closing the issue.

Queues are emptying as anticipated, and events are being processed with workflows being triggered accordingly. Please note that the duration of this process may vary based on the queue size specific to your tenant.

The queued data is currently being processed in a stable way, it will need time to catch up the past 24h.

We have implemented fixes to restore the system and are closely monitoring the progress. The queued data is being processed.

We will provide updates as we move forward.

We have identified the culprit causing the issue and are working on restoring our services with our service partner.
For the past 16 hours, our customers have been able to log in to CDP and use the system, but without access to recent data from the past 24 hours.

The primary cause of the outage lies with our cloud service provider. We are actively coordinating with them to expedite resolution and are closely monitoring their progress, as our recovery depends on their actions.

We are persistently working, alongside our cloud provider, to address the issue impacting our primary database. Once we have more information, we will provide more updates.

We are diligently continuing our efforts and have identified some promising leads in collaboration with our cloud provider.

We are actively working with multiple teams together with our cloud provider to identify and resolve the issue affecting our primary database. Our priority is to restore full functionality as quickly and efficiently as possible.

We are experiencing an outage for our primary database. All incoming events are being fail overed to our secondary database however due to this events and workflows are not being processed.

We are investigating this issue.