Profile update processing delays on CEP

This incident is now resolved. Profile updates are being processed in real time again, and the data used for one-time and recurring campaign targeting, and attribute-change automations has fully caught up, with no profile data lost. Separately, the earlier disruption to transactional message sending and in-app display on July 7 between 19:45 and 20:15 CEST was resolved the same evening. All services are operating normally.

Profile updates are now being processed in real time again at the source, and profile analytics have caught up. A downstream indexing backlog remains, however: the profile data used for campaign targeting, including targeting based on attribute changes, is currently delayed by under an hour and is steadily catching up. No profile data has been lost, and all updates will be reflected once the backlog fully clears. We continue to monitor closely and will share a further update tomorrow morning.

Between 19:45 and 20:15 CEST, a disruption affected our MEP Transactional API, during which some transactional messages may not have been sent, and some in-app messages on CEP may not have been displayed. Both are now fully resolved and operating normally again, and transactional requests from this window can be safely retried. Separately, our CEP profile processing pipeline saw a renewed delay this evening and is still catching up to real time, with no profile data lost. We will continue to monitor closely and share a further update within the next hour.

Recovery is nearly complete on our CEP profile processing pipeline, with real-time processing restored for the large majority of profile updates. We expect the remaining delay to fully clear during this evening. As a precaution, we have put in place reinforced monitoring and on-call coverage overnight to quickly detect and address any recurrence. We will share a further update tomorrow morning.

Recovery is underway on our CEP profile processing pipeline. Real-time processing has been restored for approximately 99% of profile updates, and the remaining share is still catching up. During this time, a small portion of attribute updates may still be delayed in one-time & recurring campaign targeting, automations triggered by attribute changes, and profile analytics. No data has been lost, and all delayed updates will be processed as the backlog clears.

We are seeing recovery on our CEP profile processing pipeline. Real-time processing has already been restored for the majority of profile updates, while a subset is still catching up, so some attribute updates may remain delayed a bit longer in one-time & recurring campaign targeting, automations triggered by attribute changes, and profile analytics. No data has been lost, and all delayed updates will be processed as the backlog clears. We are continuing to monitor closely until processing is fully back to real time.

Since approximately 04:40 CEST, our CEP profile processing pipeline has been delayed, with profile updates currently around 5 hours behind real time. As a result, some profile attribute updates may not yet be reflected in one-time & recurring campaign targeting, automations triggered by attribute changes, and profile analytics. Campaign audience selection and most event-based automations are not affected. We are actively investigating the issue; no data has been lost, and delayed updates will be processed as the backlog clears.