[Piano Audience] User Profiles Update Backend Processing Delays

The incident has been resolved after monitoring for more than 24 hours. We are seeing the processing of the data in real-time

Data processing on the Piano Audience User Profiles backend has returned to near real-time and the backlog is cleared. We are setting this incident to Monitoring while our team completes residual cleanup from the earlier data change. We will share a further update once the incident is fully resolved.

Data processing delays continue on the User Profiles backend, with current delays of 1–2 hours.

We have identified that a recent data change led to a significant increase in data volume for a subset of customers, which is the primary driver of the backlog. Our team is carefully increasing processing throughput while closely monitoring overall system stability to avoid introducing further issues.

We are optimistic about stabilizing the situation soon and will share an update as soon as we have more information.

We are observing data processing delays on the User profile backend caused by data improvement efforts for managing the cx prefix and the composer application ID. If you are using the cxense's external profile update API (https://api.cxense.com/profile/user/external/update) you will observe 1-2 hours of delays when reading the same profile using (https://api.cxense.com/profile/user/external/update).

Our team is now reconciling the data back into a single, correct view, and this reconciliation is the source of the ongoing delay. No data has been lost, all events are being processed, currently with an approximate delay of 1–2 hours. We apologize for the inconvenience and will share an update as the backlog clear