Data Cloud Feature Degradation

Our third-party provider’s deployment to fix older pending jobs is still in progress and is expected to complete within approximately 4 hours.

In the meantime, we are continuing to evaluate alternate ways to safely retry and reprocess jobs in parallel to expedite mitigation.

Customers may continue to see failed or delayed activation files, Calculated Insights, Data Transforms, and data ingestion jobs during the recovery process. We'll confirm the retry approach for each service as soon as it's validated.

We appreciate your patience and will share another update as soon as there's significant progress.

Our third-party provider is still deploying a fix for older pending jobs and expects to clear that backlog within three hours. When the fix finishes, many of those older jobs may fail. We'll retry them automatically where possible. Customers may see failed or delayed activation files, Calculated Insights, Data Transforms, and data ingestion jobs during this time. We're reviewing each service to confirm how it will handle these jobs.

We'll provide an update in 60 minutes or sooner if additional information becomes available.

Before our third-party provider can handle older queued jobs, the provider needs to finish deploying its fix. Once the fix is complete, many of those older jobs will fail. We'll retry them automatically where possible. Customers may see failed or delayed activation files, Calculated Insights, Data Transforms, and data ingestion jobs during this time. Some jobs may need to be rerun. We're reviewing each service to confirm retries work as expected.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

Most affected services are now processing new jobs successfully. Services that moved to another processing path, including activations, data services, and insights, are completing new jobs. We're confirming recovery for the remaining services. Jobs that were pending before the recovery are still delayed. Our third-party provider expects to clear them within four hours. We're working with the provider on the safest way to recover these older jobs and avoid additional impact. Customers may continue to see delayed activation files, Calculated Insights, Data Transforms, and data ingestion jobs.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

Our third-party provider is deploying a fix for older pending jobs and expects to clear that backlog within about four to five hours. We moved data ingestion to another processing path, and new data ingestion jobs are now completing. Data ingestion jobs submitted before that change are still pending. We haven't canceled any pending jobs, and we're assessing the safest way to recover them without additional impact.Customers may continue to see delayed activation files, Calculated Insights, Data Transforms, and data ingestion jobs.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

Our third-party provider is processing new jobs successfully on affected instances. The provider identified an issue that is slowing older pending jobs. It's deploying a fix to resolve. Customers may continue to see delayed Calculated Insights, Data Transforms, activation files, and data ingestion jobs until the backlog clears. We're reviewing each service to decide whether to wait for older jobs or cancel and resubmit them.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances experiencing impact, and customers in these instances will receive communications related to this incident going forward.

We're still waiting on our third-party provider to finish rolling back the change that triggered this issue, and we don't yet have a completion time. In the meantime, we've moved some affected services to another processing path and confirmed that new jobs are running successfully in a test environment. We're applying the same change to more instances, and we'll then move on to the live environment.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

We've identified the trigger of this issue with a third-party provider and they are rolling back a recent change. An estimated timeline for completing the rollback is currently unavailable. In parallel, we're validating a fallback that sends affected jobs through another processing path.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

We're testing a switch to alternative infrastructure to restore processing of activation jobs, and we're seeing positive results. We're working to apply this change across the impacted features and will continue to monitor progress. We remain in contact with the third-party service provider responsible for this issue.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

We're in the process of testing whether we can switch to an alternative infrastructure to restore the processing of activation jobs. The third-party service provider is also continuing to work to resolve the issue on their end. We've also identified additionally impacted customers and customers in these instances will receive communications related to this incident going forward. We’ve also determined that customers using Data 360 CRM Connectors may be seeing impact.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

Following further investigation, we've identified additionally impacted customers and customers in these instances will receive communications related to this incident going forward. We've also determined that these features may be experiencing the same issue: Calculated Insights and Data Transform.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

We're still investigating whether we can take any action on our side to restore service for affected customers. The third-party provider has confirmed it has the right resources in place and is continuing to work on a resolution.

We'll provide an update in 30 minutes or sooner if additional information becomes available.

Data 360 customers are experiencing activation jobs are queued and are timing out. A subset of customers using Marketing Cloud Connector are seeing the issue as well. We have identified that the issue is a result of a third-party provider issue. We’re working alongside the third-party provider to find a resolution to this issue. We’re also evaluating a switch to alternative infrastructure to restore processing sooner.

We'll provide an update in 30 minutes or sooner if additional information becomes available.