Performance Degradation

The Salesforce Technology team has verified that database DB72 is currently stable, but there is ongoing performance degradation due to rate limits placed on push/email messaging, data processing, and Journey Builder activities. These limits were implemented to maintain the database's health and availability.

We are actively working on remediation activities to gradually alleviate the rate limits in line with the vendor's efforts to address the issue.
We continue to monitor the health and performance of the impacted database to ensure stability and performance.

We’ll provide further updates as additional information becomes available.

DB72 remains in a state of performance degradation, resulting in slow write speeds. We have taken steps to minimize the impact, including pausing and resuming scans to monitor progress. In the meantime, non-critical tasks related to this database have been put on hold.

We’ll provide further updates as additional information becomes available.

The Salesforce Technology team has determined that this issue may impact additional services. Customers who have linked Marketing Cloud data with services such as Data Cloud, Audience Builder, the Marketing Cloud Intelligence app family (Marketing Cloud Reports, Marketing Cloud Intelligence, Intelligence Reports), and the Marketing Cloud Einstein app family (Data Analytics, Engagement Scoring, Insights, Send Time Optimizations) will experience data delays. Data is not flowing from Marketing Cloud to these services, and data that appears may be several hours old.

We are continuing our investigation into this issue. We’ll provide further updates as additional information becomes available.

Teams continue to explore the cause of the trailing performance degradation on database DB72 and continue to perform scans and validate logs for the degradation in conjunction with the third-party vendor. In parallel, the teams are exploring ways to improve the efficiency of non-essential workloads on the database to see if they can ease the degradation.

We’ll provide further updates as additional information becomes available.

At 00:44 on June 05, 2024, we restored the database DB72, and it is online. The team confirmed that the services are accessible, and we are experiencing a trailing performance degradation due to backlog processing.
We will continue to monitor the performance until service levels return to normal.
We will keep you posted with significant updates.

The service for DB2, DB40, DB121, and DB139 databases was restored at different times and has been confirmed as healthy.

Trust posts for the resolved databases will be posted individually to specify start and end times for impact, and updates for DB72 will be provided here.

The vendor is still working on recovering DB72, and it is estimated to take approximately two hours.

Further updates will be given as more information is received.

As of 23:43 UTC, Service to the DB139 has been restored and validated as healthy. We will continue to monitor the environments to ensure stability.

The recovery process for DB72 is still underway and is expected to take approximately 2 more hours to complete.

We’ll provide another update as additional information becomes available.

Service to the DB2, DB40, and DB121 databases was restored at 20:56, 21:12, and 21:34, respectively, and has been validated as healthy. We will continue to monitor the environments to ensure stability.

The vendor-side recovery process for DB139 and DB72 is still underway and is expected to take approximately four more hours to complete.

We’ll provide another update as additional information becomes available.

Service to the DB2 database has been restored and its health is being validated. We will continue to monitor the environment to ensure stability and will restore service to DB121 and DB40 once DB2 proves to be in a healthy state.

The vendor-side recovery process for DB139 and DB72 is still underway and is expected to take approximately three to six hours to complete.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

The database restore process for DB121, DB40, and DB2 has been completed, and prep work is underway to bring them back online shortly.

The vendor-side recovery process for DB139 and DB72 is expected to take several more hours to complete. We continue working closely with the vendor to expedite this process as much as possible.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

The database restore process for DB121, DB40, and DB2 and the vendor-side recovery process for DB139 and DB72 are still underway. We will continue working to expedite these restoration and recovery processes as much as possible.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

The database restore process for DB121, DB40, and DB2 and the vendor-side recovery process for DB139 and DB72 are still underway. Both are expected to take several hours to complete.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

We have initiated the database restore process for DB121, DB40, and DB2. However, that has been determined not to be a feasible solution for DB139 and DB72. The recovery of DB139 and DB72 will remain dependent on the vendor’s completion of their recovery process. We will continue working closely with the vendor to determine whether the recovery of these databases can be expedited.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

We continue working closely with the third-party vendor to restore functionality and are investigating the feasibility of performing database restores from backups.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

We have determined that a third-party vendor inadvertently removed an incorrect director board and is working to restore the storage array for the impacted databases. The vendor has provided an estimate of several hours for the restoration to be completed. We will continue to investigate potential workarounds to mitigate the impact as quickly as possible.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

On June 04, at 15:12 UTC, the Salesforce Technology team noted a service disruption on multiple database instances. Due to a storage issue, customers may be unable to access their services. Our teams are actively troubleshooting the issue to restore services as quickly as possible.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.