Performance Degradation

The Salesforce Technology team have confirmed that restarts of the Message Queues, Search have completed and were successful. Customers should begin to see signs of improvement. The indexing and processing of a backlog of requests is underway.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

The Salesforce Technology team have confirmed that restarts of the Message Queues have completed and were successful. The Message Queues are currently processing a backlog of requests.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

It has been determined that Salesforce Search is also impacted by this issue. Engineers are working to restart the unhealthy Search services in an attempt to mitigate the impact.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

The Salesforce Technology team are continuing with their extended mitigation phase. The restarts are underway and are being tracked to see if they are successful. Engineers have confirmed that this issue is no longer a service disruption as of 11:00 UTC following the stabilization of the DNS, it is now being considered a performance degradation.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

Engineers have entered an extended mitigation phase as they are restarting impacted Message Queue brokers and the application tiers. This is expected to take some time to complete as they must be restarted and validated individually.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

The action that was taken to increase resources handling Domain Name System (DNS) had a positive result as this service has now stabilized. The Message Queue broker restarts and the application tier restarts are in progress and are the outcomes of the restarts are being validated. These collective actions have brought us on a path to resolution.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Restarts of various backend services are being undertaken. This is being done in two instances and validated if this is successful. If positive results are shown this will be done to all impacted instances.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The Salesforce Technology team is attempting to restart a subset of Message Queues as a mitigation attempt.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The Salesforce Technology team has determined that this issue has become a service disruption. During a service disruption, end users can’t access the service.

A subset of customers may experience issues with both outbound and inbound emails. Customers using Live agent, Pub/Sub APIs (publishing and subscribing to platform events) and event bus APIs are also seeing impact.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Following further investigation, the Salesforce Technology team has narrowed down the network issue to a Domain Name Server (DNS) issue. Additional resources are being added as an attempt to help reduce the strain on the DNS.

It has also been identified that due to this issue the creation of cases to Salesforce Support via the Help and Training portal are failing.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The Salesforce Technology team is continuing its efforts to resolve this issue. It is understood that the impact is a result of a networking issue. Engineers are working with additional teams to understand the issue and devise a mitigation strategy.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

At 07:09 UTC, on June 20, 2023, the Salesforce Technology Team became aware of an issue resulting in a performance degradation for customers on multiple USA instances. Customers may experience Message Queue delays, delays in asynchronous processing, including Web-to-Case, Web-to-Lead, Email-to-Case, and dashboard refreshes. Engineers are actively investigating this issue and have engaged additional teams to assist.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.