Performance Degradation

Performance has remained stable. The team is continuing to monitor and is taking additional actions to prevent to potential for a recurrence. We’ll provide an update in 30 minutes or sooner if additional information becomes available.

All indications point to service being fully restored when the restarts were completed at 18:06 UTC. The team is continuing to monitor to ensure performance remains healthy. We’ll provide an update in 30 minutes or sooner if additional information becomes available.

We have confirmed that the Technology team has now completed all restarts of the load balancers. We continue to receive positive indications that the trend toward a return to expected service levels is continuing. The team is continuing validations and health checks to confirm.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Monitoring indicates and customers have communicated that performance has improved. The team is validating to confirm that the actions taken are continuing to have a positive impact on performance.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

We previously incorrectly communicated that the restart of load balancers had been completed at 17:00 UTC. That restart is being done in a staggered manner, and additional load balancers continue to undergo restarts.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The team is currently performing a staggered restart of the load balancers in an attempt to alleviate traffic congestion issues. Concurrently, they are exploring the possibility of re-routing traffic as an additional mitigation path. The team is all hands on deck working to restore service to a healthy state as quickly as possible.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The Technology team continues to investigate the alerts. The network traffic spikes associated with these alerts have subsided and the load balancer traffic for the affected instances has returned to normal levels. Our teams are currently monitoring the network activity closely and performing the necessary health checks to ensure the issue does not reoccur.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

After reviewing additional diagnostic logs, the Technology team identified additional sandbox and production instances impacted as a result of this issue which are listed as CS236, CS241, CS242, CS243, CS244, CS245, CS256, CS261, CS262, CS263, CS264, CS265, CS266, CS267, CS281, CS282, CS283, NA226, NA239, and NA242. The team also noted potential connectivity issues at the network tier and have implemented necessary remediation steps that are trending positive outcomes. The teams are continuing to monitor the progress of the network activity and performing necessary health checks.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

At 12:21 UTC, on December 06, 2023, the Salesforce Technology team identified a performance degradation impacting the instances NA224, NA231, NA232, NA236, NA237, NA240, NA243, NA244, and NA245 within the Salesforce production environment. The Technology team continues to investigate the trigger of this issue and explore multiple paths to resolution.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.