Performance Degradation

The Salesforce Technology team continues to deploy the fix forward across the NA instances: NA83, NA105, NA116, and NA136. We are expecting the deployment to take several more hours and will closely monitor the progress.

Additionally, the Technology team noted sustained stability of services across the instances: EU44 as of 02:57 UTC, NA211 as of 00:57 UTC, and NA212 as of 07:25 UTC on 07 February 2023 respectively. Customers on these instances are out of the impact, following the remediation actions, as detailed in our earlier updates. As a result of varied impact duration, we have decided to update the individual Trust status pages for EU44, NA211, and NA212 instances to capture the most accurate impact times. Hence, customers on these instances will no longer receive further updates via this Trust post.

We will provide you an update once additional information is available.

The team confirmed that rolling back the change on NA211 brought customers out of impact in this instance. After further review and analysis, the team decided to roll back the change on EU instances to mitigate the impact on our customers. In parallel, the team is in the midst of developing a fix forward which will be deployed to the impacted EU and NA instances during the off-peak hours. The proposed fix is in testing and will take several hours to be completed.

We will provide you an update once additional information is available.

Upon further investigation, the Technology team identified a recent configuration change for database memory utilization as the likely trigger of the issue.
A rollback of change for the NA211 instance was validated and verified, and the team confirmed the performance levels are restored to normal. The team has initiated the pre-task for rollback to other impacted instances. We don’t have an ETA for completion at this time.

We will provide you an update once additional information is available.

Upon investigation, the team observed high database memory utilization resulting in intermittent connectivity issues between the application and database tier across these instances. The team continues to investigate the issue and explore multiple paths to resolution.

We will provide an update as soon as additional information becomes available.

At 13:17 UTC on February 6, 2023, the Salesforce Technology team became aware of performance issues impacting the following instances: NA212, NA211, NA136, NA116, NA105, NA83, and EU44. Exact timing and impacts may vary by instance. Currently, monitoring is indicating slightly higher than normal response times and customers may experience intermittent performance degradation. We are working to remediate this as quickly as possible, and we apologize for how this is impacting your business.

We will provide an update as soon as additional information becomes available.