Feature Disruption

Auto-scaling is functioning as intended and all our customers currently remain out of impact. We will continue to monitor the environment to ensure stability and prevent a recurrence.

We’ll provide an update as soon as new information becomes available.

The Salesforce Technology team completed deploying the fix to auto capacity increase functionality at approximately 03:00 UTC. As per previous updates, all customers currently remain out of impact.

We’ll continue to validate the fix by monitoring performance over the next several hours and adjust thresholds and capacity as necessary. Should the impact reoccur, we have prepared alternative measures to mitigate it.

We’ll provide an update as soon as new information becomes available.

The Salesforce Technology confirmed that the impact remains mitigated, and no further errors are observed. The team is deploying a fix for the auto capacity increase functionality.
We’ll continue to monitor performance over the next several hours and take precautionary steps to ensure stability during peak hours.

The deployment continues, and the next Trust update will be shared as soon as new information becomes available.

The Salesforce technology team has successfully completed increasing capacity on a production environment and, after a period of testing and validation, confirms that customer impact was mitigated on that pod since 21:46 UTC, November 18. The capacity fix is now being applied to all impacted pods and the Technology team will undertake a period of monitoring to assess the success of these efforts. It is expected that this may take approximately 1 hour to complete.

The team also identified that the auto capacity increase functionality was not working as expected and the team is implementing a fix to address this in other impacted instances.

Investigations continue, and the next Trust update will be shared as soon as new information becomes available.

Having been able to reproduce the issue in a test environment, the team continues to investigate multiple changes to fully restore services to customers. Two rollbacks are still being considered, which relate to a recent LiveAgent change, and a recent gater change. The team also focused on increasing capacity to try to mitigate the issues seen. The team also continues to explore traffic anomalies being seen across network load balancers. With customers on impacted instances now moving outside of core business hours, network traffic has eased and users are currently out of impact. However, users will likely experience similar issues again once network traffic increases as we come into core business hours tomorrow.

Investigations will continue overnight, and the next Trust update will be shared as soon as new information becomes available.

Our team has been able to reproduce the issue internally and believes this will significantly support remediation efforts. They have also identified a recent change that may be the trigger for this issue and a rollback of this change is being investigated. There is no ETA for the completion of this rollback at this time. A second recent change has also been identified, and a rollback of this change is also being considered.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

Upon further investigation, the Salesforce Technology team has determined that the start time of impact is different than initially understood. We have now revised the start time of the Trust post to more accurately reflect the time customers may have begun to experience impact. We apologize for any confusion caused by this.

The issue caused increased error rates related to cache values. Our team is fully engaged, analyzing logs and pinpointing the trigger while simultaneously exploring alternate solutions to mitigate the impact as quickly as possible.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

At 19:13 UTC on November 18, the Salesforce Technology team identified a disruption in Salesforce Chat, causing agent chats and voice calls to disconnect prematurely. We are actively working on multiple strategies to resolve the issue and restore services as quickly as possible.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances experiencing impact, and customers in these instances will receive communications related to this incident going forward.