Performance Degradation

The Salesforce Technology team continues monitoring the ongoing emergency release rollout, which is progressing as expected.

We’ll continue to monitor the situation closely and provide an update upon completion of the release or sooner if additional information becomes available.

The staggered fleetwide rollout of the emergency release is ongoing and progressing without issue.

We’ll provide another update upon the completion of the release or sooner if additional information becomes available.

The emergency release is progressing well for most impacted instances, while manual remediation is continuing for those that require it.

We’ll provide an update in 60 minutes or sooner if additional information becomes available.

The emergency release is continuing to progress well for most impacted instances, while manual remediation is continuing for those instances that require it.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The emergency release is continuing to progress well for most impacted instances, while those requiring manual intervention continue to progress at a slower rate.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances experiencing impact, and customers in these instances will receive communications related to this incident going forward.

Upon further investigation, the Salesforce Technology team has determined that the start time of impact is different than initially understood. We have now revised the start time of the Trust post to more accurately reflect the time customers may have begun to experience impact. We apologize for any confusion caused by this.

Additional impacted instances have been identified and we have updated the posting to reflect the additional instances experiencing impact. Customers in these instances will receive communications related to this incident going forward.

The emergency release is taking longer than expected for a small subset of instances due to the need for manual intervention. However, it has been progressing successfully for most impacted instances.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The emergency release continues rollout in a staggered fashion across all impacted instances. Customers may experience a brief window of service disruption when the release is rolled out to their instance.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Additional impacted instances have been identified and we have updated the posting to reflect the additional instances experiencing impact. Customers in these instances will receive communications related to this incident going forward.

The second phase of the emergency release rollout is in progress. Customers may experience a brief window of service disruption when the release is rolled out to their instance. Additional capacity has been added to all instances in an attempt to minimize the impact.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Engineers are continuing final preparations for the second phase of the emergency release. Due to the broad scope and timing of the issue, the team is evaluating how to execute the release with minimal additional impact on customers.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances experiencing impact, and customers in these instances will receive communications related to this incident going forward.

A capacity increase is currently being implemented to ensure a smooth transition to the next phase of emergency release (ERR). Meanwhile, we continue to validate the first stage of ERR and are closely observing the progress.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances BRA14, DEU120, NA224, USA352, USA558, USA600, USA612, USA708 experiencing impact, and customers in these instances will receive communications related to this incident going forward.

A number of customers are reporting that they are out of impact and are not seeing additional issues. The team continues to perform health checks. The first phase of the Emergency Release is nearing completion at which stage validation checks will take place.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The team is performing health checks on any instance that received the manual fix. Any customers who are seen to be experiencing any additional issues will be added to the schedule for the Emergency Release.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The first phase of the Emergency Release is progressing and is expected to complete within the next hour. We don’t have an ETA for the second phase of the rollout to begin, as validation will need to happen to ensure that the release resolves the issue for customers.
We will continue to add additional instances as impact is discovered.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The impacted instances that have received the manual fix forward will not generally be part of the Emergency Release (ERR) deployment. However, we have received reports from a subset of customers that although the manual fix initially resolved the issue, the issue reappeared. Customers who continue seeing impact after the manual fix will be added to the schedule to receive the Emergency Release.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

An Emergency Release (ERR) will be deployed to all impacted instances as part of a permanent fix. During the rollout of the ERR, customers will experience a brief period of service disruption. Meanwhile, a discussion with the third-party vendor is ongoing to add capacity to accommodate the ERR without additional impact.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

We are also aware of intermittent accessibility issues with the Salesforce Trust site. The Technology teams have fixed the issue. Affected customers may need to refresh their browser after which the Salesforce Trust site should be accessible.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances of CHE30, DEU66, DEU10s, DEU38, DEU106, GBR118, SWE24, SWE66, USA1 experiencing impact, and customers in these instances will receive communications related to this incident going forward.

More than half of the impacted instances have implemented the forward fix. A subset of customers who received the fix should no longer experience the impact, and their services should work as expected. The longer-term fix is still being discussed.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances DEU8s, DEU54, DEU68, DEU84, DEU124, EU51, EU52, FRA34, FRA44, FRA66, FRA104, GBR36, GBR50, GBR66, GBR68, GBR80, IND80, IND84, IND88, IND108, SWE78, experiencing impact, and customers in these instances will receive communications related to this incident going forward.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

An emergency release plan is in place to perform the fix forward across a number of instances. We have received positive results after the fix has been tested in the staging environment, and it will be released to production shortly. Meanwhile, the Hyperforce instances will receive the fix forward in a phased manner. Rolling out the fix may introduce a period of service disruption to customers where customers will not be able to access their services.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances experiencing impact, and customers in these instances will receive communications related to this incident going forward.

Further investigations have highlighted that DEU58, DEU78, DEU94, DEU108, DEU110, FRA42, FRA46, FRA56, FRA76, FRA90, GBR38, GBR46, GBR70, GBR76, GBR86, GBR96, GBR98, GBR100, ITA30, SWE8, SWE20, SWE18, SWE26, SWE52, SWE72, USA1000 are also impacted and will start receiving incident-related communications. The fix forward release implementation continues to progress on the impacted instances.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The manual fix forward on IND70, IND120, and IND132 instances has resolved the customer impact. The teams are now working on implementing the fix forward to the remaining impacted instances in a phased manner. Further investigations have highlighted that DEU126, FRA36, FRA54, FRA70, FRA88, GBR72, GBR102, IND98, ITA16, SWE38, SWE56, SWE70, and SWE76 are also impacted and will start receiving incident-related communications.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

The teams are continuing to implement the fix forward in a staggered manner to avoid additional impact. Further investigations have highlighted that DEU86, FRA84, GBR54, IND122, IND124, ITA12, SWE56, SWE72 and SWE98, are also impacted and will start receiving incident-related communications.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have determined that FRA100, IND98, USA618, DEU76, IND104, DEU56, FRA52, SWE98, SWE32, DEU70, IND102, and ITA28 are also impacted and will start receiving incident-related communications.
Additionally, the teams implemented the manual fix forward to IND132 instances and are seeing positive results. IND70 is receiving the fix forward now and we are closely monitoring.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Further investigations have highlighted that FRA80 is also among the impacted instances, and it will start receiving incident-related communications. We have also identified the trigger of this issue is at the application level related to the encryption logs.
Additionally, the teams are working on performing a fix forward to one of the impacted instances, IND132, and observing the performance levels.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Further investigations determined that the incident start time has been revised to 06:30 UTC to align with multiple impacted instances for which alerts were received. IND120 has also been added to the list of the impacted instances, and it will start receiving communications related to the incident. Simultaneously, the team is working on testing a manual fix to restart application servers.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

Investigations have highlighted that the impact radius is broader than initially understood. We will now add IND132 to the list of impacted instances, and this instance will start receiving communications about this incident. We will continue to update impacted instances as we determine the full scope of the impact.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.

At 03:04 UTC on October 1, 2024, the Salesforce technology team noted a performance degradation impacting the IND70 instance. Upon initial investigation, the team observed an issue at the application tier as the likely cause of the issue. We are exploring multiple paths to resolution.

We’ll provide an update in 30 minutes or sooner if additional information becomes available.