Investigating Salesforce Integration Disconnection

We are considering this issue fully resolved. We have monitored our incoming logs and confirmed the disconnects have stopped occurring since we rolled out our fix to all customers.

The root cause was confirmed to be the update in the "Summer 24" release from Salesforce that blocked simultaneous refresh tokens.

We resolved the issue by adding a more robust retry strategy in addition to better logic in how we handle refresh tokens. As these changes were not trivial and required significant engineering resources, the fix took longer than usual to develop.

As an outcome, the retry mechanism will remain in place and our notification emails related to dropped connections were adjusted to better guide admins should any unrelated disconnect issues occur in the future.

Our fix is now deployed to our production environments, and we are now monitoring in realtime to validate that the fix is working as expected.

Please make our team aware of any additional disconnects you experience via support@chilipiper.com so that we can act quickly to investigate it further.

We have successfully replicated and validated the issue's root cause through testing.

Our solution is confirmed to resolve the disconnects and will be deployed to our production environment shortly.

Once it has deployed to all customers, we will provide another update to monitor the fix before confirming it is fully resolved.

We have begun deploying some solutions to our testing environments to start validating them internally. Once we have completed tests we will start rolling it out to customers to monitor the fix.

We have also updated our automated email sent to admins in order to make it clearer where to go in order to reconnect. Previously it had indicated the personal-level connection, but please be sure to reconnect the global admin-level connection as the new notification indicates.

We understand the urgency and concern this incident has caused. The resolution is taking longer than anticipated due to the need for significant changes in our authentication infrastructure rather than if it was a more trivial bug.

This issue has been escalated internally and is our top priority. We've allocated additional engineering resources to expedite the fix so we can get it rolled out as soon as possible.

We aim to have a solution deployed to our testing environment before the end of today. Once verified through quality assurance, it will be rolled out to all customers.

In the meantime, please continue to monitor your integration user email for any Salesforce disconnects.

Thanks again for your patience and understanding.

We are currently investigating an issue affecting the global Salesforce connection (integration user) for a small subset of customers using our legacy product line. Please note that our Demand Conversion Platform products and HubSpot CRM customers are not impacted.

While disconnected, there may be issues with creating or updating Lead records or Events. Additionally, routing will be limited to form data and will not function properly based on existing Lead/Contact data.

Our preliminary findings suggest that recent updates from Salesforce, which block specific refresh token strategies, may be the underlying cause. We are working on a fix to implement a more robust refresh token logic.

We have confirmed that a notification email is sent to the integration user email account when a disconnection occurs, prompting them to reconnect as soon as possible.

We are actively working on a solution. In the meantime, please be aware that the service may disconnect again even after reconnecting. To mitigate this, we have implemented active monitoring to detect disconnections promptly and may reach out directly if needed.

We urge admins to monitor their integration user inbox closely to ensure timely action in the event of a disconnection.

Thank you for your understanding and patience as we resolve this issue.