Trouble with Slackbot and AI-powered features

Current status and actions being taken: We've deployed an emergency fix to address the issues contributing to intermittent errors with Slackbot and Workflows. We're now seeing significant recovery across all affected services and are continuing to monitor the situation closely to ensure performance remains stable. 

We'll provide another update in 60 minutes.

Current status and actions being taken: We’ve deployed a configuration fix, resulting in significant improvements to Slackbot success rates and health metrics. We're now monitoring the situation as we investigate a secondary cache issue that may be contributing to intermittent errors or degraded workflow performance.

We'll provide another update in 60 minutes.

Current status and actions being taken: Upon further review, we've identified issues with secondary caching and are currently working on a configuration fix. In parallel, we're continuing to investigate and assess several solutions to restore ‌services.

We'll provide another update in 60 minutes.

Current status and actions being taken: The code fix hasn't successfully addressed the impact and we're resuming our investigations. 

We'll provide another update in 60 minutes.

Current status and actions being taken: We've begun deploying the code fix to production. Once complete we'll begin monitoring the performance of Slackbot, as well as our other AI-powered features.

We'll provide another update in 60 minutes.

Current status and actions being taken: We believe we've identified an issue that's still causing impact to occur even after our upstream provider has resolved impact from their end. We expect a code fix to take at least an hour to complete and pass through various test environments. 

We'll provide another update in 60 minutes.

Current status and actions being taken: We're continuing to engage with our third-party provider to find a solution. No further alleviative actions have been undertaken since our last update.

We'll provide another update in 60 minutes until the impact is resolved for all users.

Current status and actions being taken: We've taken steps to try to reduce the impact on Slackbot and its response capabilities. Customers may notice signs of improvement, however, the impact may still be intermittent until full resolution. 

Other AI-powered features, including Search Answers, Meeting Notes, and Translation, are still experiencing issues due to the ongoing upstream problem. We're investigating similar alleviative actions for these features.

We'll provide another update in 60 minutes until the impact is resolved for all users.

Current status and actions being taken: We're continuing to engage with our third-party provider to find a solution. We're also exploring multiple avenues on our side to try to alleviate impact on Slackbot and AI-powered features until the upstream issue is resolved. 

We'll provide another update in 60 minutes until the impact is resolved for all users.

Our work on this issue is still ongoing. Users may continue to experience intermittent trouble with Slackbot responses and other Slack AI-powered features such as Search Answers, Meeting Notes, and Translation.

Current status and actions being taken: We remain engaged with our third-party provider. 

We'll provide another update in 60 minutes until the impact is resolved for all users.

Our work on this issue is still ongoing.Current status and actions being taken: Users may continue to experience trouble with Slackbot responses and other Slack AI-powered features. We've identified the trigger as a third-party infrastructure issue and are working with the provider to resolve this issue. We'll provide another update in 30 minutes until the impact is resolved for all users.

We're investigating a new incident.

Current impact to end users: Some users may be experiencing trouble with Slackbot responses. Other Slack functionality that leverages AI services may also be experiencing issues. We're on the case and we'll provide an update within 30 minutes including further details on current actions being taken.