Increase API Error Rate and Push Error Rate

Typing events have been re-enabled. At this point all functionality has been restored and the service appears to be operating as designed. Oncall Engineering will continue to monitor the service and will work to understand the root cause of this issue with our provider partners.

Oncall Engineering has brought the media infrastructure back online, media embeds should be functional again. Message acknowledgement has also been re-enabled. We are continuing to work to restore full functionality. Typing events are still disabled at this time and we are investigating reports from bot developers regarding unexpected 403 errors.

Remediations are working and traffic is coming back online. While we work to restore full service some functionality will remain intentionally disabled until the service stabilizes, typing events and message acknowledgement. Other functionality remains to be restored, media embeds may not work correctly at this time.

We are working with our providers to correct the root cause. We believe the cause is upstream of our service and our providers are working on determining and correcting the issue upstream. In the interim we are implementing a set of remediations to work around the issue. As the service comes back up some functionality will be intentionally disabled, namely, typing events and message acknowledge.

While we continue to investigate the root cause, work has begun on restoring service by working around the issue. Oncall Engineering will begin allowing more traffic through as we restore service.

Oncall Engineering continues to investigate the root cause of this issue. We have engaged our partners and are preparing contingencies to restore service.

We are continuing to investigate the issue impacting the API to find root cause.

While monitoring this issue a new issue has occurred causing an major outage of the API. Oncall Engineering is working to correct this situation.

As part of recovery, the root cause was also detected in our streaming service. A controlled restart was performed of this service which would have caused a temporary disruption of streaming, this should be operating correctly at this time.

Remediations appear to have restored service to normal operation, Oncall Engineering will monitor for full recovery

The root cause has been determined, remediations have been executed to restore service.

We are currently investigating an increase in API Errors and Push Notification Errors.