Delayed Message Integration

As of 15:11 UTC, all queued notifications, including iOS push notifications delayed as a side effect, had either been successfully sent or were marked as permanently failed.

As of 16:48 UTC, our internal monitoring indicated error rates for sending push notifications to Android devices via Google's services had stabilized at normal levels.

At 17:05 UTC, Google confirmed their corresponding incident had been resolved as of 16:51 UTC.

Given both observed and confirmed upstream resolution, we're marking this incident as resolved.

We are continuing to experience a slight elevation in error rates when attempting deliver push notifications to Android users due to slow responses with Google's notification service although failed notifications are eventually sent to users due to our attempts to retry notifications that fail to send.

Google has indicated that their mitigation to drop the errors rates has made an improvement, but are still working to bring the notification service response times down to normal operation.

We are continuing to monitor our services, and will post an update when new information becomes available.

Google has indicated that their error rates for sending push notifications have decreased, and we have observed that we are successfully delivering push notifications to users.

We are going to continue to provide updates here every 30 minutes, and will not resolve this incident until Google indicates they have resolved their service outage.

We are continuing to monitor errors with delivering notifications to Android users. Google has indicated that they are putting mitigation in place on their notification services.

Google continue to have an incident with delivery push notifications to android app users.

Due to a mitigation put in place iOS users should begin to see notification delivered to their devices.

The cause of the backlog of notifications failing to be sent to app users is caused by a Google outage of their notification services.

We are monitoring our system, and investigating ways to mitigate overall impact of the outage to users.

We have a put a mitigation in place, and have caught up on the backlog of messages that began being queued to be integrated around 13:04.

At this time, we also believe that notifications to app users are delayed. We are investigating on the cause of this to resolve notifications.

We are continuing to investigate the issue around delivering messages to messaging web services. We believe we have found a root cause and are working to put a mitigation in place to get messages flowing to integrations again.

We are currently experiencing a delay integrating messages sent by drivers to message integration services. Messages sent since 12:55 UTC are affected by this delay in integration.

We will post updates here every 20 minute during the incident, or when a new update is available over the course of our investigation.