Impacts due to AWS outage

At 00:03 UTC on 10/21 error levels for the Toast integration returned to baseline. As this was the last remaining impacted Omnivore service, we are marking this incident as resolved.

We are still seeing a near 100% error rate for the Toast integration. For further updates, check the Toast status page (https://status.toasttab.com).

We are still seeing a near 100% error rate for the Toast integration. We will continue to monitor for improvement.

Around 19:27 UTC, we observed a spike in the error rate for our calls to the Toast API, approaching a 100% failure rate. At this time there are no actions we can take and we will continue to monitor the situation.

Around 17:07 UTC the number of degraded agents returned to baseline levels. We have not observed any other spikes in the error level for Brink API. We are currently not seeing any other areas of concern, but since AWS's incident is still open, we will continue monitoring all of our services for new issues.

Around 07:11 UTC this morning, Amazon Web Services began experiencing an outage (https://health.aws.amazon.com/health/status). The Omnivore team was alerted at 07:24 UTC and has been monitoring ever since. So far, we haven't seen any clear signal of disruption in Omnivore services other than the following:
* ~2.4k agents have entered a degraded state. They are not going offline, so API calls to these locations are succeeding, however customers may be experiencing slower than normal response times.
* The Brink API experienced elevated error rates from 15:21-15:41 UTC. Omnivore customers would have also seen an elevated error rate for Brink locations and possibly delayed webhooks.

We will continue to monitor for new impacts and will send out updates as needed.