Generalized latency issues throughout infrastructure

Event processing is now functioning normally.

Upstream is slowly recovering and our ingestion rate has increased. We will continue to monitor the situation as we move toward backlog processing.

We continue to monitor the upstream incident as we process incoming events as quickly as possible.

Google Cloud Platform has begin to mitigate the problem (https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW). Though slowly, we continue to ingest and process events during this incident and we will be able to begin the process of burning down backlogs after recovery.

We have received acknowledgement from our cloud provider that there is a wide-scale impact hitting their services. We will continue to work with them and prepare our systems for the return of service.

We have identified the problem as being with our cloud infrastructure provider and have contacted them for further diagnosis. Customer impact is delays in ingestion and intermittent failures for both EU and US customers.

We are currently investigating this issue which is impacting both US and EU localities.