Delays and intermittent drops in log ingestion across multiple projects

This incident has been resolved.

We’ve implemented a temporary workaround and 503 errors are back to normal levels.

Our upstream provider is still working on a permanent fix. We will continue to follow the situation closely and provide further updates.

Our upstream provider has identified the issue and is working on a fix. Log ingestion is slowly getting back to normal, however we are still seeing elevated error levels.

We will continue to monitor and provide updates as the situation improves.

Our upstream provider has acknowledged the issue and their team is actively investigating the root cause. In the meantime, we are exploring workarounds to minimize the impact on log ingestion.
We will continue to monitor the situation closely and provide updates as we receive more information.

We’re experiencing delays and intermittent drops in log ingestion across multiple projects.

Impact
All products are impacted to varying degrees.
Where possible we will look to retry and backfill but some logs may be permanently lost.

The cause of the issue looks to be rate limiting by an upstream provider and we are working with them to resolve the issue