Intermittent failure on all app networking calls (including product apis, egress, storage and resolvers)

No customer impact has been observed seen since mitigation began. We are now closing this incident.

We identified a networking issue that increased timeouts for fetch requests originating from Forge apps. A small portion (< 0.1%) of requests resulted in Connection Timeout errors. Other symptoms included timeouts in product API requests, storage calls and resolver executions.

Timeline:
2025-09-03 10:00 UTC: impact start
2025-09-15 16:56 UTC: impact mitigated

We will continue to monitor our mitigation and investigate the root cause.

We are continuing to investigate and mitigate the issue.
Upto 0.1% of all requests originating from Forge apps could result in Connection Timeout errors

We are continuing to diagnose the root cause and are attempting mitigation in affected regions. We will continue to monitor requests and report on any improvements.


Next Steps: We will apply the mitigation in other regions and continue to monitor. Next update will be within 4 hours.

We have identified a networking issue that is impacting fetch requests originating from forge apps. All forge-enabled products are impacted.

Impact start time: Thursday, September 4th

Symptoms include:
- product fetch call timeout
- egress fetch call timeout
- storage calls timeout
- resolver timeout

The following errors may be present in your app logs
- UND ERR CONNECT TIMEOUT
- ERR SOCKET TIMEOUT

We are currently investigating the root cause
- seeing evidence of unstable network connections between distributed infrastructure
- Packet loss
- TCP resends
- Eventual timeout on fetch operations

We will provide another update at Sep 15 7:00 AM UTC