Forge authenticated egress intermittent failure

Between 06:58 12 Oct UTC to 18:30 12 Oct UTC, we experienced calls from a some forge apps to external services (outside of Atlassian) intermittently fail. The issue has been resolved and the service is operating normally.

We have identified the root cause of the intermittent errors and have mitigated the problem by putting in a temporary measure.

We are now monitoring closely.

We are deploying a fix that will alleviate some of the issues.
We are still investigating the root cause behind the intermittent failures.
Our team continues to closely monitor the errors and explore additional mitigations to ensure stability.

Impact
The issue is affecting Forge Egress within the Ecosystem, leading to intermittent failures in authenticated egress calls.
This has impacted functionalities such as Jira Incoming Mail, causing delays in fetching emails from providers like Gmail and Microsoft, and consequently affecting work item creation.
Additionally, Forge Compass apps are currently failing to send data to Compass.

Current Status
We have identified the root cause and are in the process of deploying a fix.

Next Steps
Post-deployment, we will continue to closely monitor the fix to ensure stability.
The next update will be provided in 60 minutes.