Intermittent connection issues between CDN and AWS Clusters

After many hours of work together with Fastly and AWS the root cause has been found and resolved. The workaround has been removed in April 2023. Active monitoring over the last weeks shows that the connection issues have been permanently resolved.

We are continuing to monitor and trying to find the root cause of this issue, as there are many different engineering teams involved this takes time. We are though very certain that the current implemented workaround solves the issues and therefore there should be no impact on customer websites from this issue.

We will keep this issue open and update it as soon as we found the root cause issue and a permanent resolution.

Over the last 7 days environments that are using the amazee.io CDN (Fastly) and are hosted on AWS clusters have experienced elevated connection issues. While this only affected a very small amount (less than 0.01%) of requests, we started to analyze and investigate this issue together with the teams at Fastly and AWS. While we have not found the exact root cause yet, we found a workaround on the AWS Loadbalancers that reduces the connection issues to regular levels expected of regular internet connection issues.

We are continuing to monitor this issue and trying to find the root cause together with Fastly and AWS. We will continue to provide updates here.