Quay.io Outage History

Quay.io experienced HTTP 502 errors that prevented users from pulling and pushing container images, caused by a problematic deployment. The team identified the root cause, reverted the deployment, and gradually restored traffic to the primary region, with pulls restored first followed by pushes. The incident was fully resolved after 3.1 hours of service disruption.

A gateway configuration mismatch caused cert-api.access.redhat.com to return HTTP 500 errors for certificate-based requests, impacting Red Hat Lightspeed registrations and data uploads for both direct and Satellite-connected hosts. The issue was resolved after 3.1 hours by implementing a configuration fix.

Quay.io experienced degraded performance affecting both push and pull API operations for 1.4 hours. The pull API was restored first, followed by the push API being brought back online. The service was fully operational with both APIs working as expected by the end of the incident.

Quay.io's backend service processed incorrect data for non-Pay as you go products, causing users to receive erroneous subscription threshold exceeded notifications with misleading usage values. The notifications capability within Subscription Watch was temporarily disabled to prevent further incorrect alerts from being sent. The issue was resolved after 312.6 hours with notifications for non-PAYGO products now functioning correctly.

Quay.io experienced API failures causing 502 errors that prevented users from pushing and pulling container images for 3.1 hours. Users reported receiving "502 bad gateway" errors when attempting image pulls and accessing podman containers. The service was temporarily set to read-only mode to restore pull functionality before a full fix was implemented to restore all operations.

Quay.io experienced SSO timeouts on access.redhat.com that prevented users from authenticating properly. The authentication service disruption lasted 23 minutes before the issue was identified and resolved. Users were unable to access Quay.io services that required Red Hat SSO authentication during this period.

Red Hat Enterprise Linux 9 users experienced intermittent failures when attempting to download specific packages, with the issue also affecting Red Hat Update Infrastructure (RHUI) and causing RHEL 9 AppStream repository synchronization failures on Red Hat Satellite 6 with duplicate key value errors. This was a recurring issue that had been previously reported and reopened during the incident period. The incident was resolved after 189.6 hours of investigation by the RHEL team.

Red Hat Enterprise Linux 9 users experienced intermittent failures when attempting to download specific packages, with the issue also affecting Red Hat Satellite 6 repository synchronization (causing duplicate key value errors) and Red Hat Update Infrastructure (RHUI). The incident lasted 67.9 hours and has been resolved.