Kong Outage History

From 14:00UTC to 15:38UTC, some Kong Konnect customers in the US region were unable to update custom plugins due to a validation service failure that prevented schema and handler code validation from the control plane. The issue affected Kong Konnect Cloud components for 1.6 hours until the team reloaded the validation service, restoring normal plugin update functionality.

Developer portals in the Singapore region became unreachable due to a networking load balancer issue that prevented access to the gateway, while existing app registrations and dataplane traffic remained functional. The team identified the networking problem as the root cause and implemented a fix. The incident was resolved after 23 minutes of monitoring to ensure the portals were fully accessible again.

The Kong Konnect Control Plane in the US East region became unavailable, preventing new dataplanes from pulling configurations while existing dataplanes with valid configurations continued operating normally. The control plane recovered after 35 minutes of downtime, with the team monitoring for stability before declaring the incident fully resolved.

Kong Konnect Cloud's analytics service in the AU (ap-southeast-2) region experienced issues for 13 minutes due to an incorrect node configuration. The engineering team deployed a fix to correct the configuration, and all analytics services recovered with data returning to normal. No other regions were affected during this incident.

Kong Konnect Cloud experienced a major incident where analytics data from data plane nodes in the US region was being lost for 3.1 hours. The engineering team identified the root cause, deployed a fix, and monitored the system to ensure stability. The incident was fully resolved with analytics data collection restored to normal operation.

Kong's 3.14 release introduced "secure by default" changes that caused plugin configuration defaults for ssl_verify and hide_credentials to flip from false to true, breaking existing configurations for customers using deck sync without explicitly defining these values. This affected numerous plugins including authentication, AI, logging, and proxy plugins, causing unexpected behavior changes when configurations were updated. Kong resolved the incident by rolling back to the original default values after 5.3 hours.

Kong Konnect Cloud experienced degraded functionality for Personal Access Token (PAT) management operations, affecting the ability to create, read, and delete PATs through both the API and UI. Authentication using existing PATs remained unaffected during the incident. The issue was resolved after 1 hour of monitoring and remediation.

Kong Konnect Cloud experienced a major incident where Dev Portals became unreachable in the ME region for 2.5 hours. The service was restored and monitored for stability before the incident was fully resolved.

Kong experienced a major service degradation in the ME region that impacted the DevPortal, Debugger, and node listing functionality within Kong Konnect Cloud. The issue was identified as a problematic instance that required removal, and the service was restored after 23.4 hours of downtime.

Kong Konnect's Singapore region experienced partial degradation affecting portal emails and application registration, causing delays in message processing due to a backlog in the message handling infrastructure. The incident was complicated by concurrent cloud service provider maintenance that initially prevented the team from scaling up infrastructure to address the backlog. The issue was resolved after 3.3 hours once the cloud provider completed maintenance and the message handling infrastructure was scaled up to process the accumulated backlog.