Major outage in Konnect Middle East region due to loss of mutliple availability zones

Current Status Summary for Konnect Middle East Region:

All services have been moved to our failover region in Middle East
Almost all services are operational, with 2 caveats:
- Analytics data is delayed several hours. Konnect is accepting and persisting Analytics data, but it is not yet visible or queryable as we continue investigating restoring service.
- The Debugger is operational including trace data; however session logs will not be available in this region.

Everything else is back online. The team will update as we make progress on these remaining items.

Current Status Summary for Konnect Middle East Region:

All services have been moved to our failover region in Middle East
Almost all services are operational, with 2 caveats:
- Analytics data is delayed several hours. Konnect is accepting and persisting Analytics data, but it is not yet visible or queryable as we continue investigating restoring service.
- Debugger is unable to accept incoming trace data. While debugging sessions can be started, none of the trace data will be persisted in Konnect until the datastore for traces is restored.

Everything else is back online. The team will update as we make progress on these remaining items.

The data store for traces sent to Debugger is still experiencing issues in the Middle East region. Debugging sessions can be started but no spans will be able to be ingested. The team will update as they bring that service online.

Service remains restored, and Kong engineers will continue to monitor the apps in the new region.

User permission data is now propagating normally again.

Status Update (PDP – ME-SOUTH-1)

User permission data replication in ME-SOUTH-1 (Bahrain) region are experiencing extended latencies. Mitigation has been implemented to cap this to 15 minutes.
Updated data should now propagate within a maximum of 15 minutes.
We continue to monitor the situation and will provide further updates as needed.

The following services have been restored:

- All Konnect UI-accessible products except for Event Gateway and Metering
- Dataplane connections to the Konnect Control Plane for configuration download and analytics upload

Analytics data is delayed several hours. Konnect is accepting and persisting Analytics data, but it is not yet visible or queryable as we continue investigating restoring service.

The Kong engineering team is now verifying restored services in the Middle East region

We have restored partial service to Konnect. The UI is working again for Gateway Manager and Dev Portal, as well as all backing APIs.

Dataplane connections to the Control Plane and Telemetry endpoints have been restored for 3 hours and should continue to work.

We continue to make progress on restoring to a failover region for the Middle East region. Key AWS services to support our failover efforts are beginning to work as of 11am PST. The Konnect engineering team is working to restore and validate all services and data and will update in one hour.

Please note the control plane and telemetry endpoints are already restored, so dataplanes can connect to Konnect.

Control Plane restored for Config Download and Telemetry

We have restored the control plane and telemetry endpoints. Dataplanes can now connect to the control plane api at `me.cp0.konghq.com` and `me.tp0.konghq.com`

This is a read-only instance set up for this incident. Configuration changes are still blocked. But now customers should be able to connect new dataplanes and download their configuration, as well as send telemetry.

The Admin API for changing control plane configuration remains down.

We are actively migrating the affected services to operate on other Availability Zones in the Bahrain region (ME-SOUTH-1 Region), but critical AWS services are not responding in that region. Our engineers are working to restore full service functionality and we are communicating with AWS to attempt to determine our failover options. At this time we cannot provide an ETA.

We will provide further updates as they become available. Thank you for your patience.

We are actively migrating the affected services to operate on the two healthy Availability Zones in the Bahrain region (ME-SOUTH-1 Region). Our engineers are working to restore full service functionality and expect the migration to be completed within the next few hours.

We will provide further updates as they become available. Thank you for your patience.

The incident is ongoing. We are working closely with our cloud provider to monitor restoration efforts.
Due to significant physical infrastructure damage in both the UAE and Bahrain regions, recovery is expected to take more time.
We appreciate your patience as we work toward a full resolution.

We are currently observing renewed elevated error rates across multiple Konnect services in ME region related to the AWS outage in that region.

We are seeing many services in ME region recovering and some services are in degraded state. Dataplanes are not affected as part of this incident. All other Konnect regions are 100% operational and healthy.

Our engineers are closely monitoring the situation. We will provide additional updates as soon as more information becomes available.

We are currently observing renewed elevated 5XX error rates across multiple Konnect services in ME region. Our engineering team is actively investigating the issue and assessing customer impact.

We will provide additional updates as soon as more information becomes available.

AWS continues to experience issues some services. However most of Konnect services are now recovered and operating as expected. Our engineers are closely monitoring the services and it's operational efficiency.

The increased error rates are associated with the AWS me-central-1 outage. Our engineers are monitoring the situation closely.

We are seeing increased 5xx in the ME Region. Our engineers are currently investigating the issue.