Outage in amazee.io

Lagoon API unavailability and slowness

Resolved Major
August 09, 2023 - Started over 1 year ago - Lasted 19 days
Official incident page

Need to monitor amazee.io outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including amazee.io, and never miss an outage again.
Start Free Trial

Outage Details

Currently experiencing degraded API performance
Latest Updates ( sorted recent to last )
RESOLVED over 1 year ago - at 08/28/2023 02:10PM

We're closing the incident - The earlier-mentioned changes show that the API stability is back to normal levels.

MONITORING over 1 year ago - at 08/23/2023 07:00AM

We have identified the most likely root cause of the slowness and stability issues over the last couple of weeks. We have rewritten the relevant code and deployed, and are monitoring closely. All signs are currently positive, and services are running normally.

IDENTIFIED over 1 year ago - at 08/22/2023 07:00AM

We're seeing the issue returning, leading to SSH and API timeouts. We will monitor and work on short term improvements, as required.

MONITORING over 1 year ago - at 08/21/2023 01:32PM

Performance of the API and Dashboard have improved, cause was a high amount of messages and requests to be handled by the API. We are continuing to monitor and work on improvements to be able to handle the additional load.

IDENTIFIED over 1 year ago - at 08/21/2023 12:51PM

Unfortunately we're seeing problems again with the performance of the API and Dashboard, we're working on identifying the problem and fixing it.

MONITORING over 1 year ago - at 08/15/2023 05:08AM

We have implemented a fix, we are continuing to monitor the situation

INVESTIGATING over 1 year ago - at 08/14/2023 11:57AM

We are continuing to investigate this issue.

INVESTIGATING over 1 year ago - at 08/10/2023 06:39AM

Issues with the API have started again, we are investigating

MONITORING over 1 year ago - at 08/09/2023 02:24PM

We've put measures in place to stabilize API and the Lagoon Dashboard. There might be slow responses, and our team is working on getting everything back to speed.

As we focus on fully resolving this issue, the updates regarding this incident may become less frequent.

IDENTIFIED over 1 year ago - at 08/09/2023 12:22PM

We're seeing the issue returning, leading to SSH and API timeouts. Our team is investigating.

IDENTIFIED over 1 year ago - at 08/09/2023 12:14PM

The issue has been identified and a fix is being implemented. Some customers might see intermittent SSH connectivity issues.

MONITORING over 1 year ago - at 08/09/2023 11:35AM

The limitations that have been put in place were successful, and we were able to scale up to the standard capacity of the API.

The API and Lagoon Dashboard are operating normally. Although customers may encounter intermittent delays in API response times.

We continue to monitor the situation and take appropriate action if needed.

IDENTIFIED over 1 year ago - at 08/09/2023 09:06AM

The issue has been identified - Currently, there are a few limitations in place to see how the situation stabilizes and we're working to open up the API, Dashboard and SSH connections to full capacity again.

INVESTIGATING over 1 year ago - at 08/09/2023 07:47AM

We continue to investigate the issue. There might be temporary issues with SSH connections.

There seems to be an unusual amount of API request volume, which causes issues with the Lagoon API, Dashboard and SSH connections. We are looking into isolating the problem and putting limits in place.

MONITORING over 1 year ago - at 08/09/2023 05:27AM

API is stable, but may be slow as things recover

INVESTIGATING over 1 year ago - at 08/09/2023 05:04AM

We are continuing to investigate this issue.

INVESTIGATING over 1 year ago - at 08/09/2023 04:30AM

Currently experiencing degraded API performance

Latest amazee.io outages

GCP MySQL 8 Upgrade - 8 days ago
Failing image builds - about 2 months ago
JSM Assist sync issues - about 2 months ago
Delayed logs - 3 months ago

Start monitoring all your vendors in just 5 minutes

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3643 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Never again lose time looking in the wrong place

14-day free trial · No credit card required · No code required