Outage in LiftIgniter

Issues with services in US East due to capacity issues with cloud provider

Resolved Minor
April 28, 2022 - Started over 2 years ago
Official incident page

Need to monitor LiftIgniter outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including LiftIgniter, and never miss an outage again.
Start Free Trial

Outage Details

Due to some capacity issues being experienced by our cloud provider (Google Cloud) in US East, we are or were experiencing issues with some of our services. Our query endpoint (query.petametrics.com), that is used to serve recommendations, saw (503 status) error rates rise to about 1%. Error rates were nonzero between 18:00 and 18:04 UTC. We had already started provisioning alternate capacity prior to the increase in error rates, but still got some errors as the provisioning of capacity took a few minutes. We also saw increased latency in the period from 17:51 to 18:11 UTC for the successful requests. We also provisioned alternate capacity for a few other affected services; these services had a few minutes of downtime while the alternate capacity was coming online. We significantly benefited from preparation we did after the previous incident http://status.liftigniter.com/incidents/1522vrjxbmcp.
Latest Updates ( sorted recent to last )
RESOLVED over 2 years ago - at 04/28/2022 09:33PM

Capacity is back to normal and all configurations have been returned to their defaults.

MONITORING over 2 years ago - at 04/28/2022 06:21PM

Due to some capacity issues being experienced by our cloud provider (Google Cloud) in US East, we are or were experiencing issues with some of our services.

Our query endpoint (query.petametrics.com), that is used to serve recommendations, saw (503 status) error rates rise to about 1%. Error rates were nonzero between 18:00 and 18:04 UTC. We had already started provisioning alternate capacity prior to the increase in error rates, but still got some errors as the provisioning of capacity took a few minutes. We also saw increased latency in the period from 17:51 to 18:11 UTC for the successful requests.

We also provisioned alternate capacity for a few other affected services; these services had a few minutes of downtime while the alternate capacity was coming online.

We significantly benefited from preparation we did after the previous incident http://status.liftigniter.com/incidents/1522vrjxbmcp.

Latest LiftIgniter outages

Issues with inventory API and related background processing - 7 months ago
Issues with email service (and another component that is not end-user-facing) likely due to Google Cloud infrastructure issues - 7 months ago
Capacity issues in US East region due to Google Cloud infrastructure issues (they are unable to provision new instances) - over 1 year ago
Issues with console and the API usage - about 2 years ago
Issues with services in US East due to capacity issues with cloud provider - over 2 years ago

Start monitoring all your vendors in just 5 minutes

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3278 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime

Start today for FREE