Outage in Helium

Chain Halt at 1469951

Resolved Major
August 08, 2022 - 4 months ago - Lasted about 10 hours
Official incident page
Components affected
Helium API Helium Blockchain

Details

The core team is investigating the chain halt that occurred at block height 1469951 as a result of activating the latest chain variable. All funds are safe, but pending transactions will be delayed until we can resolve this halt. Rewards will be delayed until the chain can move forward. Pending Transactions include: - Add Hotspot - Assert Location - Transfer Hotspot - Payment Transactions - Burn Transactions
Updates ( sorted recent to last )
MONITORING at 08/09/2022 05:09AM

Update: Blocks are moving again but the number of transactions in each block is low (about 10% of expected txns). We still have not seen a reward block, so we're continuing to monitor. Updates for ETL and Node are currently being tested. Validators have their update already.

IDENTIFIED at 08/09/2022 03:04AM

Update: We have an emergency release going out to Validators (and another release for ETL, Node, and Router operators shortly). The emergency release includes a new snapshot that all Validators will sync to to realign ledgers (there was a ledger drift). Update again soon.

IDENTIFIED at 08/09/2022 12:37AM

Update: Just setting some expectations here. We expect this to be a long chain halt (hours +) and there will be several emergency releases going out to Validators, Node, and ETL operators. Announcement on when that's available will be made separately. Chain is halted, not dead. Again, pending transactions will not go through, rewards are affected as the chain is not moving, but it's affecting everyone equally.

IDENTIFIED at 08/08/2022 10:45PM

Update: We have halted again at 1,469,988 and getting more clarity at the root cause. We believe receipts in blocks cannot be absorbed. More details coming as soon as we have a plan.

IDENTIFIED at 08/08/2022 08:34PM

Update: The core team has asked all Validators currently in the Consensus Group to restart their Validators in order to resume block production.

INVESTIGATING at 08/08/2022 08:18PM

Update: We're still looking, but have a potential solution percolating. In the mean time, I want to also mention that device packets are still sending on the network despite the chain halt due to the fault-tolerant system the core team has put in place.

INVESTIGATING at 08/08/2022 07:52PM

Some nodes are able to progress to block 1469953 but not all. This issue may auto-resolve but the core team is still investigating.

INVESTIGATING at 08/08/2022 07:34PM

The core team is investigating the chain halt that occurred at block height 1469951 as a result of activating the latest chain variable. All funds are safe, but pending transactions will be delayed until we can resolve this halt. Rewards will be delayed until the chain can move forward. Pending Transactions include: - Add Hotspot - Assert Location - Transfer Hotspot - Payment Transactions - Burn Transactions

One status page with allyour external cloud services

Have you ever missed an important outage from a third-party service? We've built IsDown, so you never miss another outage again. It's the easiest way to monitor all your SaaS and cloud providers and get alerted when an outage impacts your business.

Start free trial

No credit card required · Cancel anytime · 2021 services available

Integrations with Slack Microsoft Teams Google Chat PagerDuty Zapier Discord Webhook

Are you able to monitor your cloud services in a real-time and consistent way?

The Old Way
  • Subscribe to status pages one-by-one
  • Limited to 0 notification options
  • Can't monitor only the parts that matter
  • No bird's eye view over all your services
  • Losing time looking for problems elsewhere
  • No access to historical issues and stats
With IsDown
  • Easily subscribe to all status pages
  • Normalized notifications sent to your tools
  • Monitor what matters
  • Easy access to the status of all your services
  • Outages information where it's needed
  • Historical data of outages for all your providers

"If you are in SRE, IT, or Security and work in an environment with a lot of SaaS (which, let's face it, is all of them) - IsDown is your new best friend. Helpfully aggregates various Statuspages from services into a very clear dashboard. Worth every penny."

Mike, Head of Security

IsDown is the missing layer in your monitoring stack

Quickly identify external outages that impact your business. We are monitoring more than 2000 services in real time.

Birds-eye view over all your services statuses

Check the status page aggregated of all your services in one place. No more going to each of the status pages and managing them individually.

IsDown Dashboard

Outage monitoring in real time

We monitor 24 hours a day, 7 days a week and will notify you if there is an incident. No more wasting time trying to figure out why something isn't working.

Alerts in your favorite channels

Get instant notifications in your email, Slack, Teams, or Discord when we detect a service outage. Outage monitoring where you are already doing your work.

IsDown Integrations

Easily integrate with your current tools and workflows

Using Zapier or Webhooks, you can easily integrate notifications into your processes. PagerDuty integration is also available.

Avoid notifications clutter

Configure which notifications you want to receive from each service. Filter notifications by service components. You can opt to receive notifications only when a specific component is affected. You can also choose to receive notifications with a certain severity.

Notify By Components
Multiple Dashboards

Have multiple dashboards. Easily shareable with the world.

Create one dashboard for each of your teams/clients/projects. Monitor only the services that each uses. Dedicated dashboard with custom notification settings. Easily make your dashboard public and share it with the world.

Prepare for scheduled maintenances

Never again be caught off guard by unexpected maintenance from your services. A feed of the next scheduled maintenances is available.

Weekly Digest of the services' outages

Every Monday, you'll receive a weekly summary of what happened the previous week as well as the maintenance schedule for the following week.

Integrate with tools you already use and love

The data and notifications you need, in the tools you already use.

For every team in your company

DevOps & On-Call Teams

You already monitor your internal systems. What about the external services? Monitor the services your business depends on. Don't waste time looking elsewhere when external outages are the cause of issues.

IT Support Teams

Detect external outages before your clients tell you. Anticipate possible issues and make the necessary arrangements. Having proactive communication, builds trust over clients and prevents flow of support tickets.

Simple Setup. Instant Value.

  1. Step 1 Create an account

    Start with a trial account that will allow you to try and monitor up to 40 services for 14 days.

  2. Step 2 Select your services

    There are 2021 services to choose from, and we're adding more every week.

  3. Step 3 Set up notifications

    You can get notifications by email, Slack, and Discord. You can also use Zapier or Webhooks to build your workflows.

  4. Step 4 Done!

    You'll start getting alerts when we detect outages in your external dependencies! No more wasting time looking in the wrong place!

Frequently Asked Questions

Is Helium down today?
Helium seems to be up and running. We've updated the status 2 minutes ago.
I'm having issues with Helium, but the status is OK. What's going on?
There are a few things you can try:
How can I be notified when Helium is having issues?
You can subscribe for updates on the official status page or create an account in IsDown. We will send you a notification in real-time when Helium has issues.
Why use IsDown?
IsDown is a status page aggregator, which means that we aggregate the status of multiple cloud services. Monitor all the services that impact your business. Get a dashboard with the health of all services and status updates. Set up notifications via email, Slack, or Discord when a service you monitor has issues or when maintenances are scheduled.
What happens when I create an account?
You'll have access to a 14-day trial in our Pro plan. You can cancel or delete your account anytime. After 14 days, you'll need to subscribe to continue to use the service and get notifications.
How can I pay for a subscription?
You can go to the Billing section in your account and choose one of the plans. We have monthly and yearly options. We accept all major credit cards, Apple Pay, and Google Play. We use Stripe for payments.
Can I get a refund?
We'll refund your subscription if you cancel it until ten days after the subscription has started. No questions asked.
Can't find a service/integration?
Just contact us, and we'll add it ASAP.

Setup in 5 minutes or less

Try it out! How much time you'll save your team, by having the outages information close to them?

  • 14-day free trial
  • No credit card required to start
  • Cancel anytime
  • +2000 services available