Outage in Helium

Chain Halt at 1469951

Resolved Major
August 08, 2022 - Started over 1 year ago - Lasted about 10 hours
Official incident page

IsDown continuously monitors the Helium status page and multiple sources to provide up-to-date information. Check back with us for the latest Helium uptime status.

Outage Details

The core team is investigating the chain halt that occurred at block height 1469951 as a result of activating the latest chain variable. All funds are safe, but pending transactions will be delayed until we can resolve this halt. Rewards will be delayed until the chain can move forward. Pending Transactions include: - Add Hotspot - Assert Location - Transfer Hotspot - Payment Transactions - Burn Transactions
Components affected
Helium API Helium Blockchain
Latest Updates ( sorted recent to last )
MONITORING over 1 year ago - at 08/09/2022 05:09AM

Update: Blocks are moving again but the number of transactions in each block is low (about 10% of expected txns). We still have not seen a reward block, so we're continuing to monitor.

Updates for ETL and Node are currently being tested. Validators have their update already.

IDENTIFIED over 1 year ago - at 08/09/2022 03:04AM

Update: We have an emergency release going out to Validators (and another release for ETL, Node, and Router operators shortly).

The emergency release includes a new snapshot that all Validators will sync to to realign ledgers (there was a ledger drift).

Update again soon.

IDENTIFIED over 1 year ago - at 08/09/2022 12:37AM

Update: Just setting some expectations here. We expect this to be a long chain halt (hours +) and there will be several emergency releases going out to Validators, Node, and ETL operators. Announcement on when that's available will be made separately. Chain is halted, not dead.

Again, pending transactions will not go through, rewards are affected as the chain is not moving, but it's affecting everyone equally.

IDENTIFIED over 1 year ago - at 08/08/2022 10:45PM

Update: We have halted again at 1,469,988 and getting more clarity at the root cause. We believe receipts in blocks cannot be absorbed. More details coming as soon as we have a plan.

IDENTIFIED over 1 year ago - at 08/08/2022 08:34PM

Update: The core team has asked all Validators currently in the Consensus Group to restart their Validators in order to resume block production.

INVESTIGATING over 1 year ago - at 08/08/2022 08:18PM

Update: We're still looking, but have a potential solution percolating. In the mean time, I want to also mention that device packets are still sending on the network despite the chain halt due to the fault-tolerant system the core team has put in place.

INVESTIGATING over 1 year ago - at 08/08/2022 07:52PM

Some nodes are able to progress to block 1469953 but not all. This issue may auto-resolve but the core team is still investigating.

INVESTIGATING over 1 year ago - at 08/08/2022 07:34PM

The core team is investigating the chain halt that occurred at block height 1469951 as a result of activating the latest chain variable.

All funds are safe, but pending transactions will be delayed until we can resolve this halt. Rewards will be delayed until the chain can move forward.

Pending Transactions include:
- Add Hotspot
- Assert Location
- Transfer Hotspot
- Payment Transactions
- Burn Transactions

The best way to monitor Helium and all your cloud services

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3153 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime