Outage in Helium

Increased Block and Election Times due to Oracled IOT PoC

Resolved Minor
February 05, 2023 - Started almost 2 years ago - Lasted about 4 hours
Official incident page

IsDown continuously monitors the Helium status page and multiple sources to provide up-to-date information. Check back with us for the latest Helium uptime status.

Outage Details

This is placeholder comms for a decision made by the core devs and the Foundation a few minutes ago. We will come back with more specific communications on what we did, why it was necessary in an urgent manner, and what's next. Core developers, validator operators, wallet and exchange operators, and others in the community have noticed significantly slower block times in the last 12+ hours due to a large backlog of Oracled Proof-of-Coverage. No action is required by Hotspot Manufacturers, Validator operators, Data users, or Hotspot owners. We expect to resolve after we clear the backlog. This could be due to a rate limiter that hasn't been implemented yet that needs to be addressed but in order to continue processing state channels, payments, burns, Hotspot onboards, location changes, etc., we believe the correct response is to halt the oracle that is pushing transactions on chain and let it recover. We will provide more details as we execute the steps defined in the blockchain channel on Discord. Stay tuned and thanks for being patient over the weekend. We'll come back with a proper post-mortem and further steps during the week. Thanks!
Latest Updates ( sorted recent to last )
RESOLVED almost 2 years ago - at 02/05/2023 08:38PM

Core developers, validators, and other members of the community are waiting for more data on oracle performance and load shedding efforts that have been made. Block production resumed but election times are slightly longer than expected. We will keep monitoring.

MONITORING almost 2 years ago - at 02/05/2023 06:34PM

After a few elections, block production and elections have resumed and Validator operators have reported that block times are more in line with expectations.

Proof-of-Coverage has resumed and the IoT Oracle has been brought back online with the version that we planned to deploy early this week. It has been in devnet testing for the last 12 hours and we feel that it resolves some of the issues that have been pending since the mainnet launch of Oracled PoC. We will acknowledge the fixed issues and address any pending bugs.

We will continue to monitor the chain for issues and look at increasing the transaction rate above 900 Proof-of-Coverage transactions per block.

Stay tuned and thanks for being patient over the weekend. Next update should be in the next two hours..

IDENTIFIED almost 2 years ago - at 02/05/2023 05:04PM

This is placeholder comms for a decision made by the core devs and the Foundation a few minutes ago. We will come back with more specific communications on what we did, why it was necessary in an urgent manner, and what's next.

Core developers, validator operators, wallet and exchange operators, and others in the community have noticed significantly slower block times in the last 12+ hours due to a large backlog of Oracled Proof-of-Coverage.

No action is required by Hotspot Manufacturers, Validator operators, Data users, or Hotspot owners. We expect to resolve after we clear the backlog.

This could be due to a rate limiter that hasn't been implemented yet that needs to be addressed but in order to continue processing state channels, payments, burns, Hotspot onboards, location changes, etc., we believe the correct response is to halt the oracle that is pushing transactions on chain and let it recover. We will provide more details as we execute the steps defined in the blockchain channel on Discord.

Stay tuned and thanks for being patient over the weekend. We'll come back with a proper post-mortem and further steps during the week.

Thanks!

Latest Helium outages

Solana Network Congestion - 8 months ago
Solana Network Congestion - 9 months ago
Solana Halt - 11 months ago

Start monitoring all your vendors in just 5 minutes

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3278 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime