Outage in Netdata

Slow and failing Agent chart data responses

Resolved Minor
June 29, 2022 - Started almost 2 years ago - Lasted about 23 hours
Official incident page

Need to monitor Netdata outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Netdata, and never miss an outage again.
Start Free Trial

Outage Details

Users with nightly versions of the Netdata Agent are experiencing slow responses between Cloud and Agent, resulting in failing or slow charts in their Cloud dashboards. We are investigating the issue.
Components affected
Netdata Agent-Cloud Link (ACLK)
Latest Updates ( sorted recent to last )
MONITORING almost 2 years ago - at 06/30/2022 08:03AM

For completeness, the affected versions are v1.35.0-84-nightly and v1.35.0-96-nightly. Latest, corrected version is v1.35.0-104-nightly.

MONITORING almost 2 years ago - at 06/30/2022 07:20AM

The new nightly version of the Netdata Agent has been published and installed by a large portion of the agents that auto-update. We are monitoring the results.

IDENTIFIED almost 2 years ago - at 06/29/2022 02:11PM

We have identified part of the cause of failing responses for alarm values. In yesterday's nightly build of the Agent, we enabled the use of the newer MQTT5 library by default. We will create another build to revert that. In the meanwhile, you can explicitly disable this library using the mqtt5 setting in your configuration as described here: https://github.com/netdata/cloud-backend/issues/178.

Additionally the other latencies appear to be another instance of a known issue that causes responses with a small payload to be delayed. We are working on resolving this issue.

INVESTIGATING almost 2 years ago - at 06/29/2022 08:54AM

Users with nightly versions of the Netdata Agent are experiencing slow responses between Cloud and Agent, resulting in failing or slow charts in their Cloud dashboards. We are investigating the issue.

Start monitoring Netdata and all your cloud vendors in minutes

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3153 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime