Outage in Datto

Datto RMM - Managed Devices and Managed Network Devices may alert offline despite being online; Device details and activity logs are slow to populate

Resolved Minor
February 07, 2023 - Started over 2 years ago - Lasted about 1 month
Official incident page

Need to monitor Datto outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Datto, and never miss an outage again.
Start Free Trial

Outage Details

Our teams are investigating the root cause, and applying interim infrastructure changes as a response, to current issues with  - devices losing their connection to the service, alerting offline, and marking their managed network devices offline if they are network nodes - device pages and activity log data taking a longer then desired time to load  For further details please visit the Datto Community: https://community.datto.com/t5/RMM/RMM-11-5-0-release-related-issues-and-remediation/m-p/101984#M16920  Thank you for your patience!
Latest Updates ( sorted recent to last )
INVESTIGATING about 2 years ago - at 02/28/2023 04:43PM

We are continuing to investigate this issue.

INVESTIGATING about 2 years ago - at 02/21/2023 02:16PM

We are continuing to investigate the root cause of the issue while monitoring the service to rapidly respond to any events that might occur.

INVESTIGATING over 2 years ago - at 02/16/2023 06:50PM

We are continuing to investigate the root cause of the issue while monitoring the service to rapidly respond to any events that might occur.

INVESTIGATING over 2 years ago - at 02/13/2023 04:36PM

Based on metrics and feedback we have seen a positive change in the rate of offline alerts since the aforementioned hotfix related to network device offline alert behavior.

We are continuing to investigate the root cause of agent based managed devices sporadically dropping offline.

INVESTIGATING over 2 years ago - at 02/10/2023 06:40PM

Update on our progress towards resolution:

Device page and activity log loading times/intermittent failure to load data:

Throughout the course of this week the engineering team has qualified and deployed several improvements to the activity log service and scaled up related infrastructure to restore healthy performance to device and unified activity log pages. Metrics and user feedback this week suggests that performance has significantly improved. We continue to work on additional items to address various causes we identified that may lead to slow performance during higher load.


Agent based and agentless device offline alerts:

On February 10, we deployed a change across all platforms. The platform no longer marks network devices offline automatically when their network node goes offline. This alleviates the problem by not generating alerts for these network devices unless they themselves report offline. What we have observed so far is that the number of raised alerts on our platforms are trending downwards, and user feedback has also been positive.
We continue to investigate the triggers and the root cause of agent based devices intermittently dropping offline in order to resolve the issue permanently.

We encourage all our partners to let us know if they see significant changes in behavior while this issue is ongoing by contacting Datto Technical Support.

INVESTIGATING over 2 years ago - at 02/09/2023 08:56PM

A hotfix has been deployed to all platforms to reduce the false-positive alerts for network devices while we continue to investigate the root cause.

INVESTIGATING over 2 years ago - at 02/08/2023 08:35PM

As we continue to investigate this issue we are preparing a change to reduce the false-positive alerts for network devices. This change, which is scheduled to be rolled out on February 9th, would make it so the monitored network devices are not generating an offline alert as a result of a Network Node being reported offline.

INVESTIGATING over 2 years ago - at 02/08/2023 10:59AM

We are continuing to investigate this issue.

INVESTIGATING over 2 years ago - at 02/07/2023 10:57AM

Our teams are investigating the root cause, and applying interim infrastructure changes as a response, to current issues with 
- devices losing their connection to the service, alerting offline, and marking their managed network devices offline if they are network nodes
- device pages and activity log data taking a longer then desired time to load 

For further details please visit the Datto Community:
https://community.datto.com/t5/RMM/RMM-11-5-0-release-related-issues-and-remediation/m-p/101984#M16920 

Thank you for your patience!

All Your Service Status Pages in One Dashboard

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4200 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook