Grafana experienced a major incident where permissions issues with IRM prevented users from receiving pages for 5.4 hours. The root cause was identified as an RBAC issue stemming from a regression in recent code changes. The incident was resolved through a hot fix and code deployment that was rolled out across the fleet to restore access for all affected customers.
Trusted by 1,000+ teams
Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.
This incident has been resolved.
Continuing to monitor progress. Most customers affected should have all services restored, with a few remaining customers receiving updates as the rollout finishes out. Thanks again for your patience.
A fix has been released to prod and rolling out across the fleet for IRM, restoring access to affected customers. Thanks for your patience through this work. We're continuing to monitor to confirm we've returned to a steady state.
We've identified an earlier regression in one of our recent code changes that was affecting resolution of our previous fix. We're deploying this change now and applying a hot fix in the interim to restore access quickly
A fix is being deployed now, and we are monitoring the progress.
We've identified an issue with RBAC, and are working on a fix to restore permission services for those affected.
Our engineering team is still investigating this issue. We do not have any new information to share at this time, but will continue to provide timely updates.
We are currently investigating an issue impacting permissions for IRM. As a result, users are not currently getting paged. We will provide updates as they become available.
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 6320 services available
Integrations with