Trusted by 1,000+ teams
Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.
Processing for rrc00 caught up on 10 November around 22:00 UTC. This incident has been resolved.
Title updated, was: Delayed RIS updates for rrc00/rrc03, delayed bviews for rrc00/rrc03/rrc24
The pipelines are recovering. At the moment only bviews for rrc00 are delayed.
We will deploy a software change that improves behavior when the pipeline needs to recover this week. We also increased the priority of a infrastructure change that speeds up the recovery of the pipelines.
We realised update files for rrc00/rrc03 are also affected. We were aware of an earlier issue but assumed it recovered. However, in practice the issue was masked because of alerts not sending the message to chat outside office hours.
Alerts are routed to mobile phones. Some alerts are office hours only. When writing the initial update this morning we did not realise that an office hours only alert was also masked from the chat channel that contains paging alerts.
We are continuing to work on this issue.
The next bviews (08:00 UTC) for rrc00, rrc03, and rrc24 will be delayed by multiple hours due to a high volume of traffic from some peers.
We intervened by temporarily de-peering some peers yesterday. However, we are still processing a backlog of traffic.
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 6320 services available
Integrations with