Outage in Ably

Occasional anomalously-high `messages.inbound.rest.messages.refused` stat when a rest publish is refused

Resolved Minor
October 11, 2024 - Started about 1 year ago
Official incident page

Incident Report

Since Monday 7th, a REST publish that was refused by Ably's servers (for example, for being above the maximum message size) had a small probability of contributing an incorrect, anomalously high message inbound refused stat (key `messages.inbound.rest.messages.refused`) to app and account stats. Since refused messages do not count towards a customer's chargeable message quota, this bug did not result on anyone being charged more or having their quota used up. However, in a couple of cases it did momentarily trigger a customer's account-wide instantaneous message rate limit (`messages.maxRate`), for which we apologise. We are rolling out a fix for this currently.

Need to monitor Ably outages?

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Latest Updates ( sorted recent to last )
RESOLVED about 1 year ago - at 10/11/2024 12:20PM

The fix for this has been fully rolled-out.

IDENTIFIED about 1 year ago - at 10/11/2024 10:12AM

Since Monday 7th, a REST publish that was refused by Ably's servers (for example, for being above the maximum message size) had a small probability of contributing an incorrect, anomalously high message inbound refused stat (key `messages.inbound.rest.messages.refused`) to app and account stats.

Since refused messages do not count towards a customer's chargeable message quota, this bug did not result on anyone being charged more or having their quota used up. However, in a couple of cases it did momentarily trigger a customer's account-wide instantaneous message rate limit (`messages.maxRate`), for which we apologise.

We are rolling out a fix for this currently.

The Status Page Aggregator Built for IT Teams

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4522 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook