Outage in Voyado

[ENGAGE] INFO: SMS degradation from earlier today

Resolved Minor
November 29, 2024 - Started 22 days ago - Lasted about 19 hours
Official incident page

Need to monitor Voyado outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Voyado, and never miss an outage again.
Start Free Trial

Outage Details

Earlier today, technical issues related to SMS send-outs affected our API connections. For 30 minutes, the Voyado system could not process membership commands, both in-store and for e-commerce. During this period, SMS delivery was slower than usual. The issue was resolved, and all services have been fully operational since 10:50. We closed the incident when the API's had been stable for some time and SMS was back at full capacity. We also advertised that we made sure that any delayed messages were to be corrected. Current Status: We still have a small portion of recipients that haven't received their SMS message (about 5%). We have decided not to send these messages to avoid the risk of sending duplicates. During the aftermath of the incident we discovered that a central part of our sending pipeline had been affected in a way we didn't anticipate which had led to some recipients receiving multiple copies of the messages intended for them. Our top priority is resolving the core issue to ensure no further duplicate messages will be sent. As a result of the previous disturbance with SMS messages, SMS statistics are currently not accurate and may still show some communications as sending. They will be changed to Sent or Stopped at a later point. We are taking all possible measures to prevent this from happening again. The problem was caused by a part of our system connected to SMS send-outs that did not auto-scale as it should have, resulting in a significant backlog that couldn’t be processed quickly enough. Over time, this issue began affecting other parts of the system. We identified the component causing the error, and once resolved, the rest of the system issues were cleared. We are now closely monitoring this to ensure stability moving forward. We apologize to all customers affected by these technical issues. We are fully staffed with on-call teams to ensure the platform operates smoothly, and errors are handled and resolved as quickly as possible. For further questions or concerns, please get in touch with our Support team.
Components affected
Voyado Messaging
Latest Updates ( sorted recent to last )
RESOLVED 22 days ago - at 11/30/2024 10:19AM

This incident has been resolved.

IDENTIFIED 22 days ago - at 11/29/2024 03:11PM

Earlier today, technical issues related to SMS send-outs affected our API connections.
For 30 minutes, the Voyado system could not process membership commands, both in-store and for e-commerce. During this period, SMS delivery was slower than usual. The issue was resolved, and all services have been fully operational since 10:50.

We closed the incident when the API's had been stable for some time and SMS was back at full capacity.
We also advertised that we made sure that any delayed messages were to be corrected.

Current Status:
We still have a small portion of recipients that haven't received their SMS message (about 5%). We have decided not to send these messages to avoid the risk of sending duplicates. During the aftermath of the incident we discovered that a central part of our sending pipeline had been affected in a way we didn't anticipate which had led to some recipients receiving multiple copies of the messages intended for them. Our top priority is resolving the core issue to ensure no further duplicate messages will be sent.

As a result of the previous disturbance with SMS messages, SMS statistics are currently not accurate and may still show some communications as sending. They will be changed to Sent or Stopped at a later point.

We are taking all possible measures to prevent this from happening again.
The problem was caused by a part of our system connected to SMS send-outs that did not auto-scale as it should have, resulting in a significant backlog that couldn’t be processed quickly enough. Over time, this issue began affecting other parts of the system.

We identified the component causing the error, and once resolved, the rest of the system issues were cleared. We are now closely monitoring this to ensure stability moving forward.

We apologize to all customers affected by these technical issues. We are fully staffed with on-call teams to ensure the platform operates smoothly, and errors are handled and resolved as quickly as possible.


For further questions or concerns, please get in touch with our Support team.

Be the first to know when Voyado and other third-party services go down

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3278 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime