Hosting provider partial downtime

The system has been working properly since our last communication. We now consider this incident resolved.

We are working on a postmortem to publish in the coming days.

Push Campaigns are now working as expected. Any Push Campaign message that should have been sent before 20:10 UTC has been canceled and will not be sent. Pushes scheduled since then have been sent up to one hour late.

All our services are now working as expected.
We will continue to monitor the services for the next few hours.

Our hosting provider brought the servers back up around 19:40 UTC. Due to the unprecedented nature of the incident, the system did not self heal as expected and required manual intervention to come back online.

So far the Push Transactional API and Push Automations are now working as expected, but Push Campaigns remain unavailable. We’re working on this issue and expect a resolution in the next couple of hours.

As our hosting provider is taking more time than originally expected to fix the issue, we would like to make a full recap of the situation:

The platform is impacted in the following ways:
- Most push notifications (mobile & web) are impacted and have not been sent since 12:00 UTC. This is: Campaigns, Trigger Automations and Transactional. In some rare cases the pushes went through, but at this time we are not able to give any more precise information.


Other products are not impacted:
- Email, SMS work as expected
- Data Ingestion (APIs, SDKs, flat file imports) work as expected
- In-app automations are working as expected

During this time, please avoid creating new campaigns as we cannot assure you when they will be executed. Deleting a campaign that should already have started sending and recreating it might not work as expected and result in duplicate notifications when the situation comes back to normal.


Our hosting provider now expects the servers to be back online around 20:00 UTC.
When the system will be back to a healthy state, Batch will try to send the pending notifications. We will update the statuspage when this happens.

To help ensure that your users do not get late/duplicate/unwanted notifications, *any notification that have been pending (that is, should have been sent) more than an hour ago will be dropped*.

According to our provider, we expect the servers to be back online at the end of the afternoon (16H UTC).
Most of the push orchestrations are impacted.
Email & SMS orchestrations are not impacted.

We are continuing to work on a fix for this issue.

We are currently experiencing issues on certain servers, leading to delays and interruptions in the push notification services.