Multiple locations - Multiple Channels processing degradation

All services at our Great Britain data center have been fully restored and with that, the impact of this outage to Infobip services across all our data centers has been successfully mitigated.
During the specified timeframe, customers attempting to reach services (Customer Portal, HTTP API, SMPP, SIP trunk, SMTP) for message processing hosted at any of our data centers globally could have experienced rejected requests, delayed traffic processing, and unexpected behavior from the Customer Portal and its modules (applications). Responses from Infobip Support (support@infobip.com) could have been answered with a delay.

The root cause of the outage happened in CrowdStrike’s defect found in a single content update for Windows hosts which affected our services supplied with the faulty version, however, it is important to note that this is not a security incident.

Start (UTC): 19 July 04:58
End (UTC): 19 July 16:00
Impact: Rejected/delayed traffic processing, unexpected errors from the Customer Portal/APIs

The service has been fully restored on Hong Kong data center.

We have restored most of our services and data centers. Our teams are finalizing the process of retrieving all data centers to functional state.

The service has been fully restored on the United Arab Emirates data center.

The service has been fully restored on Ukraine, Colombia, Sri Lanka and Bangladesh data centers.

We are continuing to work on a fix for this issue.

The service has been fully restored on Malaysia, EU Region-locked (Germany), South Africa and Vietnam data centers. The team is investing efforts to bring back the services to the rest of data centers.

The services have been fully restored on Indonesia, India, Pakistan, Singapore, Brazil, Turkey and Saudi Arabia data centers. The team is working on bringing back the services to the rest of data centers.

We are continuing to work on restoring the services on other data centers. We have all our hands on deck and this is currently top priority for us.
The issue was caused by CrowdStrike’s defect found in a single content update for Windows hosts.
It is important to note that this is not a security incident nor any kind of cyberattack.

The services on Global (Germany) and United States data centers should be fully restored now. The team is focusing on restoring the services on other data centers as well. Updates are incoming soon.

Our teams continue working on restoring the traffic and services. More updates will follow.

The team has partially restored traffic on Global (Germany) and United States data centers and it should slowly recover. Other data centers are still in progress. Updates will follow.

Preliminary investigation results are indicating the issue is connected with the global incident of our vendor. Infobip services should still misbehave due to this incident. Also, response delays from our Customer Support (support@infobip.com) department are still expected.

Joint task force is continuously trying to restore all Infobip services. We will keep sharing the updates.

Latest findings show that you might experience issues with submitting traffic toward our platform, as well as errors and delays in processing the already submitted traffic across all channels.
Customer Portal and its modules might misbehave and present unexpected errors on attempts to utilize the services.
Delays are expected in responses from our Customer Support (support@infobip.com) department as well.

Our teams are working on pinpointing and mitigating the issue with the highest priority and more updates will follow.

We are currently observing issues with several of our services. The scope of the issue is currently under investigation and more information will follow in the update.