Major WMS service disruption

Full availability of the systems have been restored.

At 13:14 CEST today 25 June, a severe failure in a firewall handling traffic to Ongoing WMS caused existing connections to fail and new connections could not be established. Due to how the traffic is routed internally, only URLs like [systemname].ongoingsystems.se was affected. This is used by warehouse workers and admins when working with the WMS. External integrations use api.ongoingsystems.se and is routed differently, which for example means automation equipment was unaffected. Scheduled jobs, such as an integration in Ongoing running a Shopify stock sync completed without errors during the incident.

Engineers were immediately alerted of the failure and at 13:20 the traffic to a secondary hosting site was initiated via a DNS change. Due to delays in DNS propagation, troubleshooting on the primary site continued. The failing component was identified and was rebooted, which successfully stabilized the primary site. By 13:35, full service was restored for all users.

For now, the root cause of the failure is not fully understood. The failure coincided with routine diagnostic activities, though causation has not been established. We have redundant firewalls at this level but the automatic failover was not initiated resulting in delayed resolution of the issue. We are working with the firewall vendor to investigate further and prevent recurrence. We apologize for the disruption and appreciate your understanding.

We are continuing to monitor for any further issues. More information about the incident will follow.

A fix has been implemented on the primary site and we are continuously monitoring the availability of the system. Traffic can now be routed to either site and connections should be successful.

Traffic has been moved to a secondary site but there is delay before this takes effect. We are fixing the issue on the primary site in parallel.

Connections to the WMS gives error messages like "Secure Connection Failed". We are looking into it.