intermittently Orders are not filled

This incident has been resolved.

Operational cleanup of pending_cancel is still going on.

The current phase is operational cleanup and client impact mitigation: normal order flow is working, and the focus is on systematically clearing pending replace/pending cancel states (including crypto) while continuing elevated monitoring.

We are still doing cleanup of backlog of all pending cancel and pending replace orders.

Our messaging system is operating normally as of now. Our team is now focused on cleaning up the remaining orders that were impacted during the disruption. We appreciate your continued patience.

We are still working on the issue and monitoring system closely. We will write next update soon.

The fix for the primary affected queue has been successfully applied , messages are flowing and the backlog is cleared. During this process, two additional queues were found to have similar synchronisation issues. Our team is now applying the same fix to these remaining components to restore full stability.

In parallel, we are evaluating longer-term improvements to our messaging infrastructure to prevent recurrence.

We will provide an update once all components are fully restored.

The majority of services have been restored. Our team has identified one remaining synchronisation issue within the messaging infrastructure and is applying a targeted fix. As a precaution, affected services have been temporarily scaled down during the repair. A small number of orders that did not reach their intended venues are being re-processed. We will confirm resolution once this final fix is verified.

After restart of problematic component in messaging cluster The messaging system has been stabilised and key trading services are back online and operating normally.A final reconnection step for our crypto trading service is currently being completed to ensure all systems are fully restored.
We expect full recovery shortly and will confirm once everything is back to normal. Thank you for your patience.

The root cause of the instability has been traced to a specific node within our messaging cluster. A targeted restart of that component is being prepared and will be executed shortly.
In the meantime, the order backlog is actively being cleared , pending orders are decreasing and filled orders are increasing as manual re-routing continues.
The crypto exchange component restart has been completed successfully.
The next step to restore full system stability is the execution of this remaining component restart. We will confirm once it is complete.

We are currently experiencing intermittent delays in order processing due to a system connectivity issue. Our team is actively working to resolve this. Here's what's happening:

1.Orders that are stuck in "pending" status are being manually processed and re-routed.
2.For any orders pending cancellation, we are coordinating directly with our trading partners to ensure they are properly canceled.
3.Crypto order processing is being restored as part of our ongoing recovery efforts.

Our engineering team has identified the underlying cause and is working on a permanent fix. We will share another update once the issue is fully resolved. We apologize for the inconvenience and appreciate your patience.

While we were able to restart the main messaging infrastructure, we are currently managing some recurring connection instability and have identified a backlog of orders that have not yet been processed.
Our engineering team is fully engaged and focused on clearing this backlog and addressing the underlying causes of the intermittent system disruptions to ensure a complete and stable resolution. We will provide another update once the system has fully stabilised.

This is due to a crash in our main messaging system (RabbitMQ).
We have restarted the messaging system and are now restarting the associated programs to re-establish stable connections. We are actively monitoring the system to ensure it is back to normal as quickly as possible.

We are still looking into the issue ,will provide next update shortly.

We are continuing to investigate this issue.

We are currently seeing intermittent issues with orders being filled. Our team is actively working on the issue with full attention.