Degraded performance across US cluster

Thank you for your patience while we worked on this issue. We encountered an incident where tasks for all entities—Models, Pipeline Ingestion tasks, and Destination Load tasks—were not executing as expected. This incident has come up due to a recent outage on the US cluster.

We have now resolved the issue. As a result, you may have noticed delays in ingestion/loading tasks or missed model run triggers. Moving forward, all scheduled tasks will run as expected and will be automatically recovered in the next schedule.

Our development team has identified and implemented a fix for this issue. We truly appreciate your patience and understanding while we worked on resolving this. We sincerely apologise for any inconvenience this may have caused.

Please note that if there are any pipelines/models where the schedule interval is high (about 3 hours or more) and you need the data on an urgent basis, please trigger these entities manually to make sure they run as expected.

The US cluster has been moved out of read-only mode, and we are monitoring the changes.

The issue has been identified, and a fix is currently being implemented. During this process, the US cluster will remain in read-only mode for a period of time.

We are currently investigating the issue.