Malfunctioning scheduler

Summary: We experienced a partial outage of the scheduler and degraded performance of Apify API.

Timeframe: The issue started at a small scale from 11.00 UTC and then escalated around 15.00 UTC. By 16.15 UTC, the issue was resolved.

Impact:
- During the incident, some scheduled Actors did not start or were run twice.
- In response, we've aborted some of the runs that started after 15:00 UTC and turned off the Schedule component for approximately 45 minutes.
- If your runs were aborted, they can now be safely resurrected; see: https://docs.apify.com/platform/actors/running/runs-and-builds#resurrection-of-finished-run
- There was no data loss for runs in progress, but some runs were not triggered.
- API performance was degraded at various levels along the incident, and some runs started via API may have been in a READY state for longer than usual.

We're sorry for the inconvenience. We take this issue seriously and will implement measures to mitigate similar issues in the future.
We will provide a post-mortem summary here with more details later this week.

We have identified the root cause and deployed a fix. Both API and Actor components are now stable, and we’re slowly starting the scheduled runs.

We are continuing to investigate this issue.

We're experiencing issues with scheduling. Some Actor runs may not start in time. We are identifying potentially started duplicate Actor runs. The first problems started around 11:00 CET at a very small scale and escalated around 3:00 UTC.

The performance of API is partially degraded, and some endpoints might be answering slower than usual.

We're investigating the issue and will keep you informed about the progress. We are sorry for the inconvenience caused; this is a critical issue.

In response to the incident, we aborted some of the runs in the ready state. This affects runs that were started by the scheduler or another way from 2025-02-24 15:00 UTC. These runs can be resurrected if needed.