Degradation in Recurring Tests and Issue Data freshness.

No further issues were observed throughout our monitoring period.

The Snyk Incident Response has now been stood down.

Our Engineers have now resolved the delays affecting the Issue Data retrieved by Reporting and Issues API.

Customers will need to initiate a manual retest to receive real-time data or wait for the Recurring Tests to complete.

All Recurring Tests (daily & weekly) have been re-enabled and will execute during their next scheduled run.

Weekly Recurring Tests that missed their scheduled run on the 17th of June will be manually triggered by us today.

We will continue to monitor the situation throughout today before standing down the incident.

Current status and actions being taken:
All Recurring Tests have been paused, which has helped decrease processing lag. However, our systems are still working through the backlog.

The Recurring Tests (both daily and weekly) scheduled to run yesterday, the 17th of June, were not executed.

Recurring Tests scheduled for today have been paused. We will share further details on this throughout today.

Current impact to users:
Issues data retrieved by Reports and Issues API is currently delayed by approximately 31 hours.

Once the data has normalised, we will provide an update on which Recurring Tests will be re-run.

Known workarounds:
None

We will provide further updates as they become available.

Current status and actions being taken:
All Recurring Tests were paused, which successfully helped reduce the processing lag, which our systems are still processing.

Earlier, we stated that today's Recurring Tests (daily and weekly) would run according to their scheduled times. However, there is likely to be a delay in initiating these to help reduce the delays.

Current impact to users:
Issues data retrieved by Reports and Issues API is currently delayed by approximately 10 hours.

Once the data has normalised, we will run all missed and delayed Recurring Tests.

Known workarounds:
None

We will provide further updates as they become available.

Current status and actions being taken:
All Recurring Tests were paused, which successfully helped reduce the processing lag, which our systems are still processing. All Recurring Tests (daily and weekly) are due to run on their next daily schedule.

Current impact to users:
Issues data retrieved by Reports and Issues API is currently delayed by approximately 9 hours.

The Weekly Recurring Tests that failed to initiate yesterday will be re-run once the lag has cleared.

Known workarounds:
None

We will provide further updates as they become available.

Our investigation has determined the cause of the delays, and we are taking measures to resolve it.

Current status and actions being taken:
The processing lag for Issues and Recurring Tests has increased.
To relieve the backlog and mitigate this we are stopping further Recurring Tests for the current schedule, and are starting to see the processing lag reduce.

Current impact to users:
Issues data retrieved by Reports and Issues API is delayed by about 5 hours at this point.
Recurring tests that have not processed already this schedule will not be run.
--Daily Recurring Tests will run on their next schedule, while skipped weekly Recurring Tests will be re-run once the delays have been resolved.

Known workarounds: None

We will provide further updates as they become available.

Snyk Engineering have detected an issue causing the following:
* Delays in updates to Issue Data; affecting Reporting and Issues Api.
* Slow Recurring Tests with the potential that some recurring tests may not complete on the current schedule.

This is only affecting customers on the primary US tenant, app.snyk.io

We will be back with another update as soon as we know more