Connect Incident - Delayed processing of Device Vulnerabilities

This incident has been resolved.

This afternoon our engineering team deployed a series of fixes that reduced unnecessary database load from vulnerability processing, which has allowed us to more than double our processing throughput. The vulnerability queue is now draining rapidly, and we expect vulnerability data to be fully caught up within the next several hours.

Our engineering team continues to actively work on this issue. We've deployed several optimizations today and have additional fixes under review that we expect to further reduce processing time.

Following an exceptionally large Microsoft Patch Tuesday (~570 fixes, nearly triple last month), Connect is processing an unprecedented volume of vulnerability changes. Vulnerability results in PDQ Connect are currently delayed by up to ~3 hours. Newly detected vulnerabilities may appear late, patched ones may take longer to show as resolved, and vulnerability counts/groups may be briefly out of date. No data is lost and no action is needed.

Deployments, device inventory & scans, and application scans are operating normally.

Due to patch Tuesday, Vulnerabilities backlog is still catching up in workload and will likely take a while to catch up. More updates to come.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

A fix has been implemented and things are trending back to looking normal.

The issue has been identified and we're working on a fix. Thank you for your patience.

The issue has been identified and a fix is being implemented.

Connect is currently experiencing long delays in updating device group counts and Connect application data. We are currently investigating the issue. Thank you for your patience.