Alert ingestion via assemblers delayed

The incident has been resolved, and all previously-available assemblers are once again fully available.

We have implemented a fix and believe the incident to be resolved. We are monitoring the health of the infrastructure as individual assemblers come back online, and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems. We will provide another update by 4:30pm EDT.

We have identified a further cause and are working on a remediation. Individual assemblers are coming back online. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 12:30 EDT.

We have identified a further cause and are working on a remediation. Individual assemblers are coming back online. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 11:30 EDT.

We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 10:30 EDT.

We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 9:30 EDT.

We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 8:30 EDT.

We are continuing to work on a fix for this issue.

We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 8:00 EDT.

We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 7:30 EDT.

We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 7:00 EDT.

At approximately 4am EDT, we began experiencing a problem with our assembler VPN system that is preventing alerts from being ingested through assemblers. Alert ingestion is delayed for some devices behind an assembler. We will provide an update by 6:30am EDT.