AWS US-EAST-1 outage

The AWS issue has been resolved and our monitoring confirms that there is no further impact to Panther customers.

As a reminder, If you have a deployment in the AWS US-EAST-1 region and have an HTTP log source, we recommend checking your log source(s) from Jul 30 at 3:00 PM until Jul 30 at 9:32 PM PDT to ensure that there are no missing logs.

If you have any questions, please contact Panther Support.

AWS has reported that their services have fully recovered, and we have confirmed that there are no longer any issues logging into the Panther Console and all HTTP log sources should be healthy. We will continue to monitor our systems to ensure that there is no further impact to Panther customers.

If you have a deployment in the AWS US-EAST-1 region and have an HTTP log source, we recommend checking your log source(s) from Jul 30 at 3:00 PM until Jul 30 at 9:32 PM PDT to ensure that there are no missing logs.

Due to the nature of the HTTP data transport, Panther is not able to keep a record of which logs failed to be received. Customers can identify this by searching through the local logs generated by the system sending logs to Panther via HTTP and finding any HTTP 500 errors returned when attempting to deliver logs to Panther. You can resend these logs to Panther and Panther will correctly backdate them in the Security data lake.

We will continue to monitor the situation and will provide updates if necessary.

Panther has confirmed that customers with deployments in the AWS US-EAST-1 region are experiencing issues logging into the Panther Console and ingesting logs via the HTTP ingest log source type. This is a result of the degradation of the AWS Kinesis service, and ~50 other AWS services that rely on Kinesis, in Panther's case in particular the API Gateway service.

Other Panther services such as ingestion via other mechanisms (S3 and other data transports as well as SaaS log source pulling), detections, alerting, data lake ingestion, and search are working as intended although we're continuing to monitor in case of further AWS service degradation.

The primary long term impact to Panther's end users is the loss of some data currently being sent to HTTP Ingest log sources. This data will not be recoverable by Panther. We encourage you to retain copies of that data if possible so you can resend it after service has resumed. Once AWS has confirmed service has resumed, we will notify impacted customers of the time frame during which data ingestion was impacted.

You can follow AWS updates here: https://health.aws.amazon.com/health/status

Panther has confirmed that customers with deployments in the AWS US-EAST-1 region may experience issues logging into the Panther Console, log processing and alert delivery delays, as well as other issues with the service due to the ongoing AWS outage that began at approximately Jul 30 3:40 PM PDT (https://health.aws.amazon.com/health/status). We will update this incident with more information as it becomes available.