[US region] Ingest API - Performance Degradation

This incident has been resolved.

Duration:
~35 minutes of processing delay. Data Ingestion delay between June 1 - 7:35 AM to June 1 - 10:35 AM PST.

Affected Customers:
All customers ingesting data to AWS US region during the incident window

Impact:
Delayed data availability in Plazma (up to 40 mins)
No data loss occurred

Latest Update:
Our team is still actively monitoring and assessing the processing of the data.

We will leave this status page in a Monitoring state until we are certain everything has been processed.

Thank you for your continued patience.

Latest Update:
Our internal graphs show that the overwhelming majority of the data has been processed. There is small amount of residual data that is taking longer than expected to process.

We will leave this status page in a Monitoring state until we are certain everything has been processed.

Thank you for your continued patience.

Latest Update:
Our team is still actively monitoring and assessing the processing of the data backlog.

Thank you for your continued patience.

Latest Update:
Our team has deployed a fix, and the system is now processing the backlog of data at a controlled rate.

Estimated Time to Full Recovery:
~7 hours

We’re actively monitoring the recovery and will provide further updates as progress continues. Thank you for your patience.

Current Status:
A fix has been applied and is currently under observation.

Impact:
An issue was identified with data ingestion(delayed ingestion) between Sunday 8:00 AM PST - 8:30 AM PST.

Remediation:
A fix has been implemented, and new incoming data is now processing normally.

Next Steps:
We are actively working to resume sending the affected data, which will arrive out of order along with new incoming data.

Updates to Follow:
Further details will be provided as the situation progresses.

Our Ingest API is experiencing a performance issue. We are investigating the cause.