Search API Down

We have the incident fully resolved, all systems are stabilized.

Systems are back to normal. Team are doing clean ups, expected to mark the incident to "Resolved" by the end of today EST.

SMC and reporting panel is fully operational. Search, Indexing, and autocomplete fully back to normal state.
We will be monitoring in the next 24 hours before we close this incident.

Our Engineering team has brought Indexing Service back.
We are still seeing reporting panel fail to render, we are working with our backend team and vendor to bring it back.
We will be updating the individual service status in the next couple hours as services stabilize.

We are seeing Search/Autocomplete fully recovered.
SMC is mostly recovered, experiencing intermittent issue with reporting (not being able to render reports properly).
Our Engineering team is still actively working on restoring our indexing services.

Reporting in SMC is currently experiencing issues due to the same ongoing AWS outage. We are working on bringing it back.

Management Console is back to Operational

Our engineering team has taken Management Console out of maintenance mode. We are still working on bringing indexing service back.

Search API/Autocomplete API are fully recovered.
Engineering team is still working on fixes to bring Indexing service back.

Search and Autocomplete is being recovered. Team is working on a fix to bring Indexing and Management back.

We continue to observe some instability on AWS side as well as our vendor who's hosted on AWS. Team is still working on restoring the services, while monitoring the restoring effort of AWS.

We are monitoring the results for Search, Autocomplete, while waiting for AWS fix to propagate

Search API is now back to normal level - we are monitoring

Part of the services are brought back to normal level. We are still waiting on another AWS-dependent vendor to come back to normal before we take Management Console back)

Updating impact: we have confirmed the impact is confined to the same region (us-east) as AWS has identified. Traffic on us-west and other regions are still fine.

In order to protect our system and make sure we get services back as quickly as possible once AWS outage is recovered, we have to put our Management Console into maintenance mode. Users are blocked from making any changes as of now. Thanks for your understanding during this difficult time!

We have isolated the problem to be fully related to AWS outages. Indexing, Search functionality and also management console are impacted.

We have identified an incident that our Search API is currently down, post AWS outage. The team is currently investigating, and identified a potential problem.