Increased error rates for Master Data queries

The fix for the residual side-effects from Master Data indexing errors has been implemented.

Merchants should no longer be experiencing issues with Content Management workflows in Headless CMS. Shoppers should no longer be experiencing issues navigating in FastStore storefronts.

We are now in the process of removing the temporary interruption Headless CMS builds.

Our incident response team is monitoring to guarantee that normal platform behavior is fully reestablished.

We will send an additional update in the next 30 minutes, or as soon as we have more information to share.

Our team has identified that there are still residual side-effects from Master Data indexing errors impacting Headless CMS builds for FastStore storefronts. This may cause unsucessful builds to go to production, impacting navigation.

We have temporarily interrupted Headless CMS builds until we are able to understand and mitigate these residual side-effects.

We are currently investigating an issue in Headless CMS.

Affected Accounts may be experiencing Storefront errors.

Our incident response team is working on a mitigation strategy and investigating the root cause of the issue.

We will send an additional update in the next 30 minutes, or as soon as we have more information to share.

This incident has been resolved. We will provide further information in a public incident report.

Summary

On Sep 09, 2024, from 09:48 UTC to 13:34 UTC, several stores experienced indexing errors in Master Data, leading to missing data in search operations. Additional side-effects were identified in Headless CMS publishing/unpublishing processes due to its use of Master Data as a dependency.

Our global sales flow was unaffected during this incident, but stores relying on Master Data for critical operations were significantly impacted.

We apologize for any inconvenience this may have caused.

Timeline

At 09:48 UTC, indexing errors in Master Data started gradually increase.

At 09:54 UTC, our monitoring systems alerted our on-call engineers.

At 10:53 UTC, the issue was escalated to our incident response team.

At 11:37 UTC, our team started working on a recovery script to reconfigure indices for affected accounts as a mitigation strategy.

At 12:26 UTC, our team started mitigating impact on affected accounts, prioritizing those with critical impact on their operations.

At 13:34 UTC, the incident was fully mitigated.

The fix for the issue in Master Data has been implemented.

Stores should no longer be experiencing indexing errors or issues with data retrieval from Master Data.

Our incident response team is monitoring to guarantee that normal platform behavior is fully reestablished.

We will send an additional update in the next 30 minutes, or as soon as we have more information to share.

Most accounts should already be recovered, we are now closing in on the final accounts that still need to be restored. We estimate that this will be completed within the next 10 minutes.

We will send an additional update in the next 30 minutes, or as soon as we have more information to share.

Our incident response has started implementing the fix for affected accounts, which should gradually recover normal behavior in the next hour.

We have also identified additional impact in Headless CMS, since Master Data is a dependency. Publishing and unpublishing content will present degraded performance as long as an account presents an unhealthy index.

We will send an additional update in the next 30 minutes, or as soon as we have more information to share.

We have identified the contributing factors of the issue in Master Data.

Our incident response team estimates that the fix will be implemented in the next 30 minutes.

We will send an additional update in the next 30 minutes, or as soon as we have more information to share.

We are currently investigating an issue in Master Data
Query requests to the module may be experiencing indexing errors, returning less data than expected.
Our incident response team is working to identify the root cause and implement a solution.
We will send an additional update in the next 30 minutes, or as soon as we have more information to share.