Partial outage of Sales Flow

This incident has been resolved.

On October 17, 2023, from 13:58 to 14:52 UTC, shoppers experienced errors when navigating in VTEX stores.

We have been monitoring for just over 50 minutes, and our services continue to operate normally. We will release a detailed report soon.

At 13:58 UTC, a system outage disrupted multiple platform services, substantially impacting the overall sales flow. Swiftly, the affected teams were alerted to the issue, and automated recovery procedures were set into motion.
By 14:15 UTC, we partially restored the sales flow, bringing back 80% of its normal operation.
At 14:32 UTC, a dedicated team identified a problem stemming from cascading failures and immediately took action to rectify the situation.
By 14:42 UTC, mitigation measures were successfully implemented, and by 14:52 UTC, all systems had fully recovered, with no discernible impact on the sales flow.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

The services have partially stabilized. Still investigating this issue.

We are currently investigating this issue.