Shopper experience impacted on both navigation and checkout

The VTEX seller register system is now operational again, starting from 08:19 PM UTC, and the sales behavior on the platform has returned to the expected level as of approximately 08:56 PM UTC.

To be the trusted partner to your success, we are diligently working on follow-up actions to ensure that this incident does not recur and to enhance our ability to detect and recover from future incidents more swiftly. We are fully committed to improving our systems to provide a reliable and trusted experience for you.

Here is some important information pertaining to this incident:

At 07:44 PM UTC, we identified an issue in one of our database systems utilized by the seller register. As a result, a subset of products belonging to some of our partner merchants were erroneously indexed with empty stock. Consequently, a portion of shoppers may have encountered difficulties while searching for products and attempting to add them to their shopping carts.

To prevent further inappropriate indexing, we promptly paused the indexing process at 08:17 PM UTC.

By 08:19 PM UTC, the mitigation process for our database systems was completed, resulting in the restoration of the expected behavior of the seller register system. This action initiated the recovery of sales behavior on the platform.

At 08:30 PM UTC, the indexer was restarted, and we began reprocessing the previously affected products. By 09:30 PM UTC, there were no pending products to be indexed in our queue.

VTEX continues to monitor the system for any potential anomalies related to this incident. If you come across a product exhibiting inventory inconsistencies in the search results, please refer to the following article for instructions on how to reindex it: https://help.vtex.com/tutorial/understanding-how-indexation-works--tutorials_256

As part of our follow-up measures, we are committed to enhancing our automation and change management processes to expedite the detection and mitigation of any regression in our seller register system.

We can confirm that after some mitigation maneuvers the anomalous behavior of the platform must have been normalized. We are monitoring the systems on which we perform these actions.

We have identified an issue in the Seller Register database that might be causing products in multiple stores to be displayed as out of stock. Our team continues to investigate and apply mitigation actions.

We are currently investigating this issue.