Elevated Errors in Sales Flow

Between 04:56 and 05:12 UTC, our shoppers and merchants experienced a major outage in the platform caused by CPU saturation in the License Manager module.

At 13:56 BRT, the License Manager module observed an abnormal increase in CPU usage. This exceeded the normal levels, indicating a potential issue. Subsequently, at 13:58 BRT, our alarms were triggered, and the platform experienced a major outage, affecting both the Sales flow and the Administrative environment.

To address the situation, we promptly took action. At 14:03 BRT, we increased the minimum nodes of the License Manager module. This adjustment aimed to alleviate the strain on the system and restore stability. By 14:07 BRT, the CPU usage of the License Manager had returned to the normal threshold.

The recovery process of the platform's nominal metrics started at 14:10 BRT. Finally, at 14:12 BRT, the platform had fully recovered, and we ensured ongoing monitoring of the License Manager and its dependencies to prevent further incidents.

We are continuing to monitor our platform for any further issues. The user experience in the platform is back to nominal.

VTEX Sales flow and Administrative environment are back to normal.

We have experienced major degradation in our License Manager module that impacted the sales flow and administrative environment. Our team quickly identified and fixed the issue. We are now monitoring the environment.

We are now recovering from the increased error rates. We continue to work towards full recovery.

We are investigating increased error rates in our platform.