Degraded performance and issues routing through Distro

Distro is back to an operational state. The issue appears to have been caused by an influx of triggered events which exceeded the maximum load possible. This caused the event processing time to grow exponentially for all customers.

We are correcting for this in several ways including improving current services to accommodate the increased load, and also improving the strategy for how events are processed.

While some of these improvements are ongoing, the incident is considered resolved.

Our team applied improvements to our Distro service, and we’re seeing better results. Still, records that should’ve been routed via Distro while the service had degraded performance may take some time to be processed.

It was also identified that this affected only one of our environments, so we don’t expect all Distro users to be affected by this.

We will keep monitoring the results and will soon provide more updates and a root cause. If you have any questions, please email support@chilipiper.com

We have been alerted to potential issues routing records through Distro. In most cases these do recover and route after a delay, but in many instances the routes don't appear to be processed at all. As such, we are treating this as a critical incident within the Distro service.

This is not impacting Concierge, Concierge routing, or the booking / scheduling platform.

Our team will continue to monitor this and work with our engineers to get to the root cause as soon as possible.