Infrastructure provider issue

This incident has been resolved. All services have remained fully operational since the last update.

All remaining impacted services have returned to normal operations since the last update.

The Analytics Event Ingestion backlog has been processed, and the Real-Time Machine Learning features that depend on them are now all caught up.

We are continuing to monitor all services closely.

We are seeing continued delays in Analytics Event Ingestion. No events are lost, at the moment they are being queued for processing.

Real-time Machine Learning features, such as Personalisation as You Go, are also impacted by the delay in Analytics Ingestion, impacting the quality of the results.

The elevated error rates which impacted the Analytics Read API in the last few hours have been mitigated and that service is now back to normal operations.

We are continuing to see elevated error rates on the Analytics Read API as well as continued delays in Analytics Event Ingestion. No events are lost, at the moment they are being queued for processing.

Real-time Machine Learning features, such as Personalisation as You Go, are also impacted by the delay in Analytics Ingestion, impacting the quality of the results.

Capacity issues have improved in the US region, and Crawling is back to normal operations.

We are continuing to see elevated error rates on the Analytics Read API as well as continued delays in Analytics Event Ingestion. No further events are being lost, at the moment they are queued for processing.

Real-time Machine Learning features, such as Personalisation as You Go, are also impacted by the delay in Analytics Ingestion, impacting the quality of the results.

Capacity problems are also slowing down our crawling operations in the US region, introducing some delays for scheduled jobs. Your content for sources crawled by our infrastructure may not be up to date. Push API and Crawling Module sources remain up to date.

We are continuing to see elevated error rates on the Analytics Read API as well as continued delays in Analytics Event Ingestion. No further events are being lost, at the moment they are queued for processing.

Real-time Machine Learning features, such as Personalisation as You Go, are also impacted by the delay in Analytics Ingestion, impacting the quality of the results.

We are continuing to see elevated error rates on the Analytics Read API and are now seeing some delays in Analytics Event Ingestion. A few analytics events were lost earlier during the main incident, but now new events are being queued for processing.

Due to the ongoing issue with our infrastructure provider, we are observing a degradation of Analytics Read services. Users of this service are currently seeing elevated error rates.

The underlying issues have been resolved by our infrastructure provider. Coveo services have been working normally since the last update and a document ingestion backlog which was built up during the incident has now caught up to real time. We are continuing to monitor the services closely.

An issue with an infrastructure provider has led to a partial degradation of some Coveo services. Mitigations are already ongoing at the level of infrastructure provider and therefore we are already seeing improvements happening. We will continue to monitor the issue until this is fully resolved.