Degraded performance for sequences and time series on EUR-C1

The engineering team has continued their monitoring during the day, and have concluded that the situation has stabilized and that the service performance and quality is at the desired level. Cognite has closed this incident.

Cognite Engineering has been monitoring the services since yesterday. The latency and error rates seem to have been stable since yesterday UTC 17:30. The load on the system is at a level where we think there is little risk for yesterday's problems to repeat. We are continuing to monitor and analyze the root cause.

The engineering team has spent additional hours working on a resolution for the incident. This incident is likely due to a change in traffic or traffic pattern. Some of the traffic has been turned off now, and the service looks healthier. We will continue to monitor, but no action will be taken until tomorrow if this remains stable. The incident is not considered resolved, but with the current traffic pattern and load, we believe that the service quality and performance is at an acceptable level.

Cognite Engineering has deployed configuration changes that seem to reduce the problem significantly. The engineering team is still working on additional improvements. End users should see reduced error rates and fewer requests with slow response times.

The end-user experience is both elevated response times, and increased error rates. The incident is re-classified as "partial outage". Cognite is still working on the investigation.

Cognite Engineering is investigating an issue where the performance of sequences and time series is reduced. We see P90 latency of up to 1 minute for parts of the services.