Outage in Cognite

Problems with timeseries and sequences in EUR-N1

Resolved Major
March 24, 2023 - Started over 2 years ago - Lasted 4 days
Official incident page

Need to monitor Cognite outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Cognite, and never miss an outage again.
Start Free Trial

Outage Details

We are currently investigating the issue.
Components affected
Cognite Data Fusion API
Latest Updates ( sorted recent to last )
RESOLVED over 2 years ago - at 03/28/2023 12:37PM

A regression in datastore configuration during the addition of additional checksum/error detection resulted in a temporary loss of resiliency.
The regression has been identified and forward-fixed to retain additional error detection.

MONITORING over 2 years ago - at 03/27/2023 02:12PM

The API is responding normally, and we are continuing to investigate the root causes.

MONITORING over 2 years ago - at 03/25/2023 12:02PM

Data is fully replicated.
The API is responding normally, and we will continue to monitor through the weekend.

MONITORING over 2 years ago - at 03/24/2023 04:00PM

Restoration of full replication level is proceeding, and expected to complete with the next 24 hours.
The API is responding normally, and we continue to monitor.

MONITORING over 2 years ago - at 03/24/2023 12:42PM

We have identified a faulty storage node, and are migrating data away from it. Until migration is complete, data is below normal replication levels.
The API is functioning normally again.

INVESTIGATING over 2 years ago - at 03/24/2023 11:47AM

The cluster error rates have returned to normal levels, but we continue to investigate the source of the problem.

INVESTIGATING over 2 years ago - at 03/24/2023 11:11AM

Error rates are declining, but we are continuing to investigate.

INVESTIGATING over 2 years ago - at 03/24/2023 10:35AM

We are continuing to investigate the cause of the issue and recover alternatives.
We are working to make the most recent backup available as a fallback.

INVESTIGATING over 2 years ago - at 03/24/2023 10:09AM

The timeseries and sequences datastore replication process began experiencing errors at 9:07 UTC, and API error rates for timeseries and sequences increased significantly.

This will be visible in services and applications reading and writing to timeseries and sequences, including Fusion.

INVESTIGATING over 2 years ago - at 03/24/2023 09:54AM

We are continuing to investigate this issue.

INVESTIGATING over 2 years ago - at 03/24/2023 09:53AM

We are currently investigating the issue.

Be the First to Know When Vendors Go Down

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4484 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook