Query failures in Dremio Cloud (US & EU)

Incident Summary
We experienced a service disruption starting at 1:15 PM PST that resulted in query failures when using reflections in multiple regions. We sincerely apologize for the impact this had on your workflows and the inconvenience it may have caused.

Incident Details
An update to the permissions used by Dremio internal sources inadvertently restricted access to some of the folders in the project store, resulting in queries using reflections or accessing those folders to fail in some AWS projects under certain conditions. Our team began investigating the issue immediately and was able to identify the issue and implement a fix at 5 PM PST. Concurrently, we encountered cloud provider issues in our Azure deployment infrastructure which impacted our ability to roll out the fix to our production environments. Once we were able to resolve issue, we were able to validate and deploy a fix to fully restore the service at 10:35 PM PST.

Our engineers are monitoring the system to ensure full stability. In addition, we are also implementing new alerts to detect such situation quickly in the future.

We have identified the issue impacting queries in Dremio Cloud (US & EU) and are in the process of deploying a fix to address the issue. We will provide an update within the next 45 minutes.

We have narrowed down the issue impacting queries in Dremio Cloud (US & EU). Engineering is in the process of building and testing a fix to address the issue. We will provide an update within the next 45 minutes.

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

We are continuing to investigate the issue with queries in Dremio Cloud (US & EU). Engineering has narrowed down the issue and are working on multiple possible mitigations. We will provide an update within the next 45 minutes.

We are continuing to investigate issues with queries in Dremio Cloud (US & EU). We have engaged multiple engineering teams to investigate and narrow down the issue. We will provide an update within the next 30 minutes.

We are continuing to investigate issues with queries in Dremio Cloud (US & EU). We have engaged multiple engineering teams to investigate the issue and will provide an update within the next 30 minutes.

We are investigating issues with queries in Dremio Cloud (US & EU). We will provide an update within the next 30 minutes.