ES-1059099

We are investigating an issue with one of the Databricks services.

Incident Details:
- Workspace authentication requests may fail or timeout.
- Cluster start/resize/termination requests may fail or time out.
- Jobs relying on cluster start/resize/termination may not execute.
- Jobs submitted through APIs/Schedulers may not execute.
- UI and Databricks SQL queries may time out.
- Users may experience failures launching Databricks Serverless SQL Warehouses.
- Users may not be able to access UC APIs.

Incident Start Time: 18:36 UTC February 28 2024

We will provide an update in the next hour, or as soon as the issue has been identified.

We have identified the problem with the Databricks service. Our team is working on a mitigation.

Incident Details:
- Workspace authentication requests may fail or timeout.
- Cluster start/resize/termination requests may fail or time out.
- Jobs relying on cluster start/resize/termination may not execute.
- Jobs submitted through APIs/Schedulers may not execute.
- UI and Databricks SQL queries may time out.
- Users may experience failures launching Databricks Serverless SQL Warehouses.
- Users may not be able to access UC APIs.

Incident Start Time: 18:36 UTC February 28 2024

We will provide an update in the next hour, or as soon as the issue has been mitigated.

We are seeing a sign of recovery. Our team is actively monitoring the system to ensure the full mitigation. Rest assured, we are diligently working to maintain this positive trajectory. Thank you for your patience.

Incident Details:
- Workspace authentication requests may fail or timeout.
- Cluster start/resize/termination requests may fail or time out.
- Jobs relying on cluster start/resize/termination may not execute.
- Jobs submitted through APIs/Schedulers may not execute.
- UI and Databricks SQL queries may time out.
- Users may experience failures launching Databricks Serverless SQL Warehouses.
- Users may not be able to access UC APIs.

Incident Start Time: 18:36 UTC February 28 2024

We will provide an update in the next hour, or as soon as the issue has been mitigated.

Mitigation has been applied, and The issue has been successfully mitigated, although you may notice some latency. It’s important to note that this latency does not impact production services. Our team continues to monitor the situation closely to ensure optimal performance.

Incident Details:
- Workspace authentication requests may fail or timeout.
- Cluster start/resize/termination requests may fail or time out.
- Jobs relying on cluster start/resize/termination may not execute.
- Jobs submitted through APIs/Schedulers may not execute.
- UI and Databricks SQL queries may time out.
- Users may experience failures launching Databricks Serverless SQL Warehouses.
- Users may not be able to access UC APIs.

Incident Start Time: 18:36 UTC February 28 2024
Incident End Time: 19:08 UTC February 28 2024

We will continue to monitor for continued stability and provide a final update in the next two hours.