Modal Labs Outage History

Modal Labs experienced a major service outage when their Volumes service went down, causing failures across CPU functions, GPU functions, snapshot restores, frontend, sandboxes, and image builds. The widespread impact affected multiple core platform components that depend on the volume storage system. The incident was resolved after 36 minutes of downtime.

Modal Labs experienced widespread internal data store failures that affected multiple core services including CPU/GPU functions, web endpoints, API, volumes, sandboxes, and image builds. The incident was classified as minor status and lasted approximately 4 hours. The team actively worked to resolve the data store issues and restore normal service functionality.

Modal Labs experienced container scheduling failures and RPC errors caused by a storage failure, affecting CPU functions, GPU functions, web endpoints, sandboxes, and image builds. The incident lasted 4.4 hours and was classified as minor severity. The team worked to mitigate the storage-related issues during this period.

Modal Labs experienced degraded H200 GPU scheduling latency due to a capacity shortage, affecting GPU functions for 8.3 hours. The incident was classified as minor and has been resolved.

Modal Labs experienced degraded sandbox functionality for 48 minutes due to overly aggressive CIDR allowlist enforcement from a code change. The issue blocked egress traffic needed for tunnels, exec API, filesystem API, and connection tokens. A fix was merged and rolled out to restore normal sandbox operations.

Modal Labs experienced degraded container scheduling for 4 minutes due to a DNS change. The incident affected CPU functions, GPU functions, and web endpoints. The issue was resolved after the 4-minute duration.

Modal Labs experienced a major outage when their us-west input plane went down, affecting CPU functions, GPU functions, and web endpoints for 8 minutes. The input plane has since recovered and service has been restored.

Modal Labs experienced a major outage affecting their web endpoints service. All web endpoints were completely down during the incident. The service was restored after 8 minutes.

Modal Labs experienced degraded container creation for both CPU and GPU functions due to a problematic change. The issue was resolved by rolling back the change, with recovery observed after approximately 2 hours.

Modal Labs experienced increased latency affecting web endpoints for 2 hours. Some requests to web endpoints were impacted by degraded response times during this period. The incident was classified as minor and has been resolved.