LiveKit Outage History

LiveKit's Global SIP service experienced a major outage where outbound TLS calls were failing for approximately 3.3 hours. The issue was initially thought to be Twilio-specific but was later determined to affect all outbound TLS calls regardless of provider. A fix was implemented and the incident was resolved after monitoring confirmed no further errors.

LiveKit Cloud Agents in the US East region experienced deployment failures and degraded performance for 5.7 hours due to an etcd service outage that caused Kubernetes API server unresponsiveness. The incident initially affected only new agent deployments but later expanded to impact existing deployments as well. The service was fully restored after the Kubernetes infrastructure was repaired with assistance from the data center provider.

LiveKit experienced SIP participant timeouts in Singapore and increased signaling connection errors primarily affecting India, with potential impact on connection latency and inbound SIP calls across multiple regions including Australia, US Central, and South Africa. The issue affected a very small minority of requests and was resolved through fine-tuning adjustments to improve stack performance. The incident lasted 116.5 hours with minimal user reports throughout the duration.

LiveKit experienced increased latencies in RoomService APIs (CreateRoom, DeleteRoom, and UpdateRoomMetadata) that initially affected the US West region but later expanded to other regions, caused by issues with a specific table in their distributed database. During the 4.8-hour incident, users experienced slow API responses and a temporary 10% failure rate when the fix was being applied. The issue was fully resolved after implementing a workaround and working with the database vendor.

The LiveKit Cloud Dashboard (cloud.livekit.io) experienced a major outage that prevented users from accessing the service. The incident lasted 13 minutes before a fix was implemented and normal access was restored.

LiveKit experienced degraded connectivity issues affecting Real Time Communication and TURN services in the EU (Frankfurt) region for 2.5 hours. The service team identified the root cause and implemented a fix to restore normal connectivity. The incident was resolved after a monitoring period to confirm service stability.

LiveKit experienced a 9.6-hour incident where analytics updates on their cloud dashboard were delayed, though real-time communication services remained unaffected. The issue impacted only the Cloud Dashboard (cloud.livekit.io), preventing users from seeing current analytics data. The problem was identified and fixed by deploying a solution that restored real-time processing, followed by backfilling the delayed analytics data to bring the dashboard fully up to date.

LiveKit Cloud dashboard experienced access issues that prevented users from reaching the service at cloud.livekit.io. The incident lasted 39 minutes while the team investigated and implemented a fix. The issue has been resolved and the dashboard is now accessible.

LiveKit Cloud analytics dashboards experienced delayed updates for 2.4 hours due to a processing pipeline issue, while data ingestion continued normally. The Cloud Dashboard (cloud.livekit.io) was affected during this period. The processing pipeline was restored and dashboards returned to normal updating functionality.