LiveKit experienced increased latencies in RoomService APIs (CreateRoom, DeleteRoom, and UpdateRoomMetadata) that initially affected the US West region but later expanded to other regions, caused by issues with a specific table in their distributed database. During the 4.8-hour incident, users experienced slow API responses and a temporary 10% failure rate when the fix was being applied. The issue was fully resolved after implementing a workaround and working with the database vendor.
Trusted by 1,000+ teams
Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.
This issue is now fully resolved. We will post a detailed RCA soon.
Our fix is fully implemented and we are not seeing any more failures and latencies in the RoomServices APIs. We are continuing to monitor the issue.
While applying a fix for the API latencies, we are temporarily seeing increased failure rates (about 10% of the calls) in RoomServices APIs, including CreateRoom and DeleteRoom APIs. We are actively working on mitigating this.
We continue to see the long Room API latencies are also impacting other regions. The latency increases appear to originate from a specific table in our distributed database. The issue has been escalated with the database vendor and we are working on a workaround for decreasing the API latencies. Other services are not impacted.
We believe these elevated latencies began around 22:00 UTC. We have confirmed that only API requests in US-West should be impacted. The current list of impacted APIs appears to be CreateRoom, DeleteRoom, and UpdateRoomMetadata. We are working on mitigating the issue to return latencies back to normal.
We are continuing to investigate this issue.
We are investigating reports of increased latencies in RoomService APIs in the US West region, specifically on CreateRoom, DeleteRoom, and UpdateRoomMetadata APIs.
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 6320 services available
Integrations with