Ada Outage History

Ada experienced delays in conversation view and article syncing in the US production environment for 7.5 hours. The team identified the root cause and confirmed that data was processing through a backlog with no data loss, while reports and API analytics remained fully operational throughout the incident. The backlog cleared automatically and the incident was resolved after extended monitoring.

A connectivity issue in the ficanex production environment caused data processing delays affecting Ada's Dashboard Analytics and Conversation View components. The disruption impacted data pipeline services, preventing normal dashboard functionality. The engineering team identified and resolved the issue after 1.6 hours of investigation and remediation work.

Ada experienced delays in conversation views affecting bots in the US2 cluster (those using the [bothandle].us2.ada.support format), impacting the Dashboard component. The issue was identified, fixed, and resolved after 1.3 hours of monitoring to ensure stability.

AI agents in the US2 cluster were failing to load their chat windows, displaying error screens instead of the expected interface. This affected the Chat Client component for approximately 2.2 hours. The issue was identified, a fix was implemented and monitored, and the incident was fully resolved.

The Ada admin dashboard failed to load for US cluster customers, who experienced 500 errors when attempting to access their AI agent dashboards, while other clusters remained unaffected. The issue was identified and a fix was implemented after 25 minutes of investigation. The incident was fully resolved after 52 minutes of total downtime.

Ada's Conversations View experienced delays for 2.9 hours, preventing conversations and new messages from appearing promptly in client dashboards. The issue was identified and a fix was implemented, though some conversation data required additional processing time before becoming visible. The incident has been fully resolved.

Ada's chat client experienced an issue where customer satisfaction surveys were incorrectly displayed at unexpected times, appearing in followup conversations that weren't tied to human or AI agent interactions. The problem affected the chat client functionality for 5.6 hours before the team identified the root cause, implemented a fix, and resolved the incident.

Azure PTU failures caused increased handoffs across all clusters in Ada's Unified Reasoner service, affecting conversations using this feature. The issue was identified as originating from an upstream service provider and lasted 14 minutes. A fix was implemented and the incident was resolved after monitoring confirmed the solution was effective.