Degraded Performance Across Multiple Services

This incident has been resolved.

All deployments have been restored, and the issue with our cloud provider has been resolved. Services are now up and running, and we’re continuing to monitor to ensure stability.

We’ve identified the root cause of the issue and are working on deploying a fix.

Update: All services, except perception, are currently operating as expected.

Update: We’re experiencing service degradation due to an issue with our cloud provider that is throttling model inference. We’ve restored some deployments and are working with them to bring all systems back to normal.

We’re currently seeing elevated error rates across multiple parts of our stack, including perception, knowledge base, Gemini conversation LLM, objectives, and guardrails.