LLM Output Degeneration

The mitigations that we deployed over the past day have resulted in significant reductions in the incidence of the garbled output across affected us-east-1 orgs.

We have additionally rerouted the relevant traffic to another cloud provider region that we expect to completely resolve the issue whilst we work with our vendor to identify the root cause and resolution path.

Problem: LLM outputs are occasionally returned as garbled text.

Impact: Garbled LLM outputs are affecting organizations in the us-east-1 region, occurring in about 0.1% of messages.

Causes: A hardware issue in us-east-1 combined with an ASCII to unicode prompt change that shifted tokenization boundaries.

Steps to resolve: We are deploying a fix and expect to confirm its effectiveness within about 30m.

We’re aware of an issue that may be impacting some users and are currently investigating. We’ll provide an update as soon as we know more.