Delayed and Timed-out requests to the Aptible LLM Gateway

Following the fixes deployed per the previous update, we have seen stable performance on the LLM Gateway since 7:12 EDT and are resolving the issue.

The root cause of this issue was a severe performance degradation in 3 specific endpoints:
• `GET /models`
• `GET /v1/models`
• `GET /metadata/models`

...all of which return a list of available models. The performance issue on these endpoints was severe enough to cause significant request queueing and degraded performance for other unaffected endpoints. The fix we deployed added caching on this endpoint to ensure fast responses. Since those fixes have been deployed, we are no longer seeing request queueing or impacts to other LLM Gateway API endpoints.

After a regression in performance starting at 9:57 UTC (5:57 EDT) and lasting until 11:12 UTC (7:12 EDT), we've deployed a potential fix and updated our service scaling. We are continuing to monitor whether these changes fully resolve the issue before closing this incident.

We have increased resource capacity, and we are monitoring the results.

We are aware of delayed and timed-out requests to the LLM Gateway. We have made some changes to help minimize the impact while engineers continue to address the root cause.