Degradation in queue times causing runners to sporadically not get assigned a job

We are currently investigating this incident.

We have identified the issue as likely being our database connection pool limits, we are rolling out a fix to see if it helps with the situation.

The deployed fix has not completely mitigated the outage. Customers may also be noticing degraded load times for app.blacksmith.sh

We are continuing to investigate the cause of this incident. All signs point to a degradation in our upstream database provider. We are actively working with them to resolve the incident.

Our database provider has declared an incident that is directly affecting our control plane - u003chttps://status.supabase.com/incidents/kfrmjq5z9n8xu003e. We are continuing to work with them on a timeline for a fix.

Our database provider has rolled out a fix and is claiming recovery. We are also seeing signs of recovery on our end.