Elevated connection rate and 500's

This incident has been resolved.

Traffic patterns and service have normalized to regular levels observed prior to the maintenance window. We will conduct an incident analysis and write up a blog post for this next week. Thank you for your patience and I'm sorry that this impacted your workflows.

We've implemented a short term workaround to restore service. We're monitoring the service.

We're isolating the issue with authz service's database queries that is taking an abnormally long time to complete.

We're investigating an unexpected elevation in fetches by the authz service from the database.

After upgrading PostgreSQL we are seeing database connections and CPU spikes. We're resizing the database to get more system resources and will provide additional updates as we have them.