API and item add downtime

Affected service teams have reported full recovery, we expect impact to be resolved now, though work continues internally to ensure database resources remain in a healthy state and the incident does not recur. Marking the incident as resolved for now, postmortem to follow.

Database recovery appears to be a success and we believe we are seeing recovery on all services that were affected by the original outage starting ~4:10 UTC. We are monitoring recovery and working on restoring other related systems (that are not currently affecting production traffic) to a similarly healthy state to prevent a recurrence.

We're expecting to potentially start seeing recovery in about 10 minutes based on database metrics. We will continue monitoring.

Impact began at ~4:10 UTC, we apologize for the delay in status page update. We have a dedicated engineer monitoring impact and remediation and updating the status page now.

We have identified the cause of the issue and a database recovery process has been in progress for ~45 minutes. We are working on identifying options for expediting the speed of this process, and also for obtaining an estimate for when impact will be resolved.

Will update with specific assessment of impacted API routes shortly.

We are currently experiencing a backend database issue that is causing downtime and increased error rates for some API endpoints, item adds, and Link flows. We will update momentarily with more details about specific impact and expected recovery time.