Dashboard login failures and logouts

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We believe the root cause has been identified as a misconfiguration of our database responsible for storing user sessions, causing replication lag resulting in user authentication being inconsistently validated by our dashboard API. We are resolving the issue now by adjusting how we read stored user sessions for our dashboard API and fixing the misconfiguration of our database for continued high performance of our database.

The reset password page is also affected.

We've identified that the issue may be due to our reCAPTCHA integration for our login page. We have changed a feature flag and have confirmed the ability to log in using the "Stay logged in" checkbox.

We are doing root cause analysis to identify why customers are unable to login unless "Stay logged in" is checked during login. Some customers have reported not being able to login to dashboard at all.