Degraded performance: login failures and slow loading of emails and account data

The permanent fix has been deployed to all production clusters and all systems are operating normally. During the incident (July 30, 22:48 UTC – July 31, ~04:30 UTC, with residual slowness until ~08:49 UTC), some users experienced login failures, missing team member data, and slow loading of emails. No data was lost. We have added memory-usage alerting to detect this class of issue before it affects users. We apologize for the inconvenience.

Cache capacity has been increased and the system is recovering. Response times are returning to normal as caches warm up. A permanent fix is being deployed.

We have identified the root cause: an internal caching service (Redis) ran out of memory and began rejecting writes. As a result, permission checks bypassed the cache and put excessive load on the account service, causing slow responses, login failures, and missing account data. We have increased the cache capacity and are preparing a permanent fix.

We are investigating reports of users being unable to log in, seeing missing team members, or experiencing slow loading of emails. Our team is working to identify the cause.