History lag and degraded performance on certain Horizon endpoints

Service has remained stable over the past week and thus the incident is resolved. Note that rate limits may dynamically adjust to preserve service health in times of high network activity or volatility.

Fix appears to have resolved performance issues, but reduced rate limits are still in effect as we continue to monitor.

A fix has been deployed. We are monitoring to ensure it is stable.

Additional reduction of rate limits.

Performance degradation observed again.

We flipped the standby and active endpoint. This temporarily fixed the problem.

Reports of degraded performance in Discord.

We reduced global rate limits by half for Horizon.

We've identified a high CPU utilization on SDF Horizon nodes due to increased request load that saturated DB connections.