Inngest dashboard performance issues

A fix has been deployed and we're monitoring the system to ensure the system is fully operational.

We have identified the issue for our Clickhouse cluster and are working to isolate resources for the Inngest dashboard to mitigate against further external load to ensure that the dashboard loads fast.

Performance should be very fast right now and we're quickly working on the new change to isolate resources to ensure performance through similar spikes in usage.

Issues have returned querying our Clickhouse cluster for runs, events, traces, metrics, and insights. We have scaled up resources and are working on additional profiling to identify bottlenecks.

The new read replica cluster is running well and we have shipped several other changes to improve dashboard loading times. Performance seems stable over the last hour. The team will continue to monitor in the morning with our new read replica cluster live.

The team has created dedicated read instances to isolate reads and writes for our Clickhouse cluster which services runs, events, metrics, and Insights. The change ensures that dedicated resources are available for all reads in our dashboard and API.

Dashboard and API load times should be performant again.

The Inngest dashboard is experiencing performance issues with loading some data including runs, events, metrics, and Insights. The team is working on fixes with the database that stores all of this data (ClickHouse). We're in contact with the provider's support team and working on mitigation efforts to alleviate performance issues ASAP.