Intermittent volume request timeouts

We've improved the underlying storage infrastructure, and are monitoring to ensure that the performance has returned to normal.

We continue to investigate intermittent volume request failures affecting persistent volumes.

Our investigation has identified instability in the underlying NFS storage layer as the likely source of the issue. We are seeing elevated timeout, retransmission, and connection reset rates between our infrastructure and the storage service backing persistent volumes.

We have engaged GCP support and are actively testing mitigations, including reducing connection pressure on the storage system and introducing additional caching to decrease request volume.

We will provide another update as we learn more or have a mitigation deployed.

We’re investigating intermittent timeouts affecting volume requests.