Investigating Issues with Serverless Project Creation and Scaling in GCP Region us-central1

This incident has been resolved.

The impact of this issue to customers remains mitigated. Changes to safeguard against additional impact have been deployed.

We are continuing to monitor the situation and will provide an update if there are any status changes.

The impact of this issue to customers has been mitigated. Our engineering team is actively monitoring the situation and deploying safeguards to ensure there is no additional impact.

We will provide an additional update in 60 minutes or as soon as we have an additional development.

We have identified additional impact of this issue impacting Serverless project creation and scaling in additional GCP regions. Our engineering team is currently working to implement and roll out a fix to mitigate the issue.

We will provide an additional update in 60 minutes or as soon as we have an additional development.

A fix has been implemented across all affected clusters, and we keep monitoring the situation closely. Initial observations indicate that services related to project creation and scaling are returning to normal operation.

We will continue to monitor the situation for stability and provide an update if there are any new developments.

A fix has been implemented across all affected clusters, and we keep monitoring the situation closely. Initial observations indicate that services related to project creation and scaling are returning to normal operation.

We will continue to monitor the situation for stability and provide an update if there are any new developments.

A fix has been implemented across all affected clusters, and we keep monitoring the situation closely. Initial observations indicate that services related to project creation and scaling are returning to normal operation.

We will continue to monitor the situation for stability and provide an update if there are any new developments.

We have a candidate fix in a testing environment where we have successfully applied it to the first set of affected clusters. We have confirmed that project creation and scaling are now functioning. The rollout of the fix to the remaining impacted environments is ongoing, and we will continue to monitor the situation closely.

We will provide our next update within 60 minutes, or as soon as we have a significant development.

We have identified the root cause of the issue affecting deployment creation and scaling on Google Cloud Platform, which is related to a recent automated infrastructure update from our cloud provider. A mitigation plan has been identified, and our teams are coordinating to apply it across the affected clusters. We have also taken steps to prevent further impact.

We will provide another update within 60 minutes with more details on the timeline for resolution.

We are currently investigating an issue that is primarily affecting new deployment creations for some customers on Google Cloud Platform. Existing deployments remain operational; however, there is a potential for impact to scaling operations which we are actively investigating. Our engineering teams are working to identify the root cause and restore full functionality.

We will provide a further update within 60 minutes, or sooner if we have significant new information.