Is Vertex AI Training Down?

First, stay calm.

• We know it's stressful when Vertex AI Training is down, but running around is not going to fix it.

• Remember that it's normal for services to go down sometimes. It can be for a number of reasons, from maintenance to unexpected issues.

Make a quick evaluation of the impact of the Vertex AI Training outage

• Try to evaluate the impact of the outage on your business. Is it impacting a majority of your users or just a few?

• Did Vertex AI Training already acknowledge the issue? And published a status update? (You can find it in the top of the page or in Vertex AI Training status page)

• If not, understand if other users are also experiencing issues. You can also check this information on the top of the page.

Communicate with your team and users quickly

• If Vertex AI Training is having problems and impacting your business, it's important to keep your team and users informed.

• Share this page with your team, so you can start incident response procedures. No need for everyone to go around searching for information. Create a post or a specific channel in your internal communication tools (Slack, Teams, Google Chat, etc.) so you can aggregate all the information in one place.

• If you have a status page, create a status update to inform your users about the issue. This way they know what's going on and you can avoid a flow of support requests overburding your team.

Understand the root cause

• After the first communication, let's move on to the next step. Do a more thorough investigation to understand the root cause of the issue and impact.

• Reach out to the service provider to understand if they are aware of the issue and if they are already working on it. You can reach them in their support channels, or their social media channels if they have any.

Is there any mitigation for the issue?

• Understand if there are workarounds or mitigations for the issue. Can you use a different provider? Is the switch easy to do? Talk with your team to understand the best course of action.

• Need to wait on the service provider to fix the issue? Understand if there is a timeline for the fix. Adapt your communication with your team and users to the situation. It's very important to keep everyone informed.

Did Vertex AI Training recover?

• Continue to monitor the situation. A lot of times, the service can see a brief recovery and declare the issue as resolved, but it can come back. Keep monitor for some time to make sure it's really resolved.

Post-incident analysis

• Once the issue is resolved, it's important to do a post-incident analysis. Understand what went wrong and if it's possible or needed to do something to prevent it from happening again.

• Sometimes if it's a critical dependency, you might need to have a backup plan in place to avoid a similar issue.

• Vendors usually publish post-mortem analysis reports, so keep an eye on their social media channels and website for more information.

Prepare for the next incident, because it will happen again

• Start monitoring your dependencies. Sign up to IsDown and aggregate all your dependencies status in one place.

• Nowadays, it's very common to have a lot of dependencies, and keeping an eye on them is not an easy task. IsDown can help you with that, by aggregating and monitoring all dependencies status and alerting you if they are having issues. Create status pages to communicate with your team and users.

• For a complete overview of all Google Cloud services, visit the Google Cloud Status Page on IsDown