Private Location Agents in Error state

As of 00:30 UTC, the issue experienced by some customers has been addressed and functionality has been restored. This incident is now resolved.

The root cause of the issue has been identified by the R&D team. Unfortunately an automated fix cannot be rolled out to the affected Agents so they will need to be manually restarted if they are in Error state due to this issue. The steps to restart the Agents are as follows:

Docker Steps:

sudo docker ps -a
sudo docker start {{crane containerId}}

Kubernetes Step:

kubectl rollout restart deployment {{deployment_name}} -n {{namespace}}

The root cause of the issue has been identified by the R&D team. Unfortunately an automated fix cannot be rolled out to the affected Agents so they will need to be manually restarted if they are in Error state due to this issue. The steps to restart the Agents are as follows:

Docker Steps:

sudo docker ps -a
sudo docker start {{crane containerId}}

Kubernetes Step:

kubectl rollout restart deployment {{deployment_name}} -n {{namespace}}

We are still continuing to identify the root cause of the issue. However most Agents can be returned to a functional state by restarting them. The steps for restarting an Agent are:

Docker Steps:

sudo docker ps -a

sudo docker start {{crane containerId}}

Kubernetes Step:

kubectl rollout restart deployment {{deployment_name}} -n {{namespace}}

We are continuing to investigate this issue.

We are aware of an issue with Private Locations at this time that has caused them to go into Error state. We are actively investigating the issue now and will provide updates as soon as possible. Restarting or Regenerating the Agents has resolved the issue for some customers so we recommend attempting that for Agents currently in and Error state.