RIPE Atlas controller problems

This incident has been resolved. It was ultimately caused by an overly restrictive permission check in the component that serves as the entry point for probe (the "registration server"). In itself this was not a major problem and would have been detected and resolved before it affected the probe population. However, independently of this, and as part of our normal software development process, we rolled out upgrades to our controlling infrastructure, which caused most of the probes to disconnect and try to re-register with the system. This is normal and expected - but in this case the registration server was not ready to answer them, and as a result they were unable to immediately re-join.

A fix has been implemented and we're monitoring the situation.

We identified the root cause of the issue and applied a fix, probes are connecting again.

A part the RIPE Atlas probe population is unable to connect to our infrastructure. We're investigating the issue.