Performance Degradation in CI/CD Pipelines

We have implemented the necessary backend mitigations to resolve the recent service disruptions, and system stability has been fully restored.

Action required: If you experienced an error, please re-authenticate your credentials and retry your jobs. Please note that the main Credential for the Org where Copado is installed must be validated first; after that is complete, all subsequent Credential records must be validated by their respective record owners in Salesforce.

Status update: Monitoring
We have observed a complete drop in authentication errors over the last hour and currently consider this issue resolved. Our team will continue to monitor the system to ensure stability.

Action required: If you experienced an error, please re-authenticate your credentials and retry your jobs. Please note that the main Credential for the Org where Copado is installed must be validated first; after that is complete, all subsequent Credential records must be validated by their respective record owners in Salesforce.

The majority of the instability being experienced is due to the "frequent rotation of Refresh Tokens" now being enforced by Salesforce, which we are in the process of further mitigating, at scale.

We are currently investigating reports of performance degradation impacting CI/CD pipelines. During this time, some users may also experience intermittent delays or unexpected validation errors when utilizing the CICD Pipeline, which may also affect the Data Deploy Service.

Our engineering teams are actively looking into the situation to identify the root cause. We will provide an update as soon as more information is available.