Processing delays - Events and Companies

This incident is now resolved. Our key metrics are back to normal and all queued jobs have been processed.

We're seeing timeouts on approximately 4% of requests coming into Chameleon with the url prefix of https://fast.chameleon.io/observe/ and are still working to get back to fully operational

We've identified a problem in how Heroku handles DNS resolution from Private spaces and are in contact with the support team. A large percentage of our servers are connecting to our database properly but the rest are failing to resolve the cluster address down to the addresses of the cluster instances. We're working on a possible workaround at the moment

We're seeing elevated error rates and retries related to processing Events and updates to Company properties. We're investigating the source of the problem and will report back here with more information.