Transformation Runs are Failing

This incident has been resolved. Transformation job runs continue to run without failure, and transformations are running successfully.

Incident Summary
Description:
Transformations started to fail with an error containing "('communication error', gaierror(-2, 'Name or service not known'))".

Timeline:
This issue began on Sept 15th at 23:24 UTC and was resolved on Sept 16th at 11:15 UTC.

Cause:
The issue started when DBT runner jobs encountered errors. This occurred due to throttling in a DBT job runner component. Additional intermittent errors were seen following the first applied fix.

Resolution:
We mitigated this issue by increasing the resources and, thus, the performance of this component. This additionally required restarting another related component of the DBT job runner to resolve intermittent errors.

We have identified an issue with a component of the Transformation runner, which was causing an issue with the start of the Transformation jobs.

This component has now been restarted, and transformation jobs are now running successfully. We are continuing to monitor to ensure the continued success of the Transformation job runs.

There are some issues with DNS resolution in a few cases, and we are working on mitigation.

The issue has been identified and we are working to resolve it.