Postgres DNS resolution issues

Cross checking DNS records has been successful for some time. Customers report negative caches have since expired. Likely cause: race condition that caused misapplication of DNS records that was exposed under higher parallelism. We have to analyze it and patch it in the coming days.

We've identified this problem, and believe negative caching of DNS records to be all that remains. We are going to continue to check for at least thirty minutes, the longest time reported for negative caching so far.

Some records are proving more difficult to address than we expected.

We've hotfixed the remaining records. Negative caching of DNS can slow recovery, depending on the upstream DNS server. We will remain in contact with all affected customer until all symptoms are resolved.

There is a defect in Postgres that causes unbound DNS records. Negative caching can exacerbate the incident after our fix. Databases affected: many at the outset, some now. Diagnosing the last records.