API and call connection issues

This incident has been resolved.

Our tests just confirmed that DNS resolution is no longer returning errors for .co domains in our tests. This issue has been resolved!

The rate of bundle download failures has decreased, but it's also tracking our overall usage volume through the day. The number of affected users appears to be very small, but we still have tests that can replicate the failure. Unfortunately this is completely out of our control. If you have users that are continuing to be affected, you can suggest that they use Google's or Quad9's DNS servers at 1.1.1.1 or 9.9.9.9.

Cloudflare has resolved their status incident, but we're unsure if that means the underlying issue is resolved. We're continuing to monitor our own error rates and health checks.

Cloudflare has posted that they've implemented a fix, and they are monitoring the results. We're still not certain that Cloudflare is the actual root cause of 100% of our affected users, but we'll be monitoring error rates.

The .co registry appears to be experiencing issues. Cloudflare's recursive DNS is affected (and they've acknowledged it). Many regional ISPs, such as AT&T in the southeast US, rely on Cloudflare's DNS to power their own DNS.

Unlike Cloudflare, other DNS providers, like Google and Quad9, are serving .co records from stale cache. If they stop doing this, then anyone using those DNS services will start to experience these same failures.

This won't be fully resolved until the .co registry comes back online. In the meantime, different DNS providers may see problems come and go. All of Daily's services remain online and functional, so if your users can resolve .co hostnames, they can use our services without problems.

Cloudflare has posted an incident concerning intermittent DNS failures for .co domains: https://www.cloudflarestatus.com/incidents/z3b5zxjtp6g1

This aligns with the troubleshooting we've done so far. Some internet discussions suggest that it can be somewhat ISP-dependent, but this is unconfirmed. If this is indeed related to the .co domain itself, it would mean that it affects participants' ability to join calls, but it could also affect the ability to make API requests to api.daily.co. Updates to follow.

We've confirmed through several different end users that this is a DNS issue. Affected users from multiple regions have been able to join calls by pointing DNS to Google or CloudFlare, using IPs like 1.1.1.1 or 8.8.8.8 for DNS. Obviously, this solution doesn't scale; we're working with AWS to identify the root cause of the DNS resolution issue for c.daily.co.

We're receiving reports of a few other regions experiencing similar connection issues. If you're monitoring client errors and you see messages that start with "Failed to load call object bundle https://c.daily.co/....", you're being affected by this. We're working with AWS to get to the bottom of this.

We've identified an issue that's causing some users to fail to join calls. The daily-js library has to download a bundle of additional JavaScript as part of joining a call. This bundle is downloaded from c.daily.co, which is using Amazon's CloudFront CDN. They aren't reporting issues yet, but we're engaging with their support to figure out why this is happening.

We're receiving reports of some users having problems connecting to Daily calls. It seems to be localized to the Texas area. We're investigating.