3rd Party: Multiple Customer.io connections are failing with 5xx error

This incident has been resolved. We have observed that instance rates are returning to normal levels and affected connectors are syncing successfully.

Incident Summary
Description:
We identified an issue for Customer.io, which resulted in syncs failing with a 502 HTTP error.

Timeline:
This issue began on 2026-06-13 20:26 UTC and was resolved on 2026-06-14 14:52 UTC.

Cause:
The issue was caused by intermittent upstream API failures on the Customer.io side. The root-cause behaviour fluctuated throughout the incident, shifting between generic server errors across multiple endpoints.

Resolution:
Fivetran implemented a skip-on-failure mechanism to bypass 502 HTTP errors and allow syncs to proceed. A permanent root-cause fix will be deployed by Customer.io.

We observed a shift in the issue behavior around 12:00 UTC for the Customer export api. While initial failures occurred during job creation, the connector is now failing during the report download phase.

- We have implemented a skip mechanism to mitigate these 502 errors on the affected endpoints, and are continuing to closely monitor all active syncs.
- Additionally, we reported the ongoing behavior to Customer.io, as their initial fix did not resolve the issue. We are currently awaiting their response.

Next Expected Update: < 3 hours >

We noticed that the connections were failing again with "502 Server Error". We've raised a hotfix, and the failures are gradually reducing. We're monitoring the affected connections.

Next Expected Update: < 3 hours >

We are continuing to monitor all affected connectors after the recent fix deployment.

Next Expected Update:3 hours <3 hours = P2, 2 hours = P1>

A hotfix has been deployed to skip the failing endpoints until the source resolves the issue. We will continue monitoring the connections while we wait for a response from the source team.

Next Expected Update: 3 hours <3 hours = P2, 2 hours = P1>

We are preparing a fix to skip the failing endpoints until the source resolves the issue.

Next Expected Update: 1 hour <1 hour = P2, 30 minutes = P1>

We identified that multiple REST API requests are failing with 502 Bad Gateway errors returned by the source API.

We have reached out to the source regarding this.

Next Expected Update: 1 hour <1 hour = P2, 30 minutes = P1>

The issue has been identified and we are working to resolve it.