Funnel Outage History

Funnel's Salesforce data sources experienced widespread authentication errors due to mandatory Salesforce security updates that invalidated existing refresh tokens, causing affected sources to show "No Access" status. The incident also included issues with the reconnection flow that prevented some users from successfully reconnecting their data sources. The service deployed a fix that resolved authentication errors for most users through UI reconnection, while continuing to address reconnection flow issues for a subset of users until full resolution was achieved.

Funnel's Data Studio Connector experienced increased error rates over a week-long period, with users encountering "Bandwidth quota exceeded" and column mismatch errors when loading data. The issue was identified as a widespread problem on Google's side affecting multiple services. Google confirmed the fix after 23.8 hours, and Funnel verified no new errors were appearing in their logs.

Funnel experienced failing import jobs for Amazon Seller Central and Amazon Vendor Central connectors over 20.1 hours, affecting most data sources. The issue was identified and fixed with a deployed solution, and affected data sources were expected to be up-to-date within a few hours of resolution.

Facebook Ads data sources experienced delays due to "Internal Server Error" responses from the Meta API, affecting data processing across multiple sources. The issue persisted for 10 days before being resolved through collaboration with Meta. Facebook Ads data is now processing normally without errors or gaps.

X Ads and X Organic connectors experienced data retrieval issues due to X platform endpoints not returning data, likely caused by a major update to the X Ad platform. The issue affected all X Ads configurations except Campaign Non-Agg and X Organic Tweet Activity reports, while User Insights continued working normally. The problem was identified and a fix was implemented, with the incident resolved after 6 minutes.

Funnel's Google Looker Studio Connector experienced an outage where the connector was not responding and data failed to load in reports. The issue was caused by a broader Google problem affecting Community Connectors in Looker Studio that prevented customers from accessing their data. Google resolved the underlying issue after 17.8 hours, restoring normal functionality to the Funnel connector.

The Funnel service experienced delays in SFTP exports that lasted 71.1 hours. A fix was implemented and monitored before the incident was fully resolved. The issue affected the Funnel component's ability to export data via SFTP in a timely manner.

The TikTok connector in Funnel experienced missing values for several Ad level fields including Ad status, Video ID, and Ad landing page URL for approximately 27 days. The issue was caused by a change in TikTok's download schedule that created long download queues, preventing proper data retrieval. The problem was resolved by implementing a fix and adding observability measures and guardrails to prevent similar issues in the future.

The Funnel service experienced missing archived and deleted Facebook Ads data due to three separate API integration updates between January 2nd and March 10th that inadvertently broke the retrieval of historical ad content. Users were missing up to 30 days of deleted ads data, with some experiencing longer gaps if they had newly created sources or requested backfills during the affected period. The issue was resolved on March 23rd by reverting the problematic changes, and the team offered to prioritize backfills for customers who experienced data discrepancies during the January 2 - March 22 timeframe.

A regression in the AppsFlyer connector caused greatly inflated monetary values for Cost and Event Revenue metrics in data retrieved between March 16th 10:05 AM and March 17th 12:04 AM UTC. The issue affected AppsFlyer data sources and required manual backfills for some sources that did not self-heal as expected due to quota constraints. The incident was resolved after 187 hours with all affected data sources successfully backfilled and corrected.