Facebook Ads Intermittent Data Delays

Today we saw a halving again of the rate of affected sources, putting the rate at less than one in every 10,000 sources. We are continuing to monitor this issue and escalate the issue with our Meta contacts. The issue is continuing to behave non-deterministically: affecting different sources each day.

This week the number of affected sources has been relatively stable at the rate observed at the end of last week: about half the rate seen previous weeks.

Previous weeks we have also seen less sources affected on Mondays than on other weekdays, but that was not true this week. This indicates that a periodic effect has been resolved, while we are still seeing at least one other, non-periodic, effect that is responsible for the residual (and that when the incident began we were observing the combined effects).

Update: Today half as many sources were affected as were the previous two days. We are continuing to monitor for further improvements.

We have received additional confirmation on our support ticket with Meta that they are actively investigating the sustained rise in errors we have been seeing since July 17th.

Following the pattern we observed last week, this Monday morning half as many sources were affected. Tuesday morning we observed around the same rate as we observed most of last week. This is a behavior pattern consistent with a load dependent error pathway.

We have escalated this issue with internal stakeholders at Meta in addition to regular support, to expedite remediation. Today we have less than half as many sources experiencing issues as last Friday, and last weekend many of those sources were able to receive at least partial updates, perhaps due to lower total load on Meta systems over the weekends.

We understand that many of you are putting together end of month reporting, and this has been a disruption to your processes. We are continuing to experiment to see if there are any additional mitigations we can apply from our side, while also continuing to make our normal requests to the Meta servers for updated data.

Update: Less than one in ten thousand sources have been persistently affected by this issue. The other sources affected by this issue (around one in every thousand) have been affected intermittently, experiencing some data updates, but not as many as usual.

We are continuing to request data for all affected sources, but the Facebook Ads API is not returning the data reliably. We are actively collaborating with Meta's support team, providing detailed debug logs and source examples to assist in their investigation.

Of the sources affected on Friday, more than 85% have returned to normal operations, and the total number of delayed sources has been reduced over 30%.

Meta has responded to our ticket, requesting additional information to aid in debugging the issue from their side.

We implemented several mitigations on Friday, and today 76% of the affected sources have returned to normal operations. We are still investigating further mitigations to remediate the remaining affected sources. We filed a bug with Meta on Friday to better understand the API behavior change, but Meta has not yet responded.

Starting on July 17th, we noticed an issue affecting less than one in every thousand Facebook Ads sources. The issue results in failed data pulls.

These sort of data pull issues usually resolve on their own within one to two days, however this issue has persisted for over a week. The issue is intermittent, with some sources downloading data for one data pull and not the next.

Greater than 85% of the affected sources have been able to update data in the last two days, and greater than 95% of the affected sources have been able to pull updated data in the last three days, however these updates are not as reliable or consistent as usual for the affected sources, so affected sources are experiencing delays.

We are currently investigating the cause of the issue, but have so far not been able to find any unique traits shared by only the affected data sources, nor differences in our environment between the runs which succeed compared to those which fail for the affected sources to point to the source of the non-deterministic behavior.