3rd Party: Few Qualtrics connections were failing with HTTP Error Code 429

This incident has been resolved. We have observed that instance rates are returning to normal levels and affected connectors are syncing successfully.

Incident Summary

Description:
We identified an issue for Qualtrics which resulted in syncs failing with HTTP Error Code 429.

Timeline:
This issue began on 10th March at around 10PM UTC and was resolved on 14th March at 10:30PM UTC.

Cause:
This issues was caused by Third party 429s for distribution History endpoint which started spiking after 10th March at around 10PM UTC, we suspect if qualtrics had enforced additional rate limits at an app level in addition to the brand level api limit of 300 per minute

Resolution:
We increased the retries and also added exponential sleep between api calls to reduce the load on qualtrics server

A fix has been implemented and we are monitoring the results.

We identified an issue where the Distribution Contact endpoint was retrying requests too frequently, which caused some syncs to fail with rate limit errors (429).

A hotfix has been raised to adjust the retry behavior so that retries are spaced out more effectively. We will continue to monitor the syncs after deployment.

The issue has been identified and we are working to resolve it.