Illuminate Data Refresh Issue in US & EU regions

This incident has been resolved.

The fix is now live in production for all instances and has also been applied to affected instances in lower environments (test and staging). We are monitoring the results.

It has been determined that the underlying issue was the result of a vendor change included in the recent Blackboard release that interrupted communication between Blackboard and Illuminate for some requests involving large amounts of data. The fix will be applied in conjunction with the next Blackboard update. For production environments, this means the fix will be deployed with the 3900.121.0 update today. For test and staging environments, the fix will be deployed on Tuesday with the 3900.123.0 update.

We have identified the root cause and the engineering team is working on a fix, with the goal of including it in the upcoming Blackboard 3900.121.0 and 3900.123.0 production releases.

TLS handshake errors in the connection between Blackboard and the Snowflake database have been identified as a potential root cause. We are actively investigating the vendor’s proposed mitigations.

Our engineering team continues to investigate the issue to identify the root cause.

We are currently experiencing issues with the data pipeline in US and EU regions, which may result in delays in the refresh of Blackboard data for clients in these regions. Our team is actively working to resolve the issue and restore full functionality.