Resolved -
This incident has been resolved.
Jul 14, 13:43 EDT
Monitoring -
The fix is now live in production for all instances and has also been applied to affected instances in lower environments (test and staging). We are monitoring the results.
Jul 14, 12:25 EDT
Update -
It has been determined that the underlying issue was the result of a vendor change included in the recent Blackboard release that interrupted communication between Blackboard and Illuminate for some requests involving large amounts of data. The fix will be applied in conjunction with the next Blackboard update. For production environments, this means the fix will be deployed with the 3900.121.0 update today. For test and staging environments, the fix will be deployed on Tuesday with the 3900.123.0 update.
Jul 10, 14:34 EDT
Identified -
We have identified the root cause and the engineering team is working on a fix, with the goal of including it in the upcoming Blackboard 3900.121.0 and 3900.123.0 production releases.
Jul 9, 19:22 EDT
Update -
TLS handshake errors in the connection between Blackboard and the Snowflake database have been identified as a potential root cause. We are actively investigating the vendor’s proposed mitigations.
Jul 9, 14:10 EDT
Update -
Our engineering team continues to investigate the issue to identify the root cause.
Jul 8, 18:37 EDT
Investigating -
We are currently experiencing issues with the data pipeline in US and EU regions, which may result in delays in the refresh of Blackboard data for clients in these regions. Our team is actively working to resolve the issue and restore full functionality.
Jul 8, 11:40 EDT