Delayed Retrievals - Canvas Clients Only

Canvas had pushed through the API fix to production. Any courses that were previously excluded due to this issue, will be pulled in overnight and will appear in your dashboards tomorrow morning. Thank you for your patience as we worked through this issue with Instructure.

All dashboards are fully updated and operational. As we shared yesterday, we orchestrated a workaround to exclude the courses that were causing the retrievals to fall due to the broken API. Canvas Support has advised that they hope to have a fix pushed to beta by the end of the day for testing. Once the fix has been pushed to production, we will bring back in the courses that we have excluded. We will continue to provide updates on this issue until it is fully resolved.

We are still waiting for Instructure to push the API fix to production. In the meantime, we have run a script to determine any course that contains discussions that are preventing the retrieval from completing. For each school, it is a very small subset of courses. We are in the process of excluding those courses from being pulled in, so that the retrieval will complete and you will have data for the vast majority of your courses. We've been successful in getting several of the schools fully up and running again. We're continuing to make progress. Tomorrow morning, we will re-evaluate and ensure that we 'caught' all of the offending courses and have successfully excluded them from interrupting the retrieval process. Once all dashboards are fully operational, we will send each of you a list of the courses that we excluded. Thank you for your patience as we attempt to work around the failing API and provide you with up-to-date dashboards.

The fix for the API bug that is keeping us from updating the dashboards of some Canvas clients is currently being reviewed by the Quality Assurance team at Canvas. We'll keep you posted.

Our retrieval process for some Canvas clients is still on hold until the Canvas engineering team pushes through a fix to the API bug that was introduced during Saturday night's production release. Instructure has not provided an ETA, but they have assured us that there is a team actively working on resolving this issue as soon as possible. We understand that you are anxious to have your dashboards updated. We are too. Our team will continue to be on stand by and the minute that a fix is pushed to production, the updating of the affected dashboards will begin immediately. We know that you rely on our solutions. And we rely on the Canvas API to retrieve the data to populate our solutions. We will continue to work closely with Instructure and keep you informed of their progress.

Per Canvas, the API bug has been escalated up to a Level 3 engineer. They have assured us that they will keep us posted on the progress for addressing this issue. We will be back in touch as soon as we receive the next update.

Canvas Support continues to work on this issue. We do not have any new information to share at this point, but wanted to assure you that they are working as quickly as possible to rectify the API issue. We are literally testing that API on a minute-by-minute basis. The very moment that the API is fixed, retrievals will spin up and we will update your dashboards as quickly as possible. Our entire team is on stand by waiting for the fix to be pushed through by Canvas. Thank you for your continued patience as we work closely with Canvas to find a solution ASAP.

Canvas Support has replicated the behavior that we identified and notified them of. They have escalated the issue to their Engineering team and created an 'engineering tracker' so that we will receive updates on this issue as soon as they are available. We have stressed the importance of how much you rely on this data and we will continue to remain vigilant on getting this issue resolved so that we can update your dashboards as quickly as possible. We appreciate Instructure's support in getting this unexpected issue addressed and we appreciate your patience while we work with them towards a successful conclusion of this issue.

We are still awaiting a response from Canvas Support. For clarification, this issue is only affecting clients that use 'group discussions' that require an initial post be made in order to view the posts made by others. In this case, the API is returning an error when we attempt to retrieve student discussion posts. This issue was introduced with the changes pushed to Canvas production last night, but was not documented in the Release Notes.

For those that are affected, your dashboard is still populated - but it is displaying yesterday's data. If you are unsure whether your instance is affected, please go view the 'last retrieval' timestamp on the bottom of your Dropout Detective dashboard.

This incident is only applicable to Canvas clients. Only some clients are affected. To determine if your account was affected, please view the timestamp at the bottom of the Dropout Detective dashboard. If the timestamp includes today's date, you were not affected. If it reflects yesterday's date, your account has been affected.

Last night, Canvas released some changes to production. Some of those changes were related to discussions, but appeared to be only applicable to the user interface - not the API. However, it appears that intentionally or not, a change was made that is prohibiting us from obtaining discussion posts from group discussions. This only affects discussions that require an initial post to be able to view the posts made by others.

We have submitted and escalated a help ticket to Instructure and are awaiting their support. This incident will be updated when we receive a response and are able to provide you with additional information.

This issue also appears to be affecting the admin user experience in a course. Therefore, we expect a swift resolution to this issue as it is also affecting the user experience in Canvas for discussions that require an initial post. Example: If the discussion settings require that a user must post before seeing posts made by others, and any admin (program director, dean, etc.) has not posted to the discussion, they are unable to see the posts made by students in the discussion page.

We will keep you informed throughout the day. Thank you!