Notification of new incident. 2023.3 Deployment forced new device UI.

Hello Everyone,

Update: Incident Resolved.

With the remediation actions in place, we are not seeing the new device UI forced out to tenants. Our metrics and confirmation from the field we will move this incident to resolved. If there are remaining concerns, please raise these via established support paths.

Date/Time: September 8th, 3:00 am UTC

We will review the findings and collected information from this incident to further improve our online services. There will be a postmortem published once the investigation and review has been completed. For listed as 'Major' this will be no longer then (max 20 business days).

We are sorry for the inconvenience this has caused.

Kind Regards
Online Operations Team

Update:
The mitigation actions have now been successfully applied. This will have the following effect:

* Express devices will be reverted to the ‘Classic UI’.

* Tenants that have NOT saved the device UI previously will be reverted to the ‘Classic UI’.

* Tenants that have saved the device UI will remain unaffected by this rollback. This will include all customers today that performed roll back procedures during support calls.

Please note that it may take up to 24 hours for devices to receive the configuration update. A device can be restarted in order to retrieve the configuration update sooner.

Next Update:
The next update will be in 24 hours or as new information becomes available.

Update:

Root Cause:
NT-ware has identified the cause of the new device UI behaviour being deployed. The intention was that the new UI was only pushed out onto new tenants. The criteria for these decisions were any tenant that had not previously saved the new device UI option. This meant that even if you stayed with the default device UI uniFLOW Online wanted this saved.

As many people did not save the default device UI this left the setting blank and resulted in the misidentification as a new tenant.

For many customers this simply meant the UI changed to the modern look. Unfortunately for a number this had a negative impact where the forced update pushed this onto devices with an earlier MEAP version. In this configuration there was a known issue with TLS setting on the device and would force the device offline.

This issue also impacted uniFLOW Online Express customers in the same way. However, the express tenants do not have the admin options available to revert this change.


Corrective Action:
To correct this for customers the suggestion was to change the device UI back to ‘Classic UI’ and save. Feedback from the field is that this had a positive result largely corrected the situation.


Next Steps:
* NT-ware have a mitigation plan in place which will be put in place over the next 4 hours.

* Express devices will be reverted to the classic device UI during this process.

* Tenants that have NOT saved the device UI previously will be reverted to ‘Classic UI’.

* Tenants that have saved the device UI will remain unaffected by this rollback. This will include all customers today that performed roll back procedures during support calls.


Next Update:
The next update will be in 24 hours or as new information becomes available.

Update:

Apologies for the delay with this update. We have identified and confirmed the root cause and working on a mitigation plan. We will provide a detailed updated within the next hour.

Next Update:
The next update will be in 60 minutes.

Update:
On further investigation we have identified a number of customers that have devices failing to connect after the device UI update. Early investigations have identified a mismatch between the MEAP version and device TLS settings. This can be resolved by following the instructions in the initial status page post to roll back the UI setting.

NOTE: This issue is also impacting uniFLOW Online Express customers who do not have the admin login option to revert to the traditional device UI. For Express customers impacted by this change please reach out to your Canon / Canon Business partner for support instructions.

Next Update:
The next update will be in 3 hours or as information becomes available.

Start Time:
As of 9am UTC Wednesday for the AU Deployment
As of 1pm UTC Wednesday for the SG Deployment

Incident Scope: 
Australian and Singapore Deployment

Description:
NT-ware deployed uniFLOW Online 2023.3 which provided a new and long-awaited device UI. This UI was intended to be optional for existing tenants (customers) and default for new tenants. Post deployment it was reported that ALL tenants were forced to the new UI.

Rollback Options:
For many customers this will have little to no impact to your environment and can be left in place. However, for customers that need to manage change control or user experience, you can roll back to the previous device UI by logging into the Tenant admin portal.

Device UI:
If disabled, the classic blue MEAP user interface is shown on the touch-panel display of the devices. Check the Enable modern device UI checkbox to enable the configuration of a custom user interface for Canon imageRUNNER ADVANCE devices (Generation 3 or newer). Once the modern device UI has been enabled, card login and PIN code login are always enabled, regardless of the further device UI configuration.

https://www.nt-ware.com/uniflowonline/doc/2023.3/contexthelp/#/home/135304/en/help

De-selecting this option will revert back to the original device User Interface.

NT-ware understands that this will be an unintended change for many customers and are investigating the issues at this time.

Next Update:
The next update will be in 5 hours or as information comes available.