Users are experiencing performance issue with the application (screens, API and Portal not responding or being very slow)

A necessary upgrade to the Dentally database performed on the weekend caused significant performance degradation. Despite comprehensive testing, this had not been observed prior to Monday. Isolating and resolving the issue took longer than expected, however full performance has been restored. No customer data was lost during this period. We are now analysing the root cause and the changes we can make to avoid such an incident occurring in future.

Service was fully restored at 12.30 today. We will continue enhanced monitoring through until 10am tomorrow after the peak of users logging in during UK hours.

Audit data will be available by 9am tomorrow morning.

We are continuing to monitor for any further issues.

Fixes have been applied. Calendar and Appointment functionality has been restored. Our monitoring shows that performance is back to normal.

We are monitoring the situation.

Audit data will be delayed and made available later today.

All users accounts that were locked out during the issue have been unlocked and users should now be able to access the application.

We have identified the cause of the issue and are looking into applying a fix so that the calendar can be enabled again. We will then look to begin restoring other services (Audit and Exports).

We are seeing performance increases while the Calendar is turned off.

As part of our investigation we turned off other areas of the system that remain impacted: audit data will be delayed - no data will be lost, and exports will be delayed. We are looking into users who were unable to login and will look to unlock their accounts who were affected by this issue.

We are temporarily disabling the calendar view from loading appointments. Users can view the appointments for a patient by searching for that patient and navigating to the “appointments” tab. Users will see errors when loading the calendar.

We continue to investigate the cause of slow application performance.

We paused ability to view appointments, including the calendar, between 10:54 and 10:55. We see an improvement over that time.

We are working with our system hosting provider to understand performance issues we have with out database. We are also now narrowing our investigation to appointment handling.

To investigate if we have an issue with Appointments we need to turn off the ability to view Appointments for 1 minute.

So far it has been possible to view the calendar. For this 1 minute window it will not be possible to view Appointments or the Calendar. Users will see an error.

We are continuing to investigate this issue. We are turning off the audit log and will backfill the data later in the day. This is to reduce the load on the database. We have also paused processing of some background jobs, such as Recall processing.

We continue to investigate the issue which is affecting the whole application (User Interface, APIs and Portal). Users will experience difficulty logging in and using any part of the application workflows.

We are continuing to investigate this issue.

We continue to investigate the root cause of the issue. At present we are increasing the system capacity to improve performance.

Starting at 8am this moring users began experiencing application performance issues. User interface slow or not responding. We are investigating.