DBaaS management degraded

There are issues after the planned maintenance of the DBaaS management system.

Connections to your databases are not affected.

The issues are purely in the management system.

Management functionality might come and go while we are fixing the issues. Backups and other jobs handled by the management system might be delayed.

Avoid doing any changes to your setup until the issues are solved.

We have discovered a few selected data stores that are experiencing connectivity issues. Investigation ongoing

Scope has been narrowed and focus has shifted to building and implementing a potential fix.

Currently we are still seeing a small amount of datastores impacted.

We still advice to not perform any changes or scheduled work with datastores until this incident is resolved.

New update will be approx. 21:30 CEST.

The implementation of the fix has not had expected results and we are moving back the status to investigating and are analyzing the results to find a new solution.

We still advice to not perform any changes or scheduled work with datastores until this incident is resolved.

New update will be approx. 23:00 CEST.

More work on a fix is in progress from findings and is being tested and evaluated at the moment.

For now we need to re-run multiple tests in parallel and will send out a new update once we have a more clear picture of a solution-ETA.

We continue to advice against modifying or doing changes to the datastore configuration(s) at the moment.

Update:
Problem trigged by earlier maintenance requires rebuilding and code-reviews for a stable fix; to avoid issues with changes being done to datastores the management-ui is currently disabled externally.

We will continue working on this and schedule a sit-rep tomorrow at 0900 internally with external resources and re-asses at that time.

Morning update: currently moving forward with 2 parallel possible solutions and are evaluating both in test-environments at the moment.

We will re-assess both options and provide an update at 1200 at the latest.

Mid-day update:
The database-management service has been partially restored however we are still seeing unknown/unstable states and issues connecting to datastores intermittently.

Status changed to Status Disruption and our current focus is restoring services to stable state for all datastores.

We still advice against doing modifications on datastores until incident is set to resolved.

Update:
Issues reaching multiple data-stores has just been resolved by a manual fix and we will continue to monitor it as we are still continuing with the service-disruption on the management-system.

If you still have issues connecting to your datastore we advice you to reach out with as much information as possible in order for us to assist.

Incident remains open and we continue working on the underlying issue with the management-service.

Currently datastores are in a stable state and we are moving down to monitoring in terms of incident-status.

Our 24/7 staffing remains available and developers will work on analyzing logs and results of todays work throughout the night for a handover tomorrow at 0900.

A new update will be provided in the morning with next expected steps and actions to be done in regards to fully stabilize the management-service.

Nightly work continues today with locating exact root-cause and issues with management-service to bring it back to full stable state.

Throughout the day all work will be done in test & stage environments unless a resolution and fix becomes ready for production-deployment in which case a MW or EMW will be communicated.

Root-cause causing management-service issues identified and a fix has been created and passed testing.

Currently we are refining the fix and preparing for an Emergency Window scheduled for 21:00-00:00 tonight.

Tonights Emergency Maintenance Window has been cancelled; final testing-results was not 100% satisfactory and we will continue working and improving the fix overnight and re-assess with the full team tomorrow at 0900.

For now we are reverting back to Monitoring the situation and will continue to ensure datastores remain intact.

Update:
A workaround has been built, tested and confirmed to be ready to deploy for resolving the issue with our management-service for DBaaS. This will stabilize the GUI for the service and restore certain background-tasks that have had issues with scheduling and maintenance-jobs.

A Emergency Maintenance Window will be set and communicated for tonight 20:00-00:00.

After tonight's Emergency Maintenance management-services are restored.

As this was a complicates issue we are maintaining this incident open in monitoring for now as we will continue checking the health and status of services over the weekend.

After the weekend we have reviewed the status and can note several issues still present with the management-service in the UI.

Status of backup-jobs are not showing correctly and some datastores not showing correct status.

We are planning for a emergency-maintenance to re-scale the management-service in the afternoon; will be communicated separately.

We still advice to not manage changes or re-configurations through the UI-management interface for now.

Summary from todays work:
Emergency Maintenance performed between 14:00 - 17:00 resolved our remaining general issues with our management-service and this is stable now since 3 hours back.

A limited amount of datastores are still lacking certain monitoring-features which we will continue to work on and provide further updates tomorrow.

From an incident-perspective we are restoring services to Operational and status Monitoring over night.

Evening update;

We have monitored multiple service-disruptions during the last 12 hours, some with limited impact to datastore-availability (datestore operational but unreachable).

We have performed another Emergency Maintenance today with positive results stabilizing servicesand again noticed limited datastore-service disruptions during the work.

At the moment we are in a stable state from a platform perspective but are still noticing limited and individual datastore disruptions, customers impacted have been proactively contacted with information.

We are still in investigating mode as this is a incident that has been ongoing with active work for 7 days straight and we have not resolved all open issues, our efforts continue to fully stabilize the service.

Status is Partial Service Disruption.
24/7 monitoring is still in effect.

Afternoon update;

We have monitored the situation and cleaned up after the incident as well as restored remaining impacted services.

At the moment we are in a stable state from a platform perspective but still recommend to avoid making changes.

As we have the odd datastore with issues, we will keep status as investigating and as partial service disruption overnight. We continue monitoring the service closely.

Afternoon update:

We have monitored the situation as well as worked on a permanent fix which will be thoroughly tested tomorrow.

The platform is stable, but still reports wrong status for the odd datastore and we recommend to avoid making changes.

We will change the status to monitoring, but partial disruption, as we are in full understanding of the situation and are working on a permanent fix.

Friday afternoon update before the weekend:

We have sustained the extended monitoring and the situation remains stable with an operational platform despite the UI reporting errors in a few cases. Backups have been further cleaned.

We continue to monitor the situation during the weekend and will decide on the exact timing of an emergency maintenance window for a permanent fix early next week.

Next update on Monday.