Intermittent Service Degradation for vCloud Director User Interface

This incident has been resolved.

Servers Australia Engineers have identified that specific database queries executed by the vCloud Director system during inventory collection were causing the sync status between vCloud director and vCenter to enter an invalid state, causing the vCenter to continually try to resync and never complete. This caused specific objects from vCenter to be unable to be collected into the VCD database, as well as affecting creating of new VMs while the sync status was not healthy. This issue was traced back to a bug in the error handling code within the vCloud Director application itself, which made the root cause difficult to pinpoint.

To address this, we have implemented changes to mitigate the issue and restore functionality. We will continue to monitor the situation closely to ensure the stability of the system before formally closing the status.

We appreciate your patience and will provide further updates as needed.

Thank you,
SAU Engineering Team

We are continuing to work with the Vendor on this issue.

As part of the process, we need to export a log bundle from the vCloud Director server.

Doing this may cause the VDC interface to become unresponsive while the log is generated.

There will be no impact to customer workloads while this takes place.

The VCD platform is back online. We are continuing to work with the software vendor to resolve intermittent issues in the UI. We will keep this status updated as work progresses and with further maintenance.

The VCD user interface will be temporarily unavailable while we perform emergency maintenance and updates. Engineers will update this status once complete.

We are once again investigating an issue with vCloud Director that causes some operations within the user interface to fail.

Our Systems Engineering and Networking teams are actively investigating this issue in collaboration with the software vendor and their support teams, maintaining constant communication throughout the process.

We will provide updates as they become available.

Maintenance is complete and all services are back online. We will continue monitoring and will update this status further if there are any more changes.

Engineers have found that certain API calls to the backend from the vCloud Director interface are timing out. Emergency maintenance will start shortly and the vCloud Director user interface will be unavailable during this time. Estimated time is 30 minutes.

Our Systems Engineers have rectified issues that were causing certain operations to fail within the VCD platform.

We have tested functionality and can no longer replicate previous issues seen. Our Engineers will continue to monitor the VCD platform throughout the night for any irregularities.

Please reach out to our support team if you encounter any issues with your VDC or Private Cloud environment.

We have identified an issue with the hardware hosting the vCenter appliance. The appliance is being moved to a new host and will be unavailable for approximately 10 minutes. This will only impact the availability of the vCenter WebUI and not customer workloads.

We are once again investigating an issue with vCloud Director that causes some operations within the user interface to fail.

We will provide updates as they become available.

After a reboot of the VCD environment, operations have returned to normal.

We are monitoring the services and will update as needed.

We are currently investigating an intermittent issue with vCloud Director that causes some operations within the vCloud Director user interface to fail.

We need to restart the vCenter Management Server and the vCloud Director Server, There will be no customer workload impact, but the VCS and VCD User Interfaces will be unavailable for 15 minutes.

This work is commencing now.

After a reboot of the VCS environment, operations have returned to normal.

We are monitoring the services and will update as needed.

As part of the normal troubleshooting procedure, we need to restart the vCenter Management Server.

There will be no customer workload impact, but the vCenter UI will be unavailable for approximately 10 minutes.

This work is being performed at 16:25 AEDT.

We are currently investigating an intermittent issue with vCloud Director that causes some operations within the user interface to fail.

We will provide updates as they become available.