Campus Network Connectivity Issues - Tuesday 5/20 PM

OIT Network Engineering is investigating network connectivity issues affecting parts of campus including, but not necessarily limited to, Groseclose, ISyE Annex, Montgomery Knight, Student Center, Library (Price-Gilbert, Crosland, Clough), and others in the southern parts of campus. This would cause limited or no connectivity for both wired and wireless networks in affected buildings. Engineers are actively working to address the issue.

The cause of the network connectivity issues was identified and isolated as of approximately 1:50pm. Normal connectivity should now be available in all affected locations, except for the Groseclose Data Center which remains offline at this time.

The larger impact to the campus network has been resolved, but some ISyE and IPAT services hosted from the data center in the Groseclose building are still unavailable. Engineers are working to restore services at this time.

We are currently waiting for a vendor's engineering support team to assist OIT, ISyE support, and IPAT support on fixing the problems with networking that are still affecting ISyE and IPAT services hosted in the Groseclose building.
We will post further updates as we get them and appreciate your patience as we work through this issue.

Services currently down for COE include the following:
ISyE:
- HTC Condor compute
- Any compute relying on OpenStack, including GPU servers.
MSE:
- Containers running on OpenShift.
IPaT:
- All VMs running on OpenShift.

Services relying on openshift and openstack continue to be unavailable, but the vendor's engineering team is continuing to work the issue and is working on an action plan to help recover services

RedHat support is running the upgrade again after making changes to bypass certain checks, since the system was broken. 

OIT and Red Hat support staff have completed one stage of the update, and now the server VMs and other resources are available to end users. While some components are functioning, OIT and Red Hat support will continue the upgrade process. Although the OpenShift console and VMs are currently accessible, they may experience disruptions during the next stages of the upgrade.

OIT and Red Hat successfully upgraded to version 17.1; however, we still need to upgrade the underlying operating system from RHEL 8.4 to 9.x. Currently, all OpenStack and OpenShift services are operational, and the campus network is stable. We have nine days remaining to complete everything before the AUS repositories are required.