Network Service Issue (Equinix Fabric)

Equinix SMC monitored that traffic on CSP ports started to recover at 17:00hrs UTC on 15-Jan. Please check the status of your connections and if you still encounter any issue, please open a ticket with Equinix Service Desk or update your existing ticket..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

Equinix SMC monitored that traffic on CSP ports started to recover at 17:00hrs UTC on 15-Jan. Please check the status of your connections and if you still encounter any issue, please open a ticket with Equinix Service Desk or update your existing ticket..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

Equinix IBX site staff reports that five chillers remain stable and colocation temperatures have returned within SLA. IBX Critical Facility Engineers will continue to monitor the chiller plant and the IBX colocation temperatures for stability. The 6th chiller will require parts for full repair, and parts are expected to arrive on site tomorrow afternoon 16-JAN-2024. IBX Site Staff is making preparations to normalize IBX Operations by closing exterior doors, which will allow for humidity levels to normalize. IBX site staff is also gradually turning off the portable coolers, all of which will remain onsite and ready to deploy if needed. Should you require any assistance, our teams are standing by to support you via complimentary Smart Hands services..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

Equinix IBX Site Staff reports that the 15th and final portable cooler has been installed to distribute cool air evenly. The site continues to recover and the average site temperatures have remained stable at 85° F. IBX Critical Facility Engineers along with onsite mechanical vendors continue working to restore the 5th chiller. Note that temperatures may vary at different locations across the site. For customers who powered down equipment, we strongly recommend waiting to restore operations until the average site temperatures reach around 80 °F..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

Please disregard the previous message that indicated the clearance for customers to resume turning up equipment. The situation is still under investigation, and our teams are actively working on resolving the issue to ensure a complete remediation.

We apologize for any confusion or inconvenience this may have caused. We will provide updated and accurate information once the issue is fully addressed..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

Operating temperatures at CH1 have returned within SLA. Resiliency to the chiller plant has been restored and our customers are clear to begin turning up any equipment that was previously shut down. Should you require any assistance our site teams are standing by to support via complimentary Smart Hands services. Equinix site staff and engineers are conducting a root cause analysis to understand the issue and ensure full remediation..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

Equinix IBX Site Staff reports that the site is still recovering and the average site temperatures have reduced an additional 2 degrees since the last update to 85° F. IBX Critical Facility Engineers along with onsite mechanical vendors continue working to restore the 5th Chiller. Note that temperatures may vary at different locations across the site. Fourteen portable coolers are online and the final portable cooler install is pending electrical wiring. For customers who powered down equipment, we strongly recommend waiting to restore operations until the average site temperatures reach around 80 °F..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

Equinix IBX Site Staff reports that the site is still recovering and the average site temperatures have decreased to 88° F. IBX Critical Facility Engineers along with onsite mechanical vendors continue working to restore the 5th Chiller. Note that temperatures may vary at different locations across the site. Fourteen portable coolers are online and the final portable cooler install is pending electrical wiring. For customers who powered down equipment, we strongly recommend waiting to restore operations until the average site temperatures reach around 80 °F..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

null.
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

The latest update from the IBX Team:

Equinix IBX Site Staff reports that one of the mechanical vendors arrived on site to assist in troubleshooting and repair. Another mechanical vendor is en route to the IBX. The site is still recovering and the average temperature remains stable at 91° F. Thirteen portable coolers are online and two more continue to be installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

The latest update from IBX Team:

Equinix IBX Site Staff reports that there was a subsequent failure to Chiller 2 which has been reset. Chiller 2 and the previous failure on Chiller 3 were caused by low suction pressure due to the extremely cold ambient temperatures. Equipment has been reset and the site is recovering but the average site temperature has risen to 91° F. The estimated time for the mechanical vendor to arrive on site to assist in troubleshooting and repair is at 09:30 Site Local Time. Thirteen portable coolers are now online and two more continue to be installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

The latests update from the IBX Team:

Equinix IBX Site Staff reports that due to a temporary issue with Chiller 3, a slight rise in the colo temperature occurred. Chiller 3 has been reset and the site is recovering. The average temperature is currently 86F degrees. Thirteen portable coolers are now online and two more continue to be installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

The latest update from the IBX Team:

Equinix IBX Site Staff reports that colocation temperatures continue to decrease slowly with a current average temperature at 83F degrees. Twelve portable coolers are now online and three more continue to be installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range. .
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

The latest update from the IBX Team:

Equinix IBX Site Staff reports that colocation temperatures continue to decrease slowly with a current average temperature at 85F degrees. Twelve portable coolers are now online and three more are being installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range. .
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

The latest update from the IBX Team:

Equinix IBX Site Staff reports that colocation temperatures continue to remain stable with a current average temperature remaining at 88F degrees. 10 portable coolers are now online and five more are being installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range. .
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

The latest update from the IBX Team:

Equinix IBX Site Staff reports that colocation temperatures continue to decrease slowly with a current average temperature of 88F degrees. 10 portable coolers are now online and five more are being installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range. .
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

Equinix SMC received below update from onsite team.

Equinix IBX Site Staff reports that colocation temperatures continue to remain stable with a current average temperature remaining at 95F degrees. Four portable coolers are now online and eleven more are being installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range. .
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

Equinix SMC reported that CSP ports in CH1 and CH2 went down due to ongoing chiller issue in CH1 data center. Please find below update from onsite team:
Equinix IBX Site Staff reports that colocation temperatures continue to decrease slowly with a current average temperature of 95F degrees. Three portable coolers are now online and twelve more are being installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range..
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

The issue has been identified and a fix is being implemented.
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]

We are currently investigating this issue.
jvtlfjw3h0y5
Incident Number: INC0026009[Equinix Fabric - Americas (CH)]