INAP Outage

INAP has now fully restored their NYJ data center. All services are operating normally. We will continue to monitor and re-issue a status alert if new issues arise.

Latest update from INAP:

Update - The Secaucus data center remains powered down at this time, per the fire marshal.

Evocative onsite management met with the fire marshal and electrical inspectors to review the current state of the facility. The fire marshal has reviewed the current level of progress on the cleaning and requested additional remediation actions around the fire system components. The follow up inspection from the fire marshal is 9:00 AM EDT, Wednesday. The Evocative team is bringing on additional personnel and vendors to comply completely with the additional requests by this deadline.

In preparation for being able to allow clients onsite, the fire marshal has stated that Evocative must perform a full test of the fire/life safety systems. This must be performed after utility power has been restored and fire system components replaced. Vendors have already been scheduled and are standing by to complete this work tomorrow.

If there are no significant changes in the timeline, the earliest INAP will be allowed back into the site to power up servers will be later in the day on Wednesday. We anticipate the next update of significance will be provided by Evocative Wednesday morning, and will keep you updated as information becomes available.
Jul 11, 2023 - 16:04 CDT

Latest update from INAP:

Update - Dear Valued Customer,

Evocative has worked with their remediation vendor through the night to clean the UPS' at the request of the fire marshal. They have made significant progress and aim to have the cleaning completed by mid-day. There is currently a 2PM EDT meeting scheduled, during which Evocative staff and local township officials will perform a walk-through of the site. Evocative will need approvals from the Mayor, the fire marshal, and the local utility before the site can be re-energized. Depending on their inspections and approvals, Evocative staff will then be able to access the site and initiate the sequence to bring all data center systems back online. Evocative will be providing us a status update after this meeting takes place. While we do not have an ETA for when power will be fully restored, the re-energizing process can take 4-5 hours after it is initiated.

Regards,
INAP Cloud Support
Jul 11, 2023 - 11:35 CDT

Latest update from INAP:

Update - Dear Valued Customer,

As we reported earlier, the incident is related to a UPS system in the Secaucus data center owned and operated by Evocative. There was a fire in the UPS room, which was extinguished, but the incident resulted in a shutdown of all power to the facility and the involvement of the local fire department. This is the reason that redundant power systems did not keep any of the servers up and operational.

The fire marshal is requiring extensive cleaning of the affected systems and areas before the site can be re-energized. That work is in progress. We were informed that Evocative will need to get an OK from the Mayor, the fire marshal, and the local utility to re-energize the site; depending on their inspection and approval, they can enter the site and begin to sequence bringing all data center systems back online. There is currently no ETA, but we will continue to provide updates.

Once that is done, we have our teams standing by both locally and within our engineering and support organizations to ensure that all servers and services are brought back online as swiftly as possible.

Regards,
INAP Cloud Support
Jul 11, 2023 - 08:06 CDT

Further updates from INAP below. It is looking likely that there will be no resolution before 08:00 EDT tomorrow.

Update - Our provider has shared further details about the incident and current status.

A small electrical failure occurred with one of the redundant UPS devices which began smoking and led to a small fire in the UPS room. The fire department was dispatched and the fire extinguished quickly. The fire department subsequently cut power to the datacenter and disabled the generators while they and the utility company inspected the electrical system.

The fire marshal is requiring extensive cleaning of the UPS devices and rooms before the site can be reenergized. There is a vendor onsite currently who will be performing the cleanup.

The next update from our provider will be at 8:00 AM EDT unless there are significant changes overnight. We will continue to pass along information as we receive it.
Jul 10, 2023 - 22:21 CDT

See below for an additional update from INAP.


Update - Dear Valued Customer,

We have spoken with our provider and received additional details and a tentative ETA.

There was a problem within the UPS system that required the site to be fully powered down. Vendors are onsite and working to resolve the issue.

The current estimate is that power should be restored within the next several hours. We will continue to work with our provider and post regular updates.

Regards,
INAP Cloud Support
Jul 10, 2023 - 18:59 CDT

From INAP

Update - Dear Valued Customer,

We are seeing intermittent power availability issues within this datacenter and are working diligently with the provider on a resolution.

Regards,
INAP Cloud Support
Jul 10, 2023 - 18:02 CDT

INAP is experiencing network connectivity issues in their NYJ data centers which may be impacting ThreatX hosted sensors in that region. Please note that this is only impacting INAP sensors in NYJ, AWS sensors are not impacted. We will update this page as more information becomes available.