Master case 01294042

CallTower Engineers have been working urgently to restore CallTower’s services. All issues with Auto Attendant and Voicemail functionality have been restored through CommPortal and CallTower Connect. Provisioning of all other services has also been restored in both platforms. We are no longer seeing any issues affecting our CallTower services.

We continue to monitor the data center situation and work closely with our Data Center provider, who is working nonstop to resolve the heating issues. They continue to report that temperatures are gradually stabilizing.

Latest update from our Data Center provider;
There was a recent failure for chillers 2 and 3 due to low suction pressure and extremely cold ambient temperatures. The Equipment has been reset, and the site is recovering. The mechanical vendors arrived on site to assist in troubleshooting and repair. Another mechanical vendor is en route to the IBX. Thirteen portable coolers are now online and two more continue to be installed. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range.

CallTower will be closing this master case and all associated cases. Within three business days, an Official Root Cause Analysis (RCA) or a Preliminary RCA will be available and provided to customers upon request.

CallTower’s Engineering team remains in close contact with our Data Center provider, who continues to make significant progress, and is still seeing temperatures gradually stabilize. Conditions are still not optimal for bringing the data center back online. We are still handling this issue with the utmost priority and appreciate your continued patience.

Latest update from our Data Center provider:
The facility continues to decrease temperatures slowly, with the current average temperature at 83F. For customers who powered down equipment, we strongly recommend waiting to restore operations until the temperatures reach a lower overall range.

Additionally, the CallTower Engineering team is reporting that Commportal is now operational and Connect provisioning should restore within the hour. CallTower understands the importance of these services for our operations, and we appreciate your patience and understanding while we work urgently to provide a resolution.

CallTower has opened a master case to investigate these issues as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com, next update scheduled for 2 hours, unless identified otherwise. This issue is being investigated as top priority.

CallTower’s Engineering team remains in close contact with our Data Center provider, who continues to make significant progress, and is still seeing temperatures gradually stabilize. Current conditions are still not optimal for bringing the data center back online. Rest assured, this matter is being handled with the utmost priority, and our dedicated team is working tirelessly to ensure that all services are up and running smoothly.

Latest update from our Data Center provider;
The facility temperatures continue to decrease slowly with a current average temperature of 88F degrees. 10 portable coolers are now online and five more are being installed. Customers may see their services returning to operation, but the issue remains in-progress. For customers who powered down equipment, site engineers are still strongly recommending waiting to restore

Additionally, the CallTower Engineering team is actively engaged in the restoration process for Commportal and Connect provisioning. CallTower understands the importance of these services for our operations, and we appreciate your patience and understanding while we work urgently to provide a resolution.

CallTower has opened a master case to investigate these issues as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com, next update scheduled for 2 hours, unless identified otherwise. This issue is being investigated as top priority.

CallTower’s Engineering team have remained in close contact with our Data Center provider, have made significant progress, and temperatures are gradually stabilizing. However, the current conditions are still not optimal for bringing the data center back online. Rest assured, this matter is being handled with the utmost priority, and our dedicated team is working tirelessly to ensure that all services are up and running smoothly.

Latest update from our Data Center provider;
The facility temperatures continue to decrease slowly with a current average temperature of 88F degrees. 10 portable coolers are now online and five more are being installed. Customers may see their services returning to operation, but the issue remains in-progress. For customers who powered down equipment, site engineers are still strongly recommending waiting to restore

Additionally, the CallTower Engineering team is actively engaged in the restoration process for Commportal and Connect provisioning. CallTower understands the importance of these services for our operations, and we appreciate your patience and understanding while we work urgently to provide a resolution.

CallTower has opened a master case to investigate these issues as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com, next update scheduled for 2 hours, unless identified otherwise. This issue is being investigated as top priority.

The efforts to address cooling issues in the Chicago datacenter are ongoing. Currently seeing temperatures stabalize, but still too high to bring the datacenter back online. This continues to be handled with the highest priority until all services are up and running.

Latest update from our Data Center provider;
Temperatures are observably decreasing at the site, with average temperatures at 98F, as chillers and cooling solutions are brought online. Customers may see their services returning to operation, but the issue remains in-progress. For customers who powered down equipment, site engineers are strongly recommending waiting to restore operations until the temperatures reach a lower overall range.

We will continue to monitor temperatures at the site, and will provide updates as further information becomes available.

CallTower Engineering, currently continuing to work on the restorations of Commportal and Connect provisioning and will push through the night, currently there is no ETA on resolution.
CallTower has opened a master case to investigate these issues as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com, next update scheduled for 6:00 AM ET unless identified otherwise. This issue is being investigated as top priority.

The efforts to address cooling issues in the Chicago datacenter are ongoing. 2 of 6 chiller units are online and temperatures are improving, but still too high to bring the datacenter back online. This continues to be handled with the highest priority until all services are up and running.
CallTower’s Engineers have been successful in restoring Auto Attendant and Voicemail service functionality to an alternative datacenter. We are continuing to work on the restoration of Commportal and Connect provisioning.

CallTower has opened a master case to investigate these issues as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com approximately every 2 hours, unless identified otherwise. This issue is being investigated as top priority.

CallTower’s Engineering Team remains in close communication with our Data Center provider to restore services to affected platforms, which are critical for reinstating Auto Attendant and Voicemail functionalities. The following update is from our Data Center partner:
Update: No ETR has been provided yet. Temperatures remain elevated at the site. Active cooling is in-place, but we have not seen a noteworthy decrease in temperatures thus far. We continue to monitor and will provide information as it becomes available.

Since CallTower’s pre-configured redundancy was unsuccessful, CallTower Engineers have been actively working with Metaswitch engineering to implement failovers. The system is up and reachable on the network, however, is not processing calls. Currently no ETA on resolution, we are monitoring the situation closely and will keep you informed with any developments. We appreciate your patience during this time and apologize for any inconvenience this may have caused.

CallTower has opened a master case to investigate these issues as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com approximately every 2 hours, unless identified otherwise. This issue is being investigated as top priority.

CallTower continues to work with our Data Center partner to restore services. Due to extreme weather conditions, on-site chillers, which are responsible for keeping the Data Center at an operational temperature, have malfunctioned. While working to resolve the issue with the chillers, additional resources have been acquired to assist with bringing in equipment to reduce the temperature in the Data Center.
Currently there is not an estimated time to resolution. We continue to monitor the situation closely and will keep you informed with any new updates that we receive. We appreciate your patience during this time and apologize for any inconvenience this may have caused.

CallTower has opened a master case to investigate these issues as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com approximately every hour, unless identified otherwise. This issue is being investigated as top priority.

CallTower's Engineering team remains in close communication with our Data Center partner to restore services to affected platforms, which is critical for reinstating Auto Attendant and Voicemail functionalities. Our partner is actively using ambient air techniques to lower temperatures and address the core issues. At present, we have not received an estimated time of resolution from the Data Center provider.
The extreme weather conditions in Chicago were a significant factor in the initial disruption and is still posing challenges to our recovery operations. We are monitoring the situation closely and will keep you informed with any developments. We appreciate your patience during this time and apologize for any inconvenience this may have caused.

CallTower has opened a master case to investigate these issues as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com approximately every hour, unless identified otherwise. This issue is being investigated as top priority.

CallTower continues to work with our Data Center provider on recovering services on the MetaSwitch, to re-enable functionality for AA and VM. Our Data Center provider continues to use ambient air to bring temperatures down in an effort to resolve the issue. Currently there is no ETA from the Data Center provider. The severe weather in Chicago contributed to the initial failure, and it continues to hamper efforts at a resolution.

CallTower has opened a master case to investigate these issues as it has been reported by more than one than on CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com approximately every hour, unless identified otherwise. This issue is being investigated as top priority.

CallTower Engineering continues to work with the Data Center provider on recovering services on the Metaswitch to re-enable functionality for AA and VM. Concurrently, our data center vendor is continuing to work on addressing the cooling issue and has begun using ambient air to try to bring temperatures down.

CallTower has opened a master case to investigate these issues as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com approximately every hour unless identified otherwise. This issue is being investigated as top priority.

CallTower’s Data Center Provider is experiencing an issue which may have an impact on multiple customers services. The below message is the latest update from our Data Center Provider:
We are currently experiencing environmental issues at the site and are engaging all resources to investigate and resolve them.
We will have remote hands investigate your devices as soon as possible, and will keep you updated
There is currently no ETA on resolution of the site issue, but is currently being worked as a top priority with all hands on deck.
Customers may experience:
Post-dial delay.
Voicemail and Auto attendants are currently down, customers may experience, not able to reach or access voicemail services. Auto Attendants could cause service for main lines to drop.
Possible CT Cloud soft phones may drop services.
Provisioning utilizing Connect is currently unavailable.
CallTower has opened a master case to investigate these issues as it has been reported by more then one than on CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltowercom. CallTower intends to post updates on status.calltower.com approximately every hour, unless identified otherwise. This issue is being investigated as top priority.