CEDAR Performance Degraded

The Georgia Tech team identified a high-priority dataset located within the cedar_cos volume that is being retrieved at very slow rates (reported at approximately 5–10 MB/s), delaying access to important research and operational data. Our vendor has been actively investigating the issue, reviewing audit logs, analyzing S3 GET activity, and exploring potential remediation options. They have determined that the requested data cannot easily be targeted at the individual directory level because the underlying StorageGRID object mapping is performed at the volume level rather than the client-share folder level. Our vendor is considering a temporary maintenance strategy to reduce non-essential retrieval activity and prioritize access to the critical dataset while longer-term engineering fixes continue to be evaluated. A meeting is scheduled between the vendor and Georgia Tech stakeholders to discuss mitigation options and next steps.

**What’s happening?**
CEDAR storage has been made unavailable starting today, July 9, at 3.20pm ET. At this time, there is not an estimated day/time for when services will be returned. Emergency work is necessary to mitigate slow data retrieval times.

**When is it happening?**
CEDAR storage became inaccessible starting today, July 9, at 3.20pm ET.

**Why is it happening? **
The Georgia Tech team identified a high-priority dataset located within the cedar_cos volume that is being retrieved at very slow rates (reported at approximately 5–10 MB/s), delaying access to important research and operational data. Our vendor has been actively investigating the issue, reviewing audit logs, analyzing S3 GET activity, and exploring potential remediation options. The vendor determined that the requested data cannot easily be targeted at the individual directory level because the underlying StorageGRID object mapping is performed at the volume level rather than the client-share folder level. Our vendor has proposed a temporary maintenance strategy to reduce non-essential retrieval activity and prioritize access to the critical dataset while longer-term engineering fixes continue to be evaluated.

**Who is affected?**
All CEDAR users.

**What do you need to do?**
Access and any data connections to/from CEDAR was automatically interrupted and cancelled at 3.20pm. At this time, users do not need to take action.

**Who should you contact for questions?**
For any questions, please contact pace-support@oit.gatech.edu. More information will be provided as it is available.

Troubleshooting work with storage vendor continues. Core engineering resources are being engaged.

Troubleshooting will continue tomorrow morning. Vendor is analyzing logs and will provide additional updates as we get them.

We finally settled on the best performance parameters to transfer a critical file set from CEDAR. NetApp engineering continues root cause analysis and are working on a patch to fix the performance issues. Storage reamins closed to all users while the critical transfer completes. Issue has been escalated to the executive VP for support and engineering.

Critical file sets are being transferred, while NetApp continues working on permanent fixes that will allow us to open the file system with the expected transfer rates. GT directors will meet with the NetApp VP of support and engineering in the next couple of hours. Another status call with NetApp is scheduled for 4pm.

NetApp engineering updated an executable on one of the storage nodes. Results are promising and more analysis is needed. In the meantime, the critical data transfer continues.