Lumen Fiber Maintenance extended

This incident has been resolved.

Lumen splicers have rolled services back to the old fiber span due to the replacement having damage, thus restoring services to a stable state. I have 1 site that is still not up (Craig R3). I am working with Lumen on verifying stability.

MOREnet NOC

Lumen continues to splice fiber. Some sites are seeing connection restoration. No estimated time for them to be fully hands off.

MOREnet NOC

Quite a few more sites in Lumens network have also gone down. I suspect this is correlated, and have opened a ticket to confirm. This brings the outage list to the following sites:

Appleton City R-II
Boonslick Regional Library - Cole Camp Branch
Calhoun R-VIII
Cass County Library - Pleasant Hill Branch
Chilhowee R-IV
Cole Camp R-I
Craig R-III
Crawford County Library District - Steelville Branch
Green Ridge R-VIII
Hardin-Central C-2 (High School)
Hudson R-IX
Kingsville R-I
Leesville R-IX
Mirabile C-1
Montrose R-XIV
Oak Hill R-I
Shawnee R-III
Smithton R-VI
Trails Regional Library - Odessa Branch
Sheldon R-VIII
Christian County Library District - Ozark Branch
Northeast Vernon County R-I (High School)
St. Clair County Library
Blue Eye R-V
Thomasville Library - Oregon County Library District
Carter County Library
Shell Knob Branch Barry-Lawrence Regional Library
Lakeland R-III
Aurora Branch Barry-Lawrence Regional Library
Main Branch - Stone County Library
Shell Knob 78
Blue Eye Branch - Stone County Library
Eagle Rock Branch Barry-Lawrence Regional Library
Mt Vernon Branch Barry-Lawrence Regional Library

Some of these sites may be up temporarily, but will go back down as Lumen diagnoses and re-builds the correct fiber ends. I will remove sites that are confirmed hands off when I have that information. Updates to follow

MOREnet NOC

Some services have been restored, but we still have a lot of circuits down. Lumen NOC, Field Operations and Tier III in collaboration with OSPE are testing to identify the correct fibers that need to be spliced and respliced to restore all services.

The new 432-count cable needs to be completely replaced. The Outside Plant team located slack on the original 144-count cable, and is working to temporarily revert traffic back until permanent repairs can be made.

Fibers are spliced back to the original 144-count cable on the south end, and the splicer is enroute to the north end to continue restoration efforts

The Lumen NOC has advised alarms have not cleared. Splicers have advised so far they have found bad or damaged fibers on the 432-count fiber which was corrected by rolling to spare fibers. As well as mismatched fibers on a 48-count fiber cable in a hand hole which were corrected. The Lumen NOC is still showing alarms and low light indications. Splicers are continuing investigations.

More updates to follow.

MOREnet NOC

Repair on the 432-count cable continues. They are also attempting to migrate services off this path to restore the connection sooner. Updates will follow.

MOREnet NOC

Fiber crews are continuing their work. No ETR at this time.

MOREnet NOC

Lumen has ran into fiber slack issues during their overnight maintenance window. They are extending the maintenance to pull a new line. This is affecting quite a few Lumen sites.
 
Appleton City R-II
Boonslick Regional Library - Cole Camp Branch
Calhoun R-VIII
Cass County Library - Pleasant Hill Branch
Chilhowee R-IV
Cole Camp R-I
Craig R-III
Crawford County Library District - Steelville Branch
Green Ridge R-VIII
Hardin-Central C-2 (High School)
Hudson R-IX
Kingsville R-I
Leesville R-IX
Mirabile C-1
Montrose R-XIV
Oak Hill R-I
Shawnee R-III
Smithton R-VI
Trails Regional Library - Odessa Branch

No ETR at this time. Updates will follow.

MOREnet NOC