We received the following update at 05:53am from Lumen. The link has been stable since 05:02am and Lumen transit has been put back into the topology.
" Event Conclusion Summary
Start Time: January 01, 2022, at 23:30 GMT Stop Time: January 02, 2022, at 03:30 GMT
Root Cause: A card issue in London UK impacted unprotected service. Fix Action: Field Operations replaced the affected card, thus restoring services to a stable state. "
Posted Jan 02, 2022 - 10:46 GMT
Update
We've received the following update from Lumen.
"The NOC reports that the card has been replaced and they are waiting for it to boot at this time."
Posted Jan 02, 2022 - 04:22 GMT
Update
We've received the following update from Lumen.
"Field Operations has arrived on-site and is working to replace the amplifier at this time."
Posted Jan 02, 2022 - 03:27 GMT
Update
We've received the following update from Lumen.
" Field Operations rebooted the affected card to no avail. The card was also replaced with another one at the site however services did not restore.
Next update by: 2022-01-02 03:30 GMT "
Posted Jan 02, 2022 - 03:22 GMT
Identified
We've received the following update from Lumen.
2022-01-02 01:29:16 GMT - Field Operations arrived on-site and reseated the affected card to no avail. Efforts to reboot the card are ongoing.
2022-01-02 00:30:51 GMT - On January 01, 2022, at 23:30 GMT, Lumen identified a service impact in London UK. As this network fault is impacting multiple clients, the event has increased visibility with Lumen leadership. As such, client trouble tickets associated to this fault have been automatically escalated to a higher priority.
The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to a live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post-incident review, details may evolve.
Posted Jan 02, 2022 - 02:17 GMT
Investigating
We are observing link flaps on Lumen transit in Welwyn Garden City Data Center and have taken the transit out of the topology to avoid any further traffic disruption. We are in contact with Lumen technical support to get it looked into. Updates to follow.
Posted Jan 02, 2022 - 01:02 GMT
This incident affected: Networking (IP Transit) and Facilities (DC2 - Welwyn Garden City).