Confirmed, traffic has been switched back and we see all our interfaces in Frankfurt operating below 75% utilization.
Posted Aug 25, 2024 - 13:04 UTC
Update
We see our interfaces between Digitalrealty FRA16 and Tornado Datacenter FRA01 in a working state since about a minute. Signal strength appears to have improved as well. Waiting for confirmation of the supplier, before we take them live, in order to avoid unwanted connectivity loss.
Posted Aug 25, 2024 - 12:25 UTC
Update
The supplier reports:
"Our fiber team is working on restoration, works are still ongoing, but many fibers are restored.
Please check status of your circuit, bounce your ports and reply to us if you still see issues on it."
We dont see our route to Digitalrealty yet online, DWDM equipment reports zero signal and according to OTDR still ending shortly after FRA16. We'll possibly shift some traffic around to keep other interconnection links within regular levels.
Posted Aug 25, 2024 - 11:37 UTC
Update
So far it looks good, the darkfiber route between Digitalrealty FRA16 and Equinix FR7 was a bit loaded, causing monitoring alarms on defined set points. Traffic got shifted precautionary to preserve spare bandwidth, capacity was still within the regular expected levels.
At Equinix, we had a bottleneck on router core01.ffm5.de ingress, causing 96% link utilization. This has been upgraded swiftly by another 200Gbps, resolving near 100% usage. Momentarily, we switched some traffic off RETN onto Gcore, while we did the upgrade.
Maintenance works are in progress at Weismuellerstrasse, Frankfurt East, nearby Digitalrealty FRA16. Talking to the fiber crew, they said it will take eventually longer - Monday is not impossible at the moment. There's quite some people working to replace the splice enclosure. We'll keep this ticket updated up on receiving updates from our darkfiber supplier.
TL;DR; busy morning, however better than expected and under active observation.
Posted Aug 24, 2024 - 11:21 UTC
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Aug 24, 2024 - 06:00 UTC
Update
Precautionary, traffic has been routed off the darkfiber line between Digitalrealty FRA16 (FFM3) and Tornado Datacenter FRA01 (FFM1). Capacity between Digitalrealty FRA16 (FFM3) and Equinix FR7 (FFM5) is within normal levels and we do not expect our spare capacity on that darkfiber to be utilized. In case you're observing any unexpected issues, please let us know.
Posted Aug 23, 2024 - 23:17 UTC
Scheduled
On 24/08/2024 between 06:00 and 25/08/2024 18:00 UTC, the darkfiber supplier will perform scheduled maintenance on one dark fiber route between Tornado Datacenter (FFM1) and Digitalrealty (ex. Interxion) FRA16 due to "Replacement damaged splice enclosure in Frankfurt. The joint is currently embargoed and not in the best condition, to keep our high network standard and avoid any outages, we need to replace it.". Traffic will be shifted beforehand on other links, we do not faciliate any customer impact with the said works.
We are aware, this is a longer going outage of that fiber route. Following our redundancy measures and proper capacity planning in place, we do not faciliate impact to our customers. Traffic will continue over alternative routes, primarily Equinix FR7.
Posted Jul 18, 2024 - 09:44 UTC
This scheduled maintenance affected: Network Infrastructure (Core Network, Interconnection).