The issue has been resolved and a RFO will be released in the upcoming days.
Posted Apr 03, 2024 - 03:38 UTC
Update
It appears that 80% of the racks within the HEL1 region are supplied with connectivity, awaiting for the rest.
Posted Apr 03, 2024 - 02:53 UTC
Update
We observe connectivity restored for the first customers, namely colocation. The onsite personel of the datacenter operator is still rewiring the new equipment with the connections of the broken one.
Posted Apr 03, 2024 - 02:44 UTC
Update
Configuration has been restored, awaiting reconnection from the onsite personel of the datacenter operator.
Posted Apr 03, 2024 - 02:21 UTC
Update
Both devices are hardware dead at once, restoring firmware doesnt help. We suspect the flash might be broken and have died during the last commit. As the Finland location is quite distant to our office address and so to say spare-part storage, while it is quite unusual to store more than a redundant pair on-site, we're currently waiting for urgent hardware replacement. This is in progress and we're currently preparing the setup, expected ETA is 1-2 hours.
Connectivity will be in the first stage exclusively done over our Equinix FR7 (FFM5) points of presence, where we got transport connectivity from HEL1. Router configuration at FFM5 is currently being prepared. Please bear with us, we're doing our utmost to restore network connectivity within this pop.
Posted Apr 03, 2024 - 01:52 UTC
Update
We were finally able to gain serial console access. Investigation takes some time due to distance and the dependency on the datacenter operator. One of the switches, the previous primary one, appears to have a faulty hardware component hindering it from booting. The other one we're currently investigating.
Posted Apr 03, 2024 - 01:17 UTC
Update
It appears that one member of the routers is unable to boot anymore. We're currently investigating rescue access to stabilize the situation.
Posted Apr 03, 2024 - 00:00 UTC
Identified
We're currently facing a outage of the core routing stack in the Helsinki, Finland region. Following a previous outage of Elisa Oyj, we have turned back BGP-Sessions and the equipment went unresponsive while doing so. We're currently waiting for hands-on to carry out a power cycle.
Posted Apr 02, 2024 - 23:37 UTC
This incident affected: Network Infrastructure (Core Network).