FFM2 Points of Presence - Outage
Incident Report for aurologic GmbH

On 07th July 2024, we got alerted at 01:14 am (CEST) about down bgp-sessions at FFM2 (Interwerk). Thorough analysis suggested either a power or network connectivity issue with the equipment within this points of presence.

Following quick analysis, we were able to pinpoint the exact root cause onto a outage of darkfiber connectivity towards Interxion, impacting multiple operators at once. Unfortunately, the outage also took down our out-of-band access, which is provided over another network operator being also affected.

Together with datacenter on-site personnel, we were able to gain access to our equipment and restored connectivity over a adhoc deployed backup link, allowing to take other operators at this site live again.

According to the datacenter operator, the reason for this outage was a human error, causing darkfiber crossconnect connectivity to be terminated at Interxion, following a inventory clean-up of the cabinet operator. This event was out of our control and caused upstream.

Posted Jul 12, 2024 - 20:08 UTC

After extensive effort, connectivity is currently restored over a alternative route onto Interxion core02.ffm3.de router. We will release a Post-Mortem within the next business days.
Posted Jul 07, 2024 - 03:35 UTC
It appears there is a partial outage on the fiber connections between FFM2 and FFM3, we're currently working with the DC operator to restore operation.
Posted Jul 07, 2024 - 00:22 UTC
We're currently observing a outage at the FFM2 Points of Presence. At the moment much is not known yet, but it seems we're not the only operator being affected at the site. Datacenter operator has been informed and on-site personnel is on the way. We'll post news when they become available.
Posted Jul 06, 2024 - 23:49 UTC
This incident affected: Network Infrastructure (Core Network).