Extracted Ticket

Ticket Number  : NORDUNETTICKET-1792
Ticket Type    : Unscheduled
Ticket Status  : Closed
Ticket Summary : Link down
Ticket Scope   : 

Ticket Opened  : 20121022 10:28 UTC
Ticket Closed  : 20121112 13:11 UTC

Problem Start  : 20121022 10:24 UTC
Problem End    : 20121023 0:14 UTC

Affected organisations:

      * NORDUnet, NDGF

Service ID:

      * NU-S000084, NU-S000161, NU-S000160, NU-S000083, NU-S000064,


      * Suspected fiber break between Kolding and Odense in Denmark.


      * Two core links in our IP network are currently experiencing
        an outage: Copenhagen - Amsterdam and Copenhagen - London. 
        This is causing traffic
        congestion and packet loss as traffic is re-routed over
        links with lower capacity.
        Additionally one of the two LHCOPN links out of
        NDGF in Copenhagen, to SARA in Amsterdam, is down.
        Lastly, one of NORDUnet\'s two GEANT connections, between
        Copenhagen and Amsterdam, is also down.


      * 20121022 10:44 UTC
        Suspected fiber break.
      * 20121022 11:20 UTC
        Escalated to Fiber supplier.
      * 20121022 12:18 UTC
        To mitigate degraded IP connectivity due to the dual fiber
        break we have applied traffic engineering to offload the 
        affected backbone.
      * 20121022 12:28 UTC
        Corrected description.
      * 20121022 13:06 UTC
        We continue to experience degraded IP connectivity due to a
        double fibre cut in Denmark. Currently most traffic is
        flowing through IP Transit but the large fibre cuts are
        affecting the IP transit providers   as well. We are working
        on several alternative solutions to offload the backbone to
        improve the quality. 
      * 20121022 14:02 UTC
        Updated impact.
      * 20121022 16:39 UTC
        Workaround has been put in place to restore backbone
        capacity and all the ip trafic engineering has been
        restored. The work is continued.
      * 20121022 19:59 UTC
        The fiber provider has confirmed fiber cut. The connection
        has been temporarily moved to a spare fiber and is going to
        be moved back after fiber repair. The fiber repair is going
        to start 23:00 UTC.
      * 20121023 8:54 UTC
        The fibers have been repaired and we have full capacity
        since 2012-10-23 00:14 UTC.
Final ticket report:

      * There was degraded IP connectivity in NORDUnet´s backbone
        due to dual fiber cut in Denmark. The traffic was rerouted
        in the mean time and the fiber was later repaired.