Extracted Ticket


Ticket Number  : NORDUNETTICKET-6787
Ticket Type    : Unscheduled
Ticket Status  : Closed
Ticket Summary : Routing issue with router fi-csc
Ticket Scope   : 

Ticket Opened  : 20200927 18:46 UTC
Ticket Closed  : 20201030 6:13 UTC

Problem Start  : 20200927 8:00 UTC
Problem End    : 20200927 20:25 UTC

Affected organisations:

      * NORDUnet, FUNET
        ----

Service ID:

      * NU-S000563
        ----

Description:

      * We are currently experiencing routing problems with the
        router fi-csc2
        ----

Impact:

      * Routing issues for customers connected to FI-CSC2
        ----

Update:

      * 20200927 19:01 UTC
        Router is being restarted.
        ----
        
      * 20200927 20:25 UTC
        Card in router in Hamburg was restarted and traffic should
        be back to normal. 
        RFO will be investigated by vendor.
        ----
        
      * 20201009 16:45 UTC
        Juniper:
        
        I could correlate the behavior to an incorrectly programmed
        next-hop in the pfe, but I cannot confirm as the device is
        not in problem state anymore and there is not enough data to
        draw conclusions.
        The troubleshooting performed by you, which is restarting
        the FPC on de-hmb, also forced the link down on fi-csc, so
        when it went back up the next-hop id in the PFE was updated
        and restored.
        
        Further, could you provide a fresh set of the following
        commands:
        >show route
        >show route forwarding-table destination
        As it's not clear from the case description when they were
        previously collected (before/after/during the event)
        
        Also, do you happen to have any traffic graph from et-0/1/5
        on fi-csc, as it would be useful to check 
        ----
        
      * 20201009 17:06 UTC
        The graphs are sent. 'show route' is a huge output, asked if
        he wants something particular that we can send. The other
        command requires also a prefix as destination.
        ----
        
      * 20201014 16:26 UTC
        Sent 'show route' outputs.
        ----
        
      * 20201020 10:01 UTC
        Sent more output.
        ----
        
      * 20201023 13:21 UTC
        Juniper:
        
        "Thank you for providing the requested output. I checked and
        everything is looking fine. As the device is not in problem
        state anymore and the syslog of the FPC doesn't cover the
        event, a clear conclusion to what triggered the problem
        cannot be drawn. The troubleshooting performed by you, which
        was restarting the FPC, forced the link down and restored
        the proper functionality of the router. Should the same
        thing happen again, please let us know immediately, in order
        to find the root cause of the event."
        ----
        
        
Final ticket report:

      * We had routing problems with the router in Helsinki. Solved
        by restart of FPC.
        ----