Extracted Ticket


Ticket Number  : SUNETTICKET-589
Ticket Type    : Unscheduled
Ticket Status  : Closed
Ticket Summary : Short outage HV
Ticket Scope   : 

Ticket Opened  : 20100227 12:12 UTC
Ticket Closed  : 20100315 11:05 UTC

Problem Start  : 20100227 12:06 UTC
Problem End    : 20100311 20:11 UTC

Affected organisations:

      * HV
        ----

Description:

      * Outage for HV on the green network.
        ----

Impact:

      * No redundancy during the outage
        ----

Update:

      * 20100227 12:19 UTC
        Looks like a hardware failure that caused a board to
        restart, escalating to supplier.
        ----
        
      * 20100228 12:52 UTC
        Another restart caused one more outage this morning.
        ----
        
      * 20100301 12:49 UTC
        IPnett will change the M6-Main card in the CN4200 tomorrow
        at 18:00 UTC.
        This will be traffic affecting.
        ----
        
      * 20100301 15:23 UTC
        There has been a problem rounding up spare-parts so the card
        will be swapped on 3/3 16:00 UTC
        ----
        
      * 20100303 16:05 UTC
        There has been some problem in delivery of spare-cards,
        hence the maintenance is postponed for a day.
        Maintenance will be carried out tomorrow 4/3 at 18:00 UTC.
        ----
        
      * 20100303 20:45 UTC
        Maintenance will be carried out tomorrow 4/3 at 16:00 UTC
        ----
        
      * 20100304 16:01 UTC
        The card replacements will start now.
        ----
        
      * 20100304 16:38 UTC
        There has been some problems with the swap of boards, after
        the board and flash was swapped the node became
        unmanageable. The node will has to be power cycled now in
        attempt to restore the service.
        ----
        
      * 20100304 17:25 UTC
        Technician on site is still working on the problem, the node
        is still unreachable. 
        ----
        
      * 20100304 18:51 UTC
        During the swap of boards the flash for some reason became
        corrupt so it had to be reinstalled. Unfortunately the now
        working flash contains too old software so a software swap
        is not possible. New old software needs to be uploaded to
        the node first in order to perform a SW swap and then
        download the new software again.
        ----
        
      * 20100304 19:50 UTC
        The new software was installed and all services was
        restored.
        ----
        
      * 20100304 19:55 UTC
        Now all that remains is to upload the latest software and
        switch again. This will cause one more hit in traffic.
        ----
        
      * 20100304 21:12 UTC
        The node is still running a bit old software but on a new
        OAV Main card.
        ----
        
      * 20100305 12:06 UTC
        Currently the node is running on version 4.2.3 because there
        was some problems when trying to upgrade the software to
        6.0.1. IPnett will take this up with Ciena.
        ----
        
      * 20100308 12:35 UTC
        A new intervention will take place on 11/3 16:00 UTC
        ----
        
      * 20100311 16:20 UTC
        Currently software upload is being done on the node.
        ----
        
      * 20100311 20:38 UTC
        The connection to HV is restored, and the node is now
        running on version 6.01. The node will be further upgraded
        to version 7.02 and there might be few more disruptions
        during reboot of the node.
        ----
        
      * 20100311 21:07 UTC
        The node is now successfully upgraded to 7.0.2.
        ----
        
        
Final ticket report:

      * The outage was due to hardware failure and the OAV card in
        main slot was replaced and the node is now upgraded to
        version 7.0.2.
        ----