Extracted Ticket


Ticket Number  : SUNETTICKET-6078
Ticket Type    : Unscheduled
Ticket Status  : Closed
Ticket Summary : IAAS hosts down
Ticket Scope   : Software

Ticket Opened  : 20191223 13:19 UTC
Ticket Closed  : 20200102 6:55 UTC

Problem Start  : 20191223 13:00 UTC
Problem End    : 20191223 16:29 UTC

Affected organisations:

      * SUNET
        ----

Description:

      * Multiple hosts in IAAS is down including
        v-mailfilter3.sunet.se.
        ----

Impact:

      * Various SUNET internal and test services including
        redundancy to e-mailfilter is affected.
        ----

Update:

      * 20191223 13:35 UTC
        Our IAAS provider is investigating.
        ----
        
      * 20191223 14:04 UTC
        Some more servers with critical services have gone down. It
        was reported to the IAAS provider for urgent investigation.
        ----
        
      * 20191223 15:33 UTC
        Presently all the hosts except docker-prod-1.sunet.se is up.
        ----
        
      * 20191223 15:40 UTC
        Provider has ongoing issues network services in our
        Stockholm datacenter (sto1). The controller nodes for the
        network services are back, and other services should be back
        shortly. They will continue investigating the contributing
        factors for the incident.
        ----
        
      * 20191227 8:01 UTC
        All the hosts are stable.
        ----
        
        
Final ticket report:

      * Multiple hosts in IAAS were down because of contrail vrouter
        problem in cloud storage.
        ----