Starting at 5:30 am EDT September 4th, 2018, Channeltvity's streaming video functionality stopped working due to an outage in the US South Central Azure data center operated by Microsoft. This is affecting only videos uploaded directly into Channeltivity and not embedded videos such as YouTube and Vimeo. Users attempting to view videos will see an error message in the application.
Current status from Microsoft:
CUSTOMER IMPACT: Starting at 09:29 UTC on 04 Sep 2018, customers in South Central US may experience difficulties connecting to resources hosted in this region. Additionally, this issue impacts Azure Active Directory authentication, which may impact services in other regions.
PRELIMINARY ROOT CAUSE: A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.
ENGINEERING STATUS: Engineers have successfully restored power to the datacenter. Additionally, engineers have recovered a majority of the impacted network devices. While some services are starting to see signs of recovery, mitigation efforts are still ongoing.
We will post updates as we receive them from Microsoft.
Update 4:30 pm EDT, from Microsoft:
CUSTOMER IMPACT: There are currently two identified impact workstreams:
1) Customers with resources in South Central US may experience difficulties connecting to resources hosted in this region. 2) Customers using non-regional services, such as Azure Active Directory, may experience intermittent authentication failures in any region.
PRELIMINARY ROOT CAUSE:
1) A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process. 2) As a result, non-regional services, such as Azure Active Directory, encountered an operational threshold for processing requests through the South Central US datacenter. Initial attempts to fail over into other datacenters resulted in temporary traffic congestion for those regions.
ENGINEERING STATUS: Engineers have successfully restored power to the datacenter. Additionally, engineers have recovered a majority of the impacted network devices. While some services are starting to see signs of recovery, mitigation efforts are still ongoing.
Update 11:24 am EDT, September 5: Microsoft is still working on the outage, but all Channeltivity video streaming services seem to be working again. We will keep updating this post until Microsoft marks it as resolved.
Zach Smith
Starting at 5:30 am EDT September 4th, 2018, Channeltvity's streaming video functionality stopped working due to an outage in the US South Central Azure data center operated by Microsoft. This is affecting only videos uploaded directly into Channeltivity and not embedded videos such as YouTube and Vimeo. Users attempting to view videos will see an error message in the application.
Current status from Microsoft:
CUSTOMER IMPACT: Starting at 09:29 UTC on 04 Sep 2018, customers in South Central US may experience difficulties connecting to resources hosted in this region. Additionally, this issue impacts Azure Active Directory authentication, which may impact services in other regions.
PRELIMINARY ROOT CAUSE: A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.
ENGINEERING STATUS: Engineers have successfully restored power to the datacenter. Additionally, engineers have recovered a majority of the impacted network devices. While some services are starting to see signs of recovery, mitigation efforts are still ongoing.
We will post updates as we receive them from Microsoft.
Update 4:30 pm EDT, from Microsoft:
CUSTOMER IMPACT: There are currently two identified impact workstreams:
1) Customers with resources in South Central US may experience difficulties connecting to resources hosted in this region.
2) Customers using non-regional services, such as Azure Active Directory, may experience intermittent authentication failures in any region.
PRELIMINARY ROOT CAUSE:
1) A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.
2) As a result, non-regional services, such as Azure Active Directory, encountered an operational threshold for processing requests through the South Central US datacenter. Initial attempts to fail over into other datacenters resulted in temporary traffic congestion for those regions.
ENGINEERING STATUS: Engineers have successfully restored power to the datacenter. Additionally, engineers have recovered a majority of the impacted network devices. While some services are starting to see signs of recovery, mitigation efforts are still ongoing.
Update 11:24 am EDT, September 5: Microsoft is still working on the outage, but all Channeltivity video streaming services seem to be working again. We will keep updating this post until Microsoft marks it as resolved.