Starting at around 3:50 pm EDT May 2nd, 2019, Channeltivity customers started experiencing intermittent downtime due to a service availability issue with Microsoft Azure Services. We will post any updates we receive from Microsoft.
Azure Update from 6:10 PM (EDT)
Microsoft engineers have identified the underlying root cause. Mitigation has been applied, and their engineering teams are continuing efforts to fully mitigate the issue. Most Azure services are showing recovery at this time.
We'll continue to post updates until the issue is fully resolved by Microsoft.
Azure Update from 7:08 PM (EDT)
Microsoft engineers have identified the underlying root cause as an incorrect name server delegation issue affecting DNS resolution, network connectivity, and downstream impact to Compute, Storage, App Service, AAD, and SQL Database resources.
Mitigation has been applied, and the majority of Azure and other Microsoft services have recovered. They are in the process of final validation to ensure full recovery.
Azure Update from 8:17 PM (EDT)
Microsoft Engineers identified the underlying root cause as a nameserver delegation change affecting DNS resolution and resulting in downstream impact to Compute, Storage, App Service, AAD, and SQL Database services. During the migration of a legacy DNS system to Azure DNS, some domains for Microsoft services were incorrectly updated. No customer DNS records were impacted during this incident, and the availability of Azure DNS remained at 100% throughout the incident. The problem impacted only records for Microsoft services.
To mitigate, engineers corrected the nameserver delegation issue. Applications and services that accessed the incorrectly configured domains may have cached the incorrect information, leading to a longer restoration time until their cached information expired.
Next steps: Engineers will continue to investigate to establish the full root cause and prevent future occurrences.