A Microsoft employee accidentally hit the wrong key and deleted cloud databases. A blackout of more than ten hours followed.
As Microsoft’s Eric Mattingly, the manager of DevOps, shared in the Azure DevOps blog, the Azure databases were accidentally deleted in mid-May. The error was caused by an employee who crippled the databases in the South Brazil region. According to Mattingly, he simply hit the wrong key.
The incident occurred when an employee wanted to start a daily routine cleaning of the databases. This consists of deleting old records that are no longer needed to keep the cloud application running quickly.
A pull request with a typo was entered during a regular database scan. This resulted in the entire Azure SQL server being deleted, not a single Azure SQL database.
The Azure database outage lasted almost eleven hours
A Microsoft employee accidentally deleted Azure databases. Source: Unsplash/Tadas Sar
The failure was detected within 20 minutes of starting the database wipe and was immediately responded to by emergency technicians. They set about recovering the SQL server and affected databases.
One of the reasons it took so long was that recovery could only be done manually. There is no automatic backup and recovery system for such cases.
Mattingly apologized to all Azure users for the incident that happened on May 24 and the subsequent outages. Fortunately, there was no data loss. Microsoft is now working on a solution to prevent similar problems from happening again.
Source: Azure DevOps
