Here is a piece of news we hardly hear from the IT industry. In a world where every IT guy keeps hundreds of backups to make sure they have all the important files in case of a catastrophe, a man accidently deleted all the files that were present on his company’s servers through a single line of code. And this included his company’s files, his client data, and backups as well.
Marco Masala, who owns a hosting provider company, told the whole world about his tragedy on Server Fault, a website which is a forum for server experts.
The reason why Masala wrote about his problem on Server Fault was to ask the experts about their suggestions. But, instead of helping him, all the people online could say was that Masala had just deleted his entire company and invited an array of lawsuits from his clients. Needless to say, he should lawyer up.
This is what Masala wrote on the website – “I run a small hosting provider with more or less 1535 customers, and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line. All servers got deleted and the offsite backups too, because the remote storage was mounted just before by the same script (that is a backup maintenance script)”
The line of code which deleted everything
The code rm –rf is infamous for deleting everything on a system if it’s not used properly.
The ‘rm’ tells the system to delete, ‘r’ deletes everything within a directory and ‘f’ forces the system to ignore all the warnings that might occur while deleting files.
The main problem with this code was Masala forgot to specify the location where it should run, and without a location, the code just ran on the entire system including main servers and backup servers.
Though backup servers shouldn’t have been deleted in a scenario like this, they were mounted to the main drives just before this code was run and that is why they got deleted as well.
A case like this proves how important it is to keep multiple backups at multiple locations, even if it might cost more. At every given time, a company should have three backup copies of its main server, one offline and two at remote locations. Also, backup strategies should be made in a way that any corruption or deletion doesn’t affect the backups at all.
If you could delete backups from the main server, then they shouldn’t be called backups at all since they won’t be solving their purpose.
Unfortunately for Marco Masala, it seems like he has just lost his entire company and it would take him a lot of time to bounce back.
