Fork me on GitHub

You are here

Sysadmin madness

Sunday evening at 10pm one of our Enterprise 10000 domains rebooted. Problem was an error on one of the CPU's. I call Sun dispatch, end up in the UK, and they tell me it is so unlikely to happen again they don't replace it on the first occurrence of the error.

Last night it happened again. Consequence: SAP offline this morning till noon. Main cause of the four hours of downtime was because the engineer was stuck in traffic.

Last evening I was busy reorganizing the file server of schamper. And disaster struck. I accidentally deleted part of the website. And I had no back ups. I knew that getting these files back would be a pain in the ***, considering the files are on an ext3 file system. From the ext3 FAQ:
Q: How can I recover (undelete) deleted files from my ext3 partition? Actually, you can't! This is what one of the developers, Andreas Dilger, said about it:

In order to ensure that ext3 can safely resume an unlink after a crash, it actually zeros out the block pointers in the inode, whereas
ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.

Your only hope is to "grep" for parts of your files that have been deleted and hope for the best.Luckily schamper is going to use Drupal this summer, so the files I deleted weren't that important after all.

I did manage to recover some files using this awesome tool. I still have to go through the files so see how much it recovered.

Add new comment