Don’t forget the power splitter
This evening I had to make an emergency run to the data center to care for a server that had decided to hang during a reboot. While there, I figured I'd go ahead and add a new drive to the Tucows server since we're pretty tight on space right now. Little did I imagine the fun I had in store.
The ailing server wanted to be physically rebooted, and then it wanted updates…and more updates…and more. Each round of updates led to a new batch of fixes that needed to be installed. When all was said and done, I think I probably rebooted the server half a dozen times in a span of about half an hour. But now it's happy again and reboots cleanly and claims not to need any more updates.
But that was only half the fun. The Tucows mirror (a Linux server) should have been a slam dunk — power down, pop in a new drive, power up, format and mount the drive, and we're set. Well, it's never as easy as you think it'll be. Oh, I popped in the drive easy enough, but there was no power for it, and I didn't have a splitter so that I could power the drive. OK, easy enough, I'll leave the drive in there and come back later with a splitter. But hey, while I've got the case open and happen to have some extra memory here, let's add some memory to the box. Done. Power up. /dev/hda1 has errors, forcing check. Arrggghh! After a lengthly fsck (file system check), I had corrected everything and finally brought the server back online about a half hour later. This should have been a 10 minute job!
The good news is that it serves pages noticeably faster with the extra memory, so perhaps it was a success after all. I just hate it that I'll need to bring it down again within the next few days to complete the hard drive install. Remember, this is a server that takes more than a MILLION hits a month! There simply are no “down” times for this box.