On sunday I decided to drive up to Fremont (about 200 miles from my house) to place one of our new servers named catbus (from Totoro). I was missing a part (64 bit left angle PCI riser card) so a good chunk of my time in the bay area was spent trying to locate said part. I thought it was possible to get any part for your computer in the bay area at any time of day — but nooo. Not on a sunday!
So, MusicBrainz friend Kevin Murphy volunteered to acquire the part and install it later this week. So, I racked the server, connected it but didn’t power it up. Better than just going home.
Then Jeff wanted me to be present for a software upgrade on zim. To complete the upgrade we rebooted zim, but he didn’t come back up cleanly, so I had to wheel the terminal cart over to zim and nurse him back to health. No big problems, but more stress. As I have zim hooked up, there is this other computer beeping really loud nearby. I think: “Man, that is annoying. Why don’t people take better care of their machines? Damn, which machine is that?”
After some snooping around, it turns out to be our very own Bender, with one of its hard drives failed out of the RAID array. ARG! So once I got done with zim, I zip things back up, drive to Fry’s and buy a new drive for bender at highway robbery prices. Back to the colo and then install the new drive on bender. I boot bender back up, tell the RAID to rebuild and head for home.
We’ll have to go and install more system monitoring tools to watch the status of the RAID array. I would’ve expected some messages in the syslog, but after considering the nature of the RAID system, I see why that wasn’t the case.
Well, live and learn. Things seem to be in good shape right now, so lets hope they stay that way.