It is a common practice among sysadmins to avoid doing actual work on Fridays. Not because we’re lazy (okay, not entirely because we’re lazy), but because if something goes wrong on Friday, it might not show up until Saturday. Just because I’m on call, doesn’t mean I actually want to go in. So of course I had reservations about installing new disks on our LDM (weather data) server, but it is important to classroom instruction and to my own geekery, so since the disks arrived on Thursday afternoon, I set aside my normal practice and did the install on Friday morning.
Backstory: our weather data server was set up about four years ago because the machine which had previously been tasked with data ingest could no longer keep up with the load. We got a grant from Unidata’s Equipment Award program to purchase a Dell PowerEdge 2850 with a 4×300 GB RAID 5, dual 3.6 GHz 64-bit processors, and 4 GB of RAM. Not exactly a top-of-the-line machine, but a big step up from the desktop-class machine that we had been using. Fast-forward four years and the size and number of data products has increased. Now the 500ish GB data partition is no longer sufficient.
The volume of data got to be so much that our scouring routine couldn’t keep enough disk space clear and eventually the data partition filled. This caused the decoders that run on the data as it comes in to freak out and core dump. This filled up the root partition, which ended up causing the machine to freak out (and also to take down an LDAP server –oops!). Since there were two more disk slots available, the solution was simple: add more disk. Which brings me to a week ago…
I scheduled a two-hour outage Friday morning, figuring it would be a quick job. Oh boy was I in for a surprise. My first discovery was that the machine was apparently configured to use three of the disks and have the fourth just chill, not even as a hot spare. The second discovery was that the array was generally cranky and felt that one of the disks needed to be rebuilt. It took a little bit to figure out how to make that happen (the PERC 4 BIOS is not entirely intuitive), but I finally figured it out. Once the rebuild started, it became clear that it would take hours to finish. It was very adamant that I not reboot the machine while that was going on, so I was pretty much stuck with it being down for the rest of the day.
Shortly after what should have been time to leave, the rebuild finished. So I added the two new disks to the RAID 5, bringing it up to 5x300GB (or 1.2 TB of usable space) and set the 6th disk as a hot spare. The array needed to do more math to grow itself, and I estimated that it would take about 13 hours to finish, which meant I’d have to come in on Saturday morning. I should have known better than to start this on a Friday, but the stupid had only just begun.
On my way in to the office on Saturday morning, I had stopped at the local farmers’ market to do some grocery shopping. It wasn’t until I got to my building that I realized I had forgotten my work keys. This was a sign of things to come, although I did not know it at the time. After returning home and then driving back to the office, I was able to get going. Now comes the easy part right?
I booted the machine, unmounted the data partition, and fired up fdisk. I decided to first create a new partition for the LDM user’s home directory, to keep / from getting filled. Then I made another partition to fill the rest of the disk that would be my larger, more awesome data partition. Once the partition table was written and the system rebooted, I created a new filesystem on the two new partitions and said “done!” Then I looked at the size of the data partition: 500 GB. That wasn’t right at all, it should have been more than double that.
Very perplexed, I began looking for an explanation. fdisk said the disk was 1.2 TB, so why wouldn’t it let me make the last partition bigger? I checked for limitations, but for the block size I was using, filesystems of 2 TB should be possible. After beating my head against it for an hour, I finally decided that 500 GB would have to work for now and that I’d just figure it out on Monday.
Over the weekend, I spent some time talking to my friend Randy about it, and he assured me he was just as confused as I was. Something was imposing an artificial constraint on the size of this partition, but I couldn’t figure out what it was. Come Monday, I sat down at my desk with a full pot of coffee ready to stab at it until it surrendered. I’m not sure what made me think of it, but all of a sudden I understood what the problem was.
In Master Boot Record-type partitioning, only four “primary” partitions can exist. If you want more than four partitions on a disk, numbers 5 and above are contained inside the fourth partition. It occurred to me that the 4th partition was never grown when I had added the two extra disks to the array. All I had to do was to delete partition 4 and then re-make it. I could then re-make the partitions inside it. Once I did that, I was able to create a 1.1 TB filesystem for data. Problem solved.
It really bothered me that I hadn’t figured that out on Saturday. It was a very simple, and quick solution that I already knew everything I needed to know for. Granted, I’d never done something like this before, but it should have occurred to me more quickly. At least one good thing came of this, though: I’ll never do actual work on a Friday again.
Don’t beat yourself up over it. Focusing on one problem inevitably narrows your peripheral vision. On the plus side, like you said, you’ll never do real work on Friday again, so hey, valuable lesson, right? 😉
Personally, I follow Weinberg’s Law: Progress is made on alternate Fridays
Doesn’t mean I do real work, it just means that something happens somewhere. Sadly, I think today is one of those “do something” Fridays.