The Treehouse Blog

This cost me too much time yesterday and today, so I’ll share it with the world in case I happen to save someone time.

I have been migrating hard drives within my Fedora 16-based server over the past months (yes, it’s taking way too long).  A project this weekend was to migrate the root file system from one drive to another.  The initial and final states are mdraid mirrors, but in the interim there are two degraded mdraid mirrors – one for the old, one for the new.  In attempting to boot from the new device, the boot would fail in dracut rescue mode.  I could proceed with mdadm -As ; exit but no matter what I looked at or tried, I could not get the process to proceed on its own.  In desperation, I tried adding mdadm -As into a custom dracut module or as part of the stock mdraid module – no luck. Eventually, after turning on and wading through all dracut debugging I could find, which didn’t really help other than making me read the dracut mdraid scripts, the obvious error was revealed. The format of the UUID field that dracut uses is matched against the output of mdadm --examine, and I had copied-and-pasted the output of blkid instead. Only after lining them up did I realize that although the identifiers are the same, the format is different:

blkid: c72b5415-c310-9955-c694-43f9004a021b
mdadm: c72b5415:c3109955:c69443f9:004a021b

The particular kernel line fragment is now: root=UUID=69c0f668-d395-4d45-b3df-5835240a0986 rd.md.uuid=c72b5415:c3109955:c69443f9:004a021b

So… if you’re setting rd.md.uuid in order for dracut to assemble your root array and it’s not working, make sure the parameter is formatted in mdadm style, and not blkid style!

Another tip on the subject: Always make GRUB2 changes using /etc/sysconfig/grub and/or /etc/default/grub and then re-build the grub.cfg file using grub2-mkconfig. Also, don’t use a symlink as a parameter for grub2-mkconfig -o. This will remove the symlink and write a new file at the location of the symlink, not the target. And hence, your changes won’t have happened.

The desktop upgrade from Fedora 15 to Fedora 16 was about on par with the upgrade from 14 to 15.  Instead of the disaster that is Gnome 3, we’re instead greeted with GRUB 2 and new systemd quirks.

The main portion of the upgrade itself went smoothly.  No unexpected surprises from anaconda until the notice that the bootloader didn’t install right.

GRUB 2

So apparently Fedora 16 incorporates GRUB 2.  While its error messages seem far friendlier than GRUB classic, I really did not delve into all of its supposed benefits.  One downside is that when built with RAID support (which I seem to need since my /boot partition is mirrored), the core.img file ends up >32KiB, and thus does not fit in the post-MBR gap present on my drives.

To address this, I used a gparted live CD and resized and moved the first partition of each drive (which happen to be NTFS drives for my Windows 7 install, one of which was the system volume).  This provided a 2MiB gap between the MBR and first partition.  Booting back into the Fedora 16 rescue mode and using grub2-install on both drives successfully installed GRUB2 and, following a reboot, allowed Fedora 16 to load.

Unfortunately, these partition table and file system hijinks left Windows 7 with a bit of a problem, seeing as it would not boot.  The recommended method of using the Windows 7 installer’s “Startup Repair” feature was unsuccessful.  The “bootrec /fixboot” would not fix it, giving an “unsupported filesystem” error.  Using diskpart to set the Windows partition to active appears to resolve this, and the fixboot succeeds.  Naturally, I ran fixmbr at some point, which wiped out GRUB again, and thus it had to be reinstalled.  Success with booting both Windows 7 and Fedora 16 was then achieved.

NFS mount

The machine has one NFS share mounted via /etc/fstab.  After the upgrade, this would fail to mount during boot, but would have no difficulty being manually mounted after boot.  After researching a variety of wrong paths with various systemctl changes, the one I found to resolve this was “systemctl enable NetworkManager-wait-online.service”.

Update 1/8:

The upgrade of my HTPC wasn’t too painful.  Mucked with partitions to make room for GRUB2 ahead of time, had to change my lirc init script to not confuse systemd, disable screensaver in gnome 3 (yes, shouldn’t be using gnome to run mythtv – need to add that to the list), re-enabling services that weren’t automatically figured out from existing init scripts, switched mounts from /dev/md* to UUID-based to get the ordering right in the new boot sequence, mythtv ownership changes, etc, etc.

Fedora 9 to 12

Over the past few days I’ve been upgrading my file/media server from Fedora 9 to Fedora 12.  I did this with yum, upgrading from 9 to 10, 10 to 11, and 11 to 12 incrementally, removing conflicts as necessary.  This actually went surprisingly well, and at the end, with just one reboot, I had gone from 9 to 12.

It may have booted, but there were a number of things I had to fix afterwards:

• The kernel video mode setting had to be disabled to not break the NVIDIA binary driver (“blacklist nouveau” somewhere in /etc/modprobe.d/).  Apparently this is supposed to be handled by installing rpmfusion’s kmod package, but that wasn’t the case for me.
• The X server would crash immediately upon a login.  I eventually figured this out to be a missing gnome-session-xsession package.  I’m not sure if this is something I had removed for dependency reasons earlier, or if it was split from another package at some point and yum missed it.  Either way, it was a real pain to debug, but easy to fix.
• The kernel would not recognize the partitions on three of my disks.  This was a major pain, and the main focus of this article.
• I don’t like MythTV 0.22’s new video gallery

Fedora 12 (2.6.31.6-145.fc12.x86_64) Disk Partitions Issue?

So… sdb, sdc, and sde each have a single partition on them, but the kernel (per /proc/partitions and other means) would only report block devices for sdb, sdc, and sde not sdb1, sdc1, and sde1 as should have additionally existed.  Naturally, the first thing to consult would be dmesg:

sda: sda1 sda2 sda3
sdb:
sdc: sdb1
sdc1
sdd:
sde:
sdd1 sdd2 sdd3
sde1
sdf:
sdf1

At first glance, this looks really, really bad.  sdb1 existing on sdc?  That’s not supposed to happen.  But after looking at it further, and having had some experience debugging multi-threaded things, I became convinced this was the mangled output of multiple parallel partition discovery processes.  If that were the case, it looked like it should have been successful, but was not.

So, is this what happened?  Googling turned up a bit on the so-called “fastboot” patches to the Linux kernel, at least portions of which have been accepted into recent mainline kernels.  Supposedly these would only be enabled with the “fastboot” kernel parameter, but searching the source and docs for the latest kernel didn’t turn up this option.  The async libata device discovery does indeed appear to be in 2.6.31 mainline, and I was unable to find a knob to turn it off.  There were also references to this interfering with partition discovery.  I started the process of rebuilding the kernel to disable this, to see if it fixed the problem, but gave up in favor of a workaround.  I’m not fond of maintaining custom kernels – I think the last I did this was to support a boca card.

The workaround.  I had noticed that the kernel could be instructed to reread the partition tables (partprobe, for instance) and the missing partitions would appear.  I threw in a quick init script to do this and assemble the array early in the startup process:

mdadm --stop /dev/md2
sleep 1
partprobe
sleep 2
mdadm --assemble --scan /dev/md2
vgchange -ay
mount /storage

So… if I have time, I should complete that kernel rebuild and report this somewhere.  In the meantime, I’m posting this for the benefit of others.  Lucky for me the partitions affected did not contain my root partition, or this could have been less-work-aroundable.

Oak had an upgrade this week. It’s been running for years with an AMD Athlon XP 2400 on an Asus A7V8X motherboard. With the new tasks of running a 5-disk software RAID5 array and H.264 video codecs added over the past few years, it has started lagging behind. The processor and motherboard have been upgraded to AMD Athlon 64 X2 5400+ Brisbane running at 2.8GHz and 65W, and a Gigabyte GA-M57SLI-S4 motherboard. That should provide something around 2.8x the processing power, more disk I/O capability since the motherboard provides 6 onboard SATA ports instead of the two PCI-to-SATA cards I had been using which would max out at 133MB/s, and I’ve also quadruped my RAM from 1GB to 4GB.

Instead of upgrading the OS in place, I opted to do a fresh install of Fedora 9 x86_64 onto a software RAID mirror, replacing the old 10-gig non-UDMA IDE system drive that I had been using. This was relatively painless. A fresh mythtv install combined with importing my old database resulted in a working system. I had to copy over my lirc customizations, of course. The on-board sound with the default PulseAudio configuration works fine for everything including the raw digital output for AC3. One quirk I may need to look into is that when skipping around in a AC3 video, there will be an absence of sound for a considerable portion of a second before it comes back. This quirk is better than the problems I’ve had recently with my (ancient) SoundBlaster Live which would cause the receiver to not switch back to PCM mode on its own after an AC3 stream stopped.

I also upgraded to a GeForce 7300GS-based PCIe graphics card which has allowed me to switch to ‘gl’ rendering for mplayer, which seems to fix most or all of my frame tearing issues that I would experience when using 1024×768 to drive the TV output.

Still some issues to resolve, and I haven’t measured it’s power draw yet, but the upgrade process has been a lot less painful than I had feared.

When I purchased and installed my PVR-500 this fall for MythTV stuff, it had very poor signal quality. I attributed this to my cable provider, and bought a ridiculous +24dB amplifier from WalMart to rectify the problem, which it did. After upgrading the box from Fedora Core 5 to Fedora Core 6, the reception on the box was awful. I confirmed that the change in kernel from 2.6.17 to 2.6.19, or the accompanying changes to the ivtv driver were the cause. After some research, it turns out that I have an “evil” Samsung tuner card, and that in 2.6.17 the internal amplifier on the tuner is not activated by the Video for Linux drivers. So, my original amplifier purchase was to compensate for a software problem. After removing the amplifier (not just turning it off as I had been foolishly trying as a test), the reception was mostly better. Some channels are better than before, but some are worse, and overall I think this was not a good change. However, there is no software setting for enabling/disabling the internal amp (apparently no good reason to turn it off), so I’m going to go with the internal amp instead of maintaining a custom kernel on the box, which is always a pain.