Stopping mdadm assemble on ubuntu boot

Soft raid setup with mdadm

I have a Raid-6 running and it currently has 6 drives. My Mobo has 6 sata ports, one of which is used up for the boot drive (not involved in the array). The sixth array drive is on a PCI sata/raid card. The server is headless, which is important because I do not want degraded array events to cause it to drop into busybox on boot.

The problem

I have a recurrent problem where my raid will drop one of the disks at boot. It will still assemble (dirty) because I had that configured when I used dpkg-reconfigure mdadm. The server is headless, so dropping into busybox wasn’t an option for me. After boot up, I can re-add the missing drive, but this causes a rebuild because the other 5 drives are in one state (5-1) and the lone drive is in another state (6 clean). My theory is that the sata card drive doesn’t come up quick enough for the assemble. dmesg seems to support this idea because mdadm starts its assemble and then several lines down the log the lone drive is mentioned and recognized.

Solutions

1. Make the assemble wait longer
2. Don’t assemble at boot, assemble by hand after boot.

I opted for choice two for now. It is important that you do not pick this solution if your root filesystem (i.e. ubuntu itself) is on the raid array. I have a seperate boot drive so I do not need assembly before boot.

Other specifics

mdadm version

# mdadm -V
mdadm - v3.1.4 - 31st August 2010

Ubuntu version

#uname -a
Linux 3.0.0-32-generic #51-Ubuntu SMP Thu Mar 21 15:50:59 UTC 2013 x86_64
x86_64 x86_64 GNU/Linux

# lsb_release -aNo LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 11.10
Release:    11.10
Codename:    oneiric

cat /proc/mdstat

# cat proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4]
[raid10]
md0 : active raid6 sdb1[0] sdc1[1] sdd1[2] sdf1[6] sde1[4]
      7814049280 blocks super 1.2 level 6, 128k chunk, algorithm 2 [6/5]
      [UUUUU_]
            bitmap: 2/15 pages [8KB], 65536KB chunk

Solution process

To get mdadm to not assemble on boot.

• I deleted(moved) all of mdadm config files out out of /etc/mdadm/ (don’t know if this was necessary)
• I commented out this line in /lib/udev/rules/85-md-raid.rules (note I am on 11.10)

ACTION=="add", RUN+="/sbin/mdadm --incremental $tempnode"

• Update initramfs

update-initramfs -u

• Cleaned out my fstab entries because these would hang up the boot if the raid volume can’t be found. If you set up fstab, just comment out the ones pertaining to your raid array.

Sidenote: During this little venture I had to drop in a video card and hook up a monitor in order to get through the busybox and such. I decided now was as good a time as any to upgrade to 12.04.2 LTS (been putting it off because I didn’t want to do it over SSH). As a known benefit I would get an upgraded mdadm version (v3.2.5).

• Reboot and the mdadm array should not assemble at boot. You can now assemble by hand at the prompt

mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1

• Re-add the one drive

mdadm /dev/md0 --re-add /dev/sdh1

We should be clean and running fine now.

If you are getting a complete rebuild you might try upgrading your distro or just mdadm (as I think that the newer version might be better at re-adding, not 100% on this). I had the newer mdadm version and I didn’t have to completely rebuild.