guy labs guy labs

Create a software RAID1 with mdadm on an active Ubuntu 13.04 till 16.10 hard drive

Today we want to create a RAID1 array with an active Ubuntu 13.04 till 16.10 hard drive. I installed my server without a RAID array on a single hard drive. And now, after spending a lot of hours configuring it, I want to create a RAID1 array to have a security if one of the hard drives fail. Create a simple RAID1 array is easy with mdadm, but creating it with a hard drive that has data on it isn’t that simple.

I created the following howto with the help of this older blog post about the same problem: http://feeding.cloud.geek.nz/posts/setting-up-raid-on-existing

[box type=”info”] Use this article with caution because this is the solution for my machine and I don’t know if it works on other machines, but I gave my best to test everything in a virtual machine and on my server with the UEFI boot mechanism. Also create a backup of your hard disk first: Create a CloneZilla disk image[/box]

Ok let’s start and we first need to install the following packages:

If you havent already installed postfix, set the option ‘no configuration’ for the postfix installation dialog.

First attach your new hard drive (best to have one with the same size or bigger) and then execute the following commands to copy the partitions of /dev/sda to the second drive (in my case /dev/sdb):

[box type=”info”] If you have a GPT partition table you need to install gdisk and copy the partition table with the following commands:

The first command here copies the partition table from /dev/sda to /dev/sdb (don’t mix it up). And the second one randomizes the GUID of the new partitions on /dev/sdb.[/box]

Next check if the partition tables are identical between both hard drives. Execute the following commands and compare the output of fdisk for each hard drive. Also note the specific partition names of your swap (/dev/sdb3 in my case) and the main boot partition (/dev/sdb2 in my case).

[box type=”info”] If you have a GPT partition table you need to compare the partitions with these commands:

[/box]

Now we need to create the a new degraded RAID array of the newly partitioned hard drive. Execute the following commands and double check the correct partition names for the swap and the main partition.

[box type=”info”] If you have a working postfix installation, then you should receive a warning email from mdadm saying that you have a degraded RAID array. So you don’t need to check that the mails arrive when you have a problem with the mdadm RAID hard drives. :)[/box]

After that we need to format the newly created degraded RAID partitions with the following commands:

Next we need to specify the devices in the mdadm.conf file. Edit the file

and add the following property to the file:

The last thing to do before installing GRUB2 on that drive is to add the RAID arrays to the end of the mdadm.conf files. To do this execute the following commands:

Now copy the output and paste it at the end of the mdadm.conf file.

After that reconfigure the mdadm package with the following command:

In the configure dialog of mdadm answer always yes and add your email address (or the user account of the configured postfix user) for important messages from mdadm. (e.g. disk failures etc.)

After the creation of the new degraded RAID array we need to install GRUB2 on each hard drive. To do this we just reconfigure the grub-pc package with the following command:

You need to select both hard drives as installation targets, /dev/sda and /dev/sdb and not /dev/md0.

[box type=”info”] If the package grub-pc is not installed (because you are using grub-efi) then you need to do the following: Copy the EFI boot partition of sda to sdb by executing the following command: (Be carful and select the correct partitions!)

sudo dd if=/dev/sda1 of=/dev/sdb1

[/box]

Now we need to copy the existing data to the new hard drive with the following commands:

After that we need to edit the /tmp/mntroot/etc/fstab as root and change the UUID’s of the data partition and the swap partitions. If you use the UUID notation for your partitions you can get them using the following commands:

If you have the old notation you can use /dev/md0 for the data partition and /dev/md1 for the swap partition.

We also need to unmount the before mounted folder and delete it. Do this by executing the following commands:

Before the reboot we need to check if the grub menu is shown at boot. Open up the /etc/default/grub file as root and change the following values:

The last value is not needed but it’s good to see whats happening at the start up. After that we need to update GRUB with the following command:

Now we reboot the machine (press ESC if there is a countdown) and in the Grub menu you can edit the start up properties by pressing e. Add/adapt the following properties:

If you are using the UUID partition format set the last parameter to: root=UUID={UUID-of-md0}

After you have finished editing the properties press F10 to boot.

After the successfull boot, test everything on the new hard drive and have a look at the output of the mount command. This should show the following:

Here you see that Ubuntu is using the new degraded RAID array which is using the /dev/sdb hard drive.

Now we are going to wipe the original hard drive by adding it to the RAID array. Be sure to have a backup of your hard disk somehow. If not follow this guide here: Create a CloneZilla disk image

Ok lets add the /dev/sda drive to the degraded RAID array with the following commands:

You can watch the progress of the synchronization of the drives with the following command:

Wait till the synchronization is finished and then you can quit it with CTRL-C.

The next step is to test if everything worked with the synchronization to the original hard drive /dev/sda. To do that we just boot of the original hard drive.
First we need to update grub with the following command: (This is just for the MBR boot option)

If you have the EFI boot option you just need to redo the Grub config with the following command:

This generates a new grub.cfg file in which the new RAID hard drive is the root from which Ubuntu now boots.

After a successful reboot (if not read the next two info boxes on how to fix Grub issues) without changing the startup properties in the Grub menu the mount command should print the following:

We are now finished with the RAID1 array creation. If you want to test the RAID1 array and simulate a hard disk failure you can read on.

For that, turn off the machine with the following command:

And then we physically disconnect the new hard drive (/dev/sdb) and start up the system normally without changing options in the grub menu. After a successful boot, shut the machine down and physically attach the new hard drive again.

[box type=”info”] If you are using the UEFI boot option and you can’t boot you need to go to your BIOS and change the UEFI boot partition (the name is by default ubuntu) to be the first to boot. Maybe you have multiple entries with the name ubuntu, then you need to figure out which one works. 🙂 If this doesn’t help follow the instructions in the next info box.[/box]

[box type=”info”] If you get a error with the following message: “error: file ‘/boot/grub/i386-pc/normal.mod’ not found.” or any other grub error and you end up with a ‘grub rescue>’ prompt you need to re install grub on the /dev/sda hard disk. To do this start from a Ubuntu live CD and use BootRepair (I did that for my EFI boot on the server and it worked like a charm) to fix Grub or try to enter the following commands in the command line: (Just for the MBR boot option. This one worked in my virtual machine with the MBR boot)

[/box]

Now start up the machine. After the boot you should be able to see the following output when executing the mount command:

Ok we are now sure that the original hard drive is working as expected in a degraded RAID state. (without the new hard drive /dev/sdb) Now shutdown the machine again and physically attach the new hard drive /dev/sdb again. Start the machine.

If the machine booted up normally you need to execute the following command to see that the RAID array is incomplete and that the second drive isn’t part of it:

To add the new drive to the RAID array and start a synchronization execute the following command:

You can again watch the status of the synchronization with the following command:

After the synchronization is complete you need to shut down the machine and physically remove the original hard drive /dev/sda and try booting with the new drive /dev/sdb. After a successful boot and a check that everything works as expected, shut down the machine again and boot up with both hard drives connected.

After that we need to add the original hard drive back to the RAID array by executing the following command:

You can again watch the status of the synchronization with the following command:

Ok now we are done. We now have a fully tested RAID1 array.

It’s always good to monitor the S.M.A.R.T. status of the hard drives. I wrote a blog post about how to setup smartmontools on your server.

If you have problems or want to remove the RAID array or just one hard drive follow these steps:

For that we need to set the desired hard drive as faulty. To do this execute the following command with the correct hard drive name:

After that we need to remove the hard drive from the RAID array with the following command:

You can do this for each hard drive you want to remove from the RAID array. To completely remove the RAID array and mdadm you need to stop the RAID array and remove the super blocks. But do a backup before executing the commands and you can just do them within a booted live CD environment. Choose the correct RAID array name and hard drive name.

Ok that was a long post, and I hope it helps someone to create their RAID array. It took me a lot of hours to find out the correct way.

Happy RAID’ing.  😉

Guy

61,246 total views, 14 views today

Share Post :

23 Comments

  • Richard
    March 20, 2014 6:02 AM at 6:02 AM 

    Thanks for the great post! Was trying how to go from mint 14 to mint 16 doing a raid 5 and this was what I needed to get me there. Stupid EFI switches changing everything on me.

    Put a few KEYwords in that to help people find this post in a search.

    • guy
      March 20, 2014 7:29 AM at 7:29 AM 

      Hi Richard,

      thanks for your comment and I’m glad that this post helped you. I already added some keywords to the SEO settings. Do you think that’s not enough?

      Cheers,

      Guy

  • Hans Ahlén
    March 20, 2014 5:18 PM at 5:18 PM 

    Hello Richard,

    I have followed your guide on a 12.4 LTS system and it worked up to the point where it was time to boot. It worked up to the point where I got the messages
    “mdadm: CREATE user root not found
    mdadm: CREATE root disk not found
    Started the RAID in degraded mode”

    Then nothing more happens. Any sugestions?

    • guy
      March 20, 2014 5:39 PM at 5:39 PM 

      Hi Hans,

      do you mean the first reboot where you need to configure the Grub menu properties to use the md0 hard drive and need to adapt the following lines:

      insmod mdraid1x
      set root='(md/0)'
      linux /boot/vmlinuz-3.8.0-23-generic root=/dev/md0 ro

      And can you send me the ouput of the following command when starting in degraded mode:

      mount

      Maybe the content of the following file would also be useful:

      /etc/mdadm/mdadm.conf

      Cheers,

      Guy

  • Hans
    March 20, 2014 10:22 PM at 10:22 PM 

    Hi Guy

    Yes I am stuck with the first reboot. The output from /etc/mdadm/mdadm.conf is:

    # mdadm.conf
    #
    # Please refer to mdadm.conf(5) for information about this file.
    #

    # by default, scan all partitions (/proc/partitions) for MD superblocks.
    # alternatively, specify devices to scan, using wildcards if desired.
    DEVICE /dev/sda* /dev/sdb*

    # auto-create devices with Debian standard permissions
    CREATE owner=root group=disk mode=0660 auto=yes

    # automatically tag new arrays as belonging to the local system
    HOMEHOST h-ahlen.com

    # instruct the monitoring daemon where to send mail alerts
    MAILADDR hans@h-ahlen.com

    # definitions of existing MD arrays

    ARRAY /dev/md0 metadata=0.90 UUID=5cc5de59:aa0d69ed:b30c511e:8320354a
    ARRAY /dev/md1 metadata=0.90 UUID=8f0cd5c8:d58fc05b:b30c511e:8320354a"

    I can’t send you the output from “mount” from booted into the mdadm environment as I am stuck.

    /Hans

    • guy
      March 20, 2014 10:45 PM at 10:45 PM 

      hmm ok. The config looks ok. But you are not able to boot with the degraded mode? Some questions:

      1. Do you have a user which has the name root?
      2. Do you have an EFI powered mainboard or a BIOS one?
      3. And you updated the fstab on the md0 partition with the new UUID’s?

      My mdadm.conf file looks exactly the same as yours, so there shouldn’t be the failure.

    • Oleg
      April 5, 2014 7:27 PM at 7:27 PM 

      Hey, I think you just have ” at the end of md1. This typo should be removed.

      • guy
        April 6, 2014 3:44 PM at 3:44 PM 

        Hi Oleg,

        nice catch! Thanks for this comment.

        @Hans: If you have this quote in the mdadm.conf then try to remove it.

  • Oleg
    April 5, 2014 4:31 PM at 4:31 PM 

    Thanks for guide, works great.

    Ubuntu server 13.10 is here, UEFI, GPT.

    Used repair-boot for fixing booting system after raid1 is done. It is enough smart tool – using UUID well.

  • Csaba Bocskai
    August 2, 2014 9:53 AM at 9:53 AM 

    I have trouble with grub. It can’t find the kernel. What is your original partition scheme? Do you have separate partition for /boot?

    • guy
      August 5, 2014 7:28 AM at 7:28 AM 

      Hi Csaba,

      my original partition scheme was the standard one from Ubuntu 13.04 (/swap and /). And after the raid1 setup I have the following partitions:

      • /dev/sda1 -> /boot/efi
      • /dev/md0 -> /

      I hope this helps.

      Cheers,

      Guy

      • Csaba Bocskai
        August 5, 2014 10:43 AM at 10:43 AM 

        Thanks, but unfortunately I have a different setup.
        There was /, /home and swap. Assigned them to md0, md1 and md2. Grub didn’t find the kernel to load. There were 2 possible places to look for but neither was helping it load Linux. I want to create a RAID1 which includes boot stuff and data (I mean everything that’s on disk).
        I also didn’t understand why did you create the degraded RAID array using only sdb2 and sdb3, and not including sdb1? Is it related to efi? I’m not using efi, so how would my scenario look like?
        If you have time, please answer everything. I’m preparing for an admin job and need to know this sort of stuff.
        Thank you!

        • guy
          August 5, 2014 5:07 PM at 5:07 PM 

          When I tested everything in a VM I could not create a RAID1 with the boot partition and I also read about it and I did not find any solution for it. So you need to have grub installed on both disks separately. And no that is not related to EFI; I always show the EFI way in the yellow warning boxes of the blog post.

          If you need to learn these things then do all the possible scenarios in a VM with very small hard drives such that it does not take long and I would read the mdadm manual first. This guide is working in my scenario and I cannot do it for all possible scenarios.

          • Csaba Bocskai
            August 9, 2014 10:40 AM at 10:40 AM 

            Thanks!
            I did it in a VM because my server is for production purposes.
            I’ll try to make separate /boot to see how it will go.

  • dKeith
    March 16, 2015 8:56 PM at 8:56 PM 

    This is a wonderful walkthrough. Unfortunately, on step 2 (verifying GPT setups) I fat-fingered the command an did an ‘sgdisk -G ‘, randomizing thej GUID’s on the original, not-yet-replicated drive.

    Should I be concerned that the bios will not be able to find its file system layout on the next boot?

    Since it is still booted, the new table is not yet in play.
    – is there a way to revert to the old one still used by the kernel?

    It’s Ubuntu 14.04.1 BIOS booted with GPT’s devices

    So much TIA

    • guy
      March 17, 2015 7:54 AM at 7:54 AM 

      Hi Keith,

      thanks! 🙂 So you say that you randomized the GUID of the original hard drive and now it still boots right? I just searched a while but I didn’t find anything to revert the original one. Maybe you see the old UUID in the /etc/fstab file. You can then change it like it’s written
      here.

      But if everything works as before then I would just start over with cloning the partition table to the second hard drive and then randomize this GUID.

      I hope that helps.

      Regards,

      Guy

  • dKeith
    March 19, 2015 8:58 PM at 8:58 PM 

    I had *not* rebooted since issuing this command. I wasn’t sure if I had created a world of hurt for myself.

    However, as I understand it, what I changed was the file system identifier that is used by fstab, mdadm and so forth to identify the partition.

    I was able to follow the instructions in the link above to reinstall the UUID’s for the relevant partitions and it worked like a champ.

    I rebooted multiple times, synced the data in single-user mode and created the other half of the RAID partitions (/ and swap). I am presently working through an issue where the system can’t find the OS (grub problem I think). That is fodder for another thread.

    Many thanks

    • guy
      March 19, 2015 9:06 PM at 9:06 PM 

      Hi Keith,

      ok good that this now works. Ok if you have issues with Grub etc. please have a look at Boot-Repair. (Like I wrote in the post in a hint) This helped me with any Grub issues.

      Hope that helps and I’m glad that I could help you.

      Regards,

      Guy

  • SuperUser
    November 1, 2016 5:42 AM at 5:42 AM 

    Still up to date and very helpful on Ubuntu 16.10. Thanks

    • guy
      November 1, 2016 6:03 AM at 6:03 AM 

      Hi,

      thanks for testing it on Ubuntu 16.10. I will just update the title to reflect this.

      Regards,

      Guy

  • Jack Straw
    February 2, 2017 5:19 PM at 5:19 PM 

    Thanks, Guy, for this very helpful guide to mirroring an ubuntu installation.
    It worked for me on a single-boot 16.04.1 release via UEFI with SecureBoot disabled
    after I adapted it to work with separate partitions
    for /, /var, /tmp, /home, and swap.
    In case it might be of use to others, here is a situation I ran into and how I solved it.
    Following your guide, I got happily all the way to the last reboot but that failed and
    I got put into grub> and then somehow (initramfs). Anyway, I eventually constructed
    the proper incantation to get from grub> to the successfully mirrored OS I knew was there.
    Then as you suggested, I ran Boot-Repair in that running OS and that got me
    to bootable state off of /dev/sda1.
    HOWEVER, and this is the main point, this Boot-Repair did nothing to install
    the newly formed correct EFI partition on /dev/sdb1 (as it did on /dev/sda1).
    In particular, sdb1 did not contain the new and properly functioning grubx64.efi .
    So, I did the following:
    mount /dev/sda1 /tmp/mntt
    rsync -rtgpv /tmp/mntt/EFI/ /boot/efi/EFI/
    where for some reason:
    /dev/sdb1 523248 6808 516440 2% /boot/efi
    even though the OS booted off of sda1.

    Then I noticed that efibootmgr -v said:
    Boot0000* ubuntu HD(1,GPT,0d9b6de7-3918-46c4-9537-94d011049f38,0x800,0x100000)/File(\EFI\UBUNTU\GRUBX64.EFI)
    Boot0007* ubuntu HD(1,GPT,65f9283e-812c-49f5-9891-6fb1731dcaf2,0x800,0x100000)/File(\EFI\UBUNTU\SHIMX64.EFI)

    blkid confirms that Boot0000 is sda and Boot0007 is sdb.
    At first I was concerned that the boot loader for sdb is SHIMX64.EFI
    and not the same as GRUBX64.EFI for sda.
    However, I discovered that, even though Secure Boot is disabled,
    SHIMX64.EFI will still just call GRUBX64.EFI .
    I confirmed this by going into UEFI on my amd HP box and pushing
    enabling me to choose to boot off of the second ‘ubuntu’ entry,
    which is for sdb.
    After the OS booted, efibootmgr -v confirmed that:
    BootCurrent: 0007

    So, it seems to me that after this long, time-consuming, painful,
    reluctantly educational process that the owners of ubuntu chose to put
    me through, I now have an operational mirrored ubuntu installation.
    I could not have done this without Guy’s help. Thanks again.
    I note that CentOS continues to be kind enough to support mdadm mirroring
    during installation whereas the owners of ubuntu took it out of the Desktop
    version.

    • guy
      February 6, 2017 7:49 AM at 7:49 AM 

      Hi Jack

      thanks a lot that you shared the solution for your issue. And nice that this post was useful.

      Regards,

      Guy

Leave a Reply

Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.