Create a RAID array using with mdadm Rumi, November 23, 2023 mdadm (multiple devices admin) is an extremely useful tool for running RAID systems. It’s is a tool for creating, managing, and monitoring RAID devices using the md driver. It can be used as a replacement for the raidtools, or as a supplement. You can use whole disks (/dev/sdb, /dev/sdc) or individual partitions (/dev/sdb1, /dev/sdc1) as a component of an array. The benefits of using mdadm are: mdadm can diagnose, monitor and gather detailed information about your arrays. mdadm is a single centralized program and not a collection of disperse programs, so there’s a common syntax for every RAID management command. mdadm can perform almost all of its functions without having a configuration file and does not use one by default. mdadm software tools work for all Linux distributions, with the same syntax. Make sure you do a system update and then install the latest mdadm program into your system : # yum clean all # yum update # yum install mdadm -y List all the devices on your system: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 7.3T 0 disk └─sda1 8:1 0 7.3T 0 part /mnt/mydrive sdb 8:16 0 7.3T 0 disk sdc 8:32 0 7.3T 0 disk sdd 8:48 0 7.3T 0 disk sde 8:64 0 7.3T 0 disk nvme0n1 259:0 0 7.3T 0 disk └─nvme0n1p1 259:1 0 7.3T 0 part / I want to RAID together sda through sde (crazy, I know). I noticed that sda already has a partition and a mount. We should make sure all the drives that will be part of the array are partition-free: $ sudo umount /dev/sda?; sudo wipefs --all --force /dev/sda?; sudo wipefs --all --force /dev/sda $ sudo umount /dev/sdb?; sudo wipefs --all --force /dev/sdb?; sudo wipefs --all --force /dev/sdb ... Do that for each of the drives. If you didn’t realize it yet, this wipes everything. It doesn’t zero the data, so technically it could still be recovered at this point! Check to make sure nothing’s mounted (and make sure you have removed any of the drives you’ll use in the array from /etc/fstab if you had persistent mounts for them in there!): $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 7.3T 0 disk sdb 8:16 0 7.3T 0 disk sdc 8:32 0 7.3T 0 disk sdd 8:48 0 7.3T 0 disk sde 8:64 0 7.3T 0 disk nvme0n1 259:0 0 7.3T 0 disk └─nvme0n1p1 259:1 0 7.3T 0 part / Looking good, time to start building the array! Partition the disks with sgdisk You could interactively do this with gdisk, but I like more automation, so I use sgdisk. If it’s not installed, and you’re on a Debian-like distro, install it: sudo apt install -y gdisk. sudo sgdisk -n 1:0:0 /dev/sda sudo sgdisk -n 1:0:0 /dev/sdb ... Do that for each of the drives. WARNING: Entering the wrong commands here will wipe data on your precious drives. You’ve been warned. Again. Verify there’s now a partition for each drive: pi@taco:~ $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 7.3T 0 disk └─sda1 8:1 0 7.3T 0 part sdb 8:16 0 7.3T 0 disk └─sdb1 8:17 0 7.3T 0 part sdc 8:32 0 7.3T 0 disk └─sdc1 8:33 0 7.3T 0 part sdd 8:48 0 7.3T 0 disk └─sdd1 8:49 0 7.3T 0 part sde 8:64 0 7.3T 0 disk └─sde1 8:65 0 7.3T 0 part ... Creating the RAID devices To list the options to create RAID device with mdadm use the –help option. There are several options when creating RAID with mdadm. I will be listing few important ones. # mdadm --create --help -C | --create /dev/mdn -l | --level 0|1|4|5 -n | --raid-devices device [..] -x | --spare-devices device [..] Linear mode – Two or more disks are combined into one physical device. – The disks are “appended” to each other, so writing linearly to the RAID device will fill up disk 0 first, then disk 1 and so on. – The disks does not have to be of the same size. – There is no redundancy in this level. – The read and write performance will not increase for single reads/writes. But if several users use the device, several users using different disks at same time, you will see a performance gain. To create two disks in linear mode running mdadm, just type a single command line: # mdadm --create --verbose /dev/md0 --level=linear --raid-devices=2 /dev/sdb /dev/sdc mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started. The same command can be run using the shorter version of the options : # mdadm --Cv /dev/md0 --l linear -n2 /dev/sdb /dev/sdc RAID 0 Also called “stripe” mode. The devices should have the same size. There is no redundancy in this level either. No data rescue is possible if a drive fails. The read and write performance will increase, because reads and writes are done in parallel on the devices. To create two disks in RAID 0 mode: # mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/sdb /dev/sdc RAID 1 This level has redundancy. RAID-1 can be used on two or more disks with zero or more spare-disks. This mode maintains an exact mirror of the information on one disk on the other disk(s). Of Course, the disks must be of equal size. If one disk is larger than another, your RAID device will be the size of the smallest disk. If up to N-1 disks are removed (or crashes), all data are still intact. If there are spare disks available, and if the system survived the crash, reconstruction of the mirror will immediately begin on one of the spare disks, after detection of the drive fault. Write performance is often worse than on a single device as same data has to be written simultaneously on 2 or more devices. You can setup RAID 1 with two disks and one spare disk: # mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc --spare-devices=/dev/sdd RAID 4 This RAID level is not used very often. It can be used on three or more disks. Instead of completely mirroring the information, it keeps parity information on one drive, and writes data to the other disks in a RAID-0 like way. Because one disk is reserved for parity information, the size of the array will be (N-1)*S, where S is the size of the smallest drive in the array. If one drive fails, the parity information can be used to reconstruct all data. If two drives fail, all data is lost. To setup RAID 4 with 4 disks and one spare disk: # mdadm --create --verbose /dev/md0 --level=4 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde spare-devices=/dev/sdf RAID 5 RAID-5 can be used on three or more disks, with zero or more spare-disks. The resulting RAID-5 device size will be (N-1)*S, just like RAID-4. The big difference between RAID-5 and -4 is, that the parity information is distributed evenly among the participating drives, avoiding the bottleneck problem in RAID-4. If one of the disks fail, all data are still intact, thanks to the parity information. If spare disks are available, reconstruction will begin immediately after the device failure. If two disks fail simultaneously, all data are lost. RAID-5 can survive one disk failure, but not two or more. Reads are similar to RAID-0 reads, writes are generally expensive as parity has to be written which becomes the overhead. To setup RAID 5 with 3 disks and 1 spare disk using mdadm: # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd --spare-devices=/dev/sde Verify the array is working For RAID 0, it should immediately show State : clean when running the command below. For other RAID levels, it may take a while to initially resync or do other operations. $ sudo mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Creation Time : Wed Nov 10 18:05:57 2021 Raid Level : raid0 Array Size : 39069465600 (37259.55 GiB 40007.13 GB) Raid Devices : 5 Total Devices : 5 Persistence : Superblock is persistent Update Time : Wed Nov 10 18:05:57 2021 State : clean Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Chunk Size : 512K Consistency Policy : none Name : taco:0 (local to host taco) UUID : a5043664:c01dac00:73e5a8fc:2caf5144 Events : 0 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1 4 8 65 4 active sync /dev/sde1 You observe the progress of a rebuild (if choosing a level besides RAID 0, this will take some time) with watch cat /proc/mdstat. Ctrl-C to exit. Format the array $ sudo mkfs.ext4 -m 0 -E lazy_itable_init=0,lazy_journal_init=0 /dev/md0 mke2fs 1.44.5 (15-Dec-2018) Discarding device blocks: done Creating filesystem with 9767366400 4k blocks and 610461696 inodes Filesystem UUID: 5d3b012c-e5f6-49d1-9014-1c61e982594f Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000, 214990848, 512000000, 550731776, 644972544, 1934917632, 2560000000, 3855122432, 5804752896 Allocating group tables: done Writing inode tables: done Creating journal (262144 blocks): done Writing superblocks and filesystem accounting information: done Mount the array Checking on our array with lsblk now, we can see all the members of md0: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 7.3T 0 disk └─sda1 8:1 0 7.3T 0 part └─md0 9:0 0 36.4T 0 raid0 sdb 8:16 0 7.3T 0 disk └─sdb1 8:17 0 7.3T 0 part └─md0 9:0 0 36.4T 0 raid0 sdc 8:32 0 7.3T 0 disk └─sdc1 8:33 0 7.3T 0 part └─md0 9:0 0 36.4T 0 raid0 sdd 8:48 0 7.3T 0 disk └─sdd1 8:49 0 7.3T 0 part └─md0 9:0 0 36.4T 0 raid0 sde 8:64 0 7.3T 0 disk └─sde1 8:65 0 7.3T 0 part └─md0 9:0 0 36.4T 0 raid0 Now make a mount point and mount the volume: $ sudo mkdir /mnt/raid0 $ sudo mount /dev/md0 /mnt/raid0 Verify the mount shows up with df $ df -h Filesystem Size Used Avail Use% Mounted on ... /dev/md0 37T 24K 37T 1% /mnt/raid0 Make the mount persist If you don’t add the mount to /etc/fstab, it won’t be mounted after you reboot! First, get the UUID of the array (the value inside the quotations in the output below): $ sudo blkid ... /dev/md0: UUID="5d3b012c-e5f6-49d1-9014-1c61e982594f" TYPE="ext4" Then, edit /etc/fstab (e.g. sudo nano /etc/fstab) and add a line like the following to the end: UUID=5d3b012c-e5f6-49d1-9014-1c61e982594f /mnt/raid0 ext4 defaults 0 0 Save that file and reboot. Verify the mount persisted. After reboot: $ df -h Filesystem Size Used Avail Use% Mounted on ... /dev/md0 37T 24K 37T 1% /mnt/raid0 Managing devices in array Drop the array If you’d like to drop or remove the RAID array and reset all the disk partitions so you could use them in another array, or separately, you need to do the following: Edit /etc/fstab and delete the line for the /mnt/raid0 mount point. Edit /etc/mdadm/mdadm.conf and delete the lines you added earlier via mdadm | tee. Unmount the volume: sudo umount /mnt/raid0 Wipe the ext4 filesystem: sudo wipefs –all –force /dev/md0 Stop the RAID volume: sudo mdadm –stop /dev/md0 Zero the superblock on all the drives: sudo mdadm –zero-superblock /dev/sda1 /dev/sdb1 … At this point, you should have back all the drives that were part of the array and can do other things with them. Adding a device To add a new device to the array : # mdadm --add /dev/md0 /dev/sdd Removing a device We can fail a device (-f) from an array and then remove (-r) it: # mdadm --manage /dev/md0 -f /dev/sdd # mdadm --manage /dev/mdadm -r /dev/sdd Ref: https://www.jeffgeerling.com/blog/2021/htgwa-create-raid-array-linux-mdadm https://unix.stackexchange.com/questions/318098/mdadm-raid-implementation-with-gpt-partitioning/320330#320330 https://www.digitalocean.com/community/tutorials/how-to-create-raid-arrays-with-mdadm-on-ubuntu-22-04 https://www.thegeekdiary.com/redhat-centos-managing-software-raid-with-mdadm/#google_vignette Administrations Configurations (Linux) Linux RAIDLinux Soft RAIDLinux Software RAIDmdadm