Replace a failed disk in Linux software RAID (mdadm)

Most dedicated servers with two disks run Linux software RAID 1: both disks hold the same data. When one fails, the server keeps working on the other, and the array is degraded. This guide takes you from the warning to a healthy mirror again. For the day-one checks of your RAID and disks, see dedicated servers.

Before you start

• Root access over SSH, and the server's IPMI console if your server has one (IPMI console and virtual media).
• A current backup. During a degraded state and the rebuild, your data exists only once; see backup strategy.
• In the examples the healthy disk is /dev/sda, the failed one /dev/sdb, and the arrays are /dev/md0 (partitions 1) and /dev/md1 (partitions 2). Use the names from your own output.

Step 1: Confirm which disk failed

cat /proc/mdstat
sudo mdadm --detail /dev/md0

In /proc/mdstat, a healthy RAID 1 shows [UU]; a degraded one shows [U_] or [_U], and a failed member is marked (F). mdadm --detail shows State : clean, degraded and lists the faulty or removed device.

Note the serial number of the failed disk, so the right disk is replaced:

lsblk -o NAME,SIZE,MODEL,SERIAL
sudo smartctl -i /dev/sdb

If the failed disk no longer answers, note the serial of the healthy disk instead and say so in the ticket.

Step 2: Remove the failed disk from every array

A disk usually has a partition in each array. Mark each one as failed and remove it:

sudo mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
sudo mdadm /dev/md1 --fail /dev/sdb2 --remove /dev/sdb2

Verify: cat /proc/mdstat lists only sda partitions in each array.

Step 3: Ask us to replace the disk

Open a ticket under Support › Tickets › Open Ticket, choose the dedicated server as Related Service, and write:

• that a RAID member failed and has been removed from the arrays;
• the serial number (and model) of the failed disk, or of the healthy disk to keep;
• the output of cat /proc/mdstat;
• when a replacement suits you.

We confirm the time and the procedure with you; see open a support ticket.

Step 4: Copy the partition layout to the new disk

After the replacement, find the new, empty disk with lsblk (no partitions, same size or larger). Check the partition table type of the healthy disk:

sudo fdisk -l /dev/sda | grep -i "disklabel type"

sudo sgdisk -R /dev/sdb /dev/sda
sudo sgdisk -G /dev/sdb

sudo sfdisk -d /dev/sda | sudo sfdisk /dev/sdb

Verify: lsblk shows the same partitions on sdb as on sda.

Step 5: Add the new partitions and let the arrays rebuild

sudo mdadm /dev/md0 --add /dev/sdb1
sudo mdadm /dev/md1 --add /dev/sdb2
watch cat /proc/mdstat

/proc/mdstat shows recovery with a progress bar and an estimated finish. Leave watch with Ctrl + C; the rebuild continues in the background.

Verify: when the rebuild is done, every array shows [UU] and mdadm --detail reports State : clean.

Step 6: Make the new disk bootable

The boot loader lives outside the arrays, so put it on the new disk too:

sudo grub-install /dev/sdb

sudo dd if=/dev/sda1 of=/dev/sdb1 bs=1M

Troubleshooting

mdadm: cannot find /dev/sdb1: No such file or directory when removing. The failed disk has vanished from the system. Remove the missing member with sudo mdadm /dev/md0 --remove detached (or failed), then continue.

The rebuild is very slow. Rebuilds share the disks with normal traffic. Avoid heavy disk work until it finishes.

The server does not boot after the replacement. The boot loader is missing on the disk the firmware tries first. Use the IPMI console to choose the other disk, then complete Step 6.

A second disk reports errors during the rebuild. Stop and open a ticket immediately with cat /proc/mdstat, mdadm --detail and dmesg output; do not force anything before your data is safe.

Next steps

• Check disk health regularly with smartctl -H on each disk; see dedicated servers.
• Keep restores possible: backup strategy.