Just a few days ago I was turning my single disk ZFS pools into mirrors. By mistake I created a top-level vdev, so I had to-recreate the pool. In this post I’ll cover how to restore the data from a snapshot and how to create a mirror from single disk.
I added a disk to my ZFS pool because I wanted to create a mirror. However, instead of creating a mirror I added that disk so that it became another top level vdev. Removing this one will not work:
root@debian:~# zpool remove zboot /dev/disk/by-id/ata-ADATA_SP900_xxx-part3
cannot remove /dev/disk/by-id/ata-ADATA_SP900_xxx-part3: operation not supported on this type of poolzpool detach won’t work neither. So I re-created the pool and restored from snapshot. For this create the snapshot first:
zfs snapshot -r zboot@backupThen send the snapshot to a file. However, this file should obviously NOT be on the pool which you’re going to destroy. I store it on another pool. You may also send to an external disk. In my case since just boot was involved it’s not a lot of required space.
root@debian:/home/jean# zfs send -cvR zboot@backup | gzip > zboot.backup.gz
full send of zboot@backup estimated size is 42.6K
full send of zboot/BOOT@backup estimated size is 42.6K
full send of zboot/BOOT/debian@backup estimated size is 271M
total estimated size is 271M
TIME SENT SNAPSHOT zboot@backup
TIME SENT SNAPSHOT zboot/BOOT@backup
TIME SENT SNAPSHOT zboot/BOOT/debian@backup
00:10:55 20.5M zboot/BOOT/debian@backup
00:10:56 37.5M zboot/BOOT/debian@backup
00:10:57 52.7M zboot/BOOT/debian@backup
00:10:58 68.4M zboot/BOOT/debian@backup
00:10:59 80.8M zboot/BOOT/debian@backup
00:11:00 92.5M zboot/BOOT/debian@backup
00:11:01 106M zboot/BOOT/debian@backup
00:11:02 120M zboot/BOOT/debian@backup
00:11:03 135M zboot/BOOT/debian@backup
00:11:04 149M zboot/BOOT/debian@backup
00:11:05 160M zboot/BOOT/debian@backup
00:11:06 174M zboot/BOOT/debian@backup
00:11:07 186M zboot/BOOT/debian@backup
00:11:08 200M zboot/BOOT/debian@backup
00:11:09 212M zboot/BOOT/debian@backup
00:11:10 216M zboot/BOOT/debian@backup
00:11:11 229M zboot/BOOT/debian@backup
00:11:12 242M zboot/BOOT/debian@backup
00:11:13 258M zboot/BOOT/debian@backupfine? Fine. Kill zboot.
root@debian:/home/jean# zpool destroy zbootRe-create the pool… in my case:
root@debian:/home/jean# zpool create \
-o ashift=12 \
-o autotrim=on \
-o compatibility=grub2 \
-o cachefile=/etc/zfs/zpool.cache \
-O devices=off \
-O acltype=posixacl -O xattr=sa \
-O compression=off \
-O normalization=formD \
-O relatime=on \
-O canmount=off -O mountpoint=/boot zboot /dev/disk/by-id/ata-ADATA_SP900_xxx-part3Mind that I could have created a mirror here by specifying [..] mirror disk1-partX disk2-partX directly. For the purpose of this post I’ll show how to properly add the second device into an existing pool to turn it into a mirror later in this post, though. Also mind that I was using compression=off here – In the ZFS on Root documentation they also use compression and it might work for you – I had trouble since this is the boot pool and because I am using grub. Hence =off here.
Now restore using the snapshot:
root@debian:/home/jean# gzcat zboot.backup.gz | zfs recv -v -d -F -o compression=off zboot
receiving full stream of zboot@backup into zboot@backup
received 43.9K stream in 0.11 seconds (388K/sec)
receiving full stream of zboot/BOOT@backup into zboot/BOOT@backup
received 43.9K stream in 0.05 seconds (818K/sec)
receiving full stream of zboot/BOOT/debian@backup into zboot/BOOT/debian@backup
received 272M stream in 14.81 seconds (18.3M/sec)Let’s take a look at zpool status:
pool: zboot
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zboot ONLINE 0 0 0
ata-ADATA_SP600_xxx-part3 ONLINE 0 0 0And zfs list:
root@debian:/home/jean# zfs list | grep zboot
zboot 272M 560M 96K /boot
zboot/BOOT 271M 560M 96K none
zboot/BOOT/debian 271M 560M 271M /bootNow mount everything using e.g. zfs mount -a and since this is the boot-pool in my case, check if update-grub still works:
root@debian:/home/jean# update-grub
Generating grub configuration file …
Found background image: .background_cache.png
Found linux image: /boot/vmlinuz-6.7.12+bpo-amd64
Found initrd image: /boot/initrd.img-6.7.12+bpo-amd64
Found linux image: /boot/vmlinuz-6.1.0-21-amd64
Found initrd image: /boot/initrd.img-6.1.0-21-amd64
Adding boot menu entry for UEFI Firmware Settings …
doneYou may get:
/usr/sbin/grub-probe: error: compression algorithm inherit not supportedThis seems to be a bug. People reported that creating the pool without compression as well as receiving the snapshot with -o compression=off will solve this – in my case it did not. I had to use grub from backports (apt-get install -t bookworm-backports grub-pc) because with version 2.12-1~bpo12+1 it works. Interestingly, if you following the debian bookworm zfs on root guide – compression works. So there seems to be some other problem. However, using backports fixes this for me.
Finally.. Add the second disc to form a mirror….
root@debian:/home/jean# zpool attach zboot /dev/disk/by-id/ata-ADATA_SP600_xxx-part3 /dev/disk/by-id/ata-ADATA_SP900_xxx-part3and check the status:
pool: zboot
state: ONLINE
scan: resilvered 274M in 00:00:04 with 0 errors on Tue Jun 11 00:34:00 2024
config:
NAME STATE READ WRITE CKSUM
zboot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ADATA_SP600_xxx-part3 ONLINE 0 0 0
ata-ADATA_SP900_xxx-part3 ONLINE 0 0 0By the way.. one of your devices is with no reason called sdX instead of using the name from /dev/disk/by-id/ and you’re luckily using a mirror? Then you can detach the disk which is using the /dev/sdX name and attach it again using it’s id from /dev/disk/by-id/:
root@debian:/etc/zfs# zpool detach zroot /dev/sdd5
root@debian:/etc/zfs# zpool attach zroot /dev/disk/by-id/ata-ADATA_SP900_xxx-part5 /dev/disk/by-id/ata-ADATA_SP600_xxx-part5Check the status, you should see a resilver in progress:
pool: zroot
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Jun 11 00:47:55 2024
4.10G / 5.33G scanned at 600M/s, 0B / 5.33G issued
0B resilvered, 0.00% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ADATA_SP900_xxx-part5 ONLINE 0 0 0
ata-ADATA_SP600_xxx-part5 ONLINE 0 0 0