ZFS, end to end: pools, datasets, snapshots, send/receive, scrub
ZFS is not just a filesystem. It’s a storage stack: a pool built from devices, datasets inside it, checksums on everything, cheap snapshots, replication and compression. The videos show the commands; this page explains why they work.
The model is different from ext4/XFS/Btrfs: you do not format a partition with
mkfs. You give devices to a pool, and create datasets inside it.
ZFS is not in the Linux kernel (CDDL licensing, unlike ext4/XFS/Btrfs), so it
comes from your distro: zfsutils-linux on Ubuntu (the module ships with
Ubuntu’s kernels); on Debian, zfs-dkms builds it against the running kernel.
Create the pool
sudo zpool create -f tank /dev/sdb
sudo zpool status tank
sudo zpool list
sudo zfs list
The pool tank is ONLINE; zpool list shows size and allocation; zfs list
shows the root dataset. -f forces the creation over an existing signature.
On a whole disk, zpool create writes a GPT with sdb1 (data) and sdb9 (a
small reserve) — ZFS does the partitioning step for you. For real systems, use
/dev/disk/by-id/... rather than /dev/sdb.
Datasets
Datasets are the filesystems inside the pool. Each one has its own mountpoint, properties and snapshots:
sudo zfs create tank/data
echo 'hello from zfs' | sudo tee /tank/data/hello.txt
sudo zfs list
No mount, no fstab: each dataset has a mountpoint property (default
/<pool>/<dataset>, so /tank/data), and ZFS mounts it on create and on pool
import (zfs-import-cache.service and zfs-mount.service on systemd). That is
also why zfs receive lands mounted. zfs get mountpoint,mounted tank/data
shows it; sudo zfs set mountpoint=/srv/data tank/data remounts it (set it back
with sudo zfs inherit mountpoint tank/data); sudo zfs unmount and
sudo zfs mount (or sudo zfs mount -a) do what they say.
The exception is mountpoint=legacy: ZFS leaves mounting to you, and you use
sudo mount -t zfs tank/data /mnt/x or an /etc/fstab entry — useful when
something else must control mount ordering (or for some root setups).
Snapshots and rollback
A ZFS snapshot is instant and cheap. It is a read-only point-in-time view of a dataset:
sudo zfs snapshot tank/data@snap1
echo 'changed' | sudo tee /tank/data/hello.txt
sudo cat /tank/data/hello.txt
sudo cat /tank/data/.zfs/snapshot/snap1/hello.txt # the old content
sudo zfs rollback tank/data@snap1
sudo cat /tank/data/hello.txt # old content again
The .zfs/snapshot/<name> path is how you read a snapshot without mounting
anything. zfs rollback discards the changes since the snapshot.
Send and receive
zfs send/receive is ZFS replication in its simplest form — and the primitive
behind backups and remote replication:
sudo zfs send tank/data@snap1 > /tmp/data.snap
sudo zfs receive tank/restore < /tmp/data.snap
sudo cat /tank/restore/hello.txt
Here the stream lands on the same pool for the demo; in practice it goes to another pool or host, for example:
sudo zfs send tank/data@snap1 | ssh backuphost sudo zfs receive backup/data
The receive side needs root (or delegated rights via zfs allow). A snapshot
on the same pool is not a backup. A snapshot sent somewhere else is.
Compression
Compression is a per-dataset property, applied on write (compression=zstd
needs OpenZFS 2.0 or later; older 0.8 systems have lz4):
sudo zfs set compression=zstd tank/data
sudo zfs get compression tank/data
sudo sh -c 'yes hello-zfs | head -c 20000000 > /tank/data/text.bin'
ls -lh /tank/data/text.bin
sudo zfs get compressratio tank/data
sudo zfs list -o name,used,logicalused,compressratio tank/data
The file is 20,000,000 bytes (19.1 MiB). zfs list shows USED 40K and
LUSED 19.1M, so the raw numbers are about 490x; the compressratio property
reports 980.22x for the dataset (including hello.txt), and it compares
compressed data, not the metadata in USED. Both are real; they measure
different things.
At ashift=12, a compressed 128 KiB block still takes at least 4 KiB, which
caps the ratio around 32x. This text is so repetitive that zstd compresses each
block below the 112-byte limit, and ZFS stores it inside the block pointer
itself (embedded_data) without allocating a data block. Real text and logs
land far lower, typically 1.5–3x.
Scrub
scrub reads every block and checks it against the checksums:
sudo zpool scrub tank
sudo zpool status tank
sudo zpool status -v tank
On a healthy pool: scrub repaired 0B ... with 0 errors, ONLINE, and
No known data errors. Debian/Ubuntu schedule scrubs with cron
(/etc/cron.d/zfsutils-linux, monthly); on OpenZFS 2.1.3+ you can also enable
zfs-scrub-monthly@tank.timer.
What ZFS can’t do
- Shrink. A pool can’t shrink in place.
zpool removecan evacuate a whole single-disk or mirror vdev (only in pools without raidz vdevs), which shrinks the pool; otherwise the answer is backup, recreate the pool smaller, restore. Datasets have no fixed size to shrink: they share the pool, and you can lower a quota or reservation at any time. Plan capacity before you build the pool. - Snapshots are not backups. They share the pool; lose the pool and the
snapshots go with it.
zfs sendis what turns a snapshot into a backup. - ARC uses RAM. ZFS caches aggressively; that is usually a feature, but it is
why a ZFS host wants memory.
arc_summary(or/proc/spl/kstat/zfs/arcstats) shows what it is doing.
What to remember
- Pool first, then datasets; grow the pool (for larger disks,
sudo zpool set autoexpand=on tankorsudo zpool online -e tank /dev/sdb), datasets follow. - Snapshots and
send/receiveare the replication story. - Compression is per dataset; scrub is per pool. Both are visible.
- A pool can’t shrink in place (except
zpool removeof a single/mirror vdev, only in pools without raidz); datasets have no fixed size — quota and reservation are adjustable.
Part of Filesystems, end to end.