Zfs on Sysadmin Tales

Deduplicated Proxmox backups with Proxmox Backup Server and ZFS

Wed, 22 Apr 2026 23:23:10 +0000

Overview

There are many ways to back up your Proxmox virtual machines.

One of the most popular ways to do so is via Proxmox Backup Server, which gives you many advantages:

Deduplication of common data.
The ability to cherry pick files out of your backups.
Tight integration with the Proxmox ecosystem.

However, how is one to achieve proper 3-2-1 backups using this mechanism?

One way is to run multiple datastores, some local and some using Amazon S3 compatible endpoints.

However, this is a bit much for me, so I came up with a simpler method.

The Details

I run my PBS on a Beelink N100 mini PC, in which I’ve installed a 2TB SATA SSD that I formatted as a single disk zpool.

I then installed Jim Salter’s excellent Sanoid utility, configuring ZFS snapshots on the PBS server.

This is the /etc/sanoid.conf that I’ve defined. You can, of course, set whatever values you like.

[pbs-local/datastore]
        use_template = pbs
        recursive = yes

[template_pbs]
        frequently = 0
        hourly = 36
        daily = 3
        weekly = 1
        monthly = 0
        yearly = 0
        autosnap = yes
        autoprune = yes

Then, once I had my initial snapshots, I went to my TrueNAS backup server and created a new ZFS replication task.

I used this to pull the snapshots for the datastore to the backup server.

The key is to match the expected snapshot naming schema for Sanoid, like so:

This allows me to maintain the deduplicated nature of the Proxmox Backup Server datastore.

You can do the same with any offsite NAS that you may have, as long as it’s running ZFS. This is easily accomplished with something like Tailscale or Netbird.

If I ever have a failure of the Proxmox Backup Server, I can simply set up a new zpool on the repaired system and reverse the direction of the replication.

I also use ZFS Delegation to avoid doing the replication as root, but I’ve covered this topic before.

As always - trust nothing and test those backups!

TrueNAS Scale - ZFS Replication Without Root

Fri, 21 Mar 2025 15:15:20 -0400

Overview

ZFS replication is my favored way of doing backups. In my homelab I have 2 systems running TrueNAS scale - one on my production hypervisor, with the disk controller passed through, and the backup server, which is just a baremetal TrueNAS Scale installation.

Unfortunately, most of the tutorials on this process seem to use either the root account or at the very least, an account with admin privileges. In this article, I’m going to go over how you can get a ZFS replication setup working with a completely unprivileged user, without ever leaving the GUI.

I will be doing a pull replication - where the production system has NO access to the backup system. The backup system will reach out and initiate the backups.

That way, there’s no risk of the backup server being compromised because you had SSH credentials for it just lying around on production.

You should also lock down access to the backup system in other ways, but that’s beyond the scope of this article.

Create user accounts for backup

Go to Credentials > Users and create new users. Just create normal user accounts. Don’t give it any extra permissions. I also recommend unchecking the box that allows it to be used for SMB authentication.

Make sure that you change the login shell from “nologin” to “bash” or “zsh”.

I normally just call mine backup_user, but you can make it anything you want.

Create SSH credentials

On the backup system:

Click Credentials > Backup Credentials
Add an SSH Keypair. Give it a name. Click on generate keypair.
- This will generate a PUBLIC key and a PRIVATE key. The private key will be used on the backup system to verify the public key.
- I recommend copying the public key into a notepad or something. We’ll need it again in a moment.
- Click Save.
Add a new SSH Connection. Name it something that makes sense.
Setup method - manual
For host, enter the IP address of the production system. If your machine is offsite and you have them both connected via Tailscale, you can use the Tailscale IP here, even if the machine is local. Tailscale connections are designed to be peer-to-peer, so it should stay in the same local network and get the full wire rate.
For username, change it to the username of the user you created on the REMOTE system. That is, the PRODUCTION system.
Select the private key you created in step 2.
Click “Discover remote host key”.
Click Save.

On the production system:

Navigate to Credentials > Users. Edit the user you created for your backup tasks.
Take the public key you copied earlier and paste it into the Authorized Keys field. (Or upload it if you saved it as a file.)
Click save.

Giving users needed permissions

Because we’re not using the root user, we need to give our backup users the ability to run certain commands with sudo, without a password. None of these commands can be used destructively.
Navigate back to Credentials > Users. Edit the backup user.

On the backup system:

Type the following into the “Allowed sudo commands with no password” box. The asterisks are wildcards.

/sbin/zfs mount *
/sbin/zfs create *
/sbin/zfs receive *

On the production system:

Type the following into the “Allowed sudo commands with no password” box. The asterisks are wildcards.

/sbin/zfs send *
/sbin/zfs snapshot *
/sbin/zfs list *
/sbin/zfs get *

This will allow a completely unprivileged user to run the backups for us.

Creating the replication task

On the backup system:

Navigate to Data Protection > Replication Tasks. Click Add.
For the source location, choose “On a different system”.
Choose the SSH connection we created earlier.
- When prompted if you would like to use sudo for ZFS commands, allow it. This is needed because the user we’re doing this operation with does not have admin privileges.
For the source, choose the dataset that you would like to back up. I usually also check off the recursive box, though this has no effect unless you’ve created nested datasets.
For the destination, choose either the pool itself or the dataset ABOVE the one you want to replicate to. The dataset CANNOT exist prior to the first replication. It will fail if you create it in advance. You’ll need to type in the name of the dataset in the text box. This is slightly unintuitive, I know. I don’t know why they did it this way.
Choose a schedule and a retention policy.
Run your first replication. The first one will take a long time if there is a lot of data to copy, but subsequent runs will take only as much time as it takes to send the changed blocks down the wire.

Keep in mind when choosing your retention policies on the source and destination systems that the source and destination systems NEED to have at least one snapshot in common!

Happy replicating!

Fedora 41 and ZFS

Mon, 09 Dec 2024 20:40:18 -0500

Note

ZFS 2.2.7 has been released with up to Linux kernel 6.12 support, so this workaround is no longer necessary. I’ll leave the post up in case this is ever useful in the future.

I previously posted about how to build a desktop OS from the Fedora Server installer.

This is a bit of a follow up to that post.

I’ve now been running Fedora 41 for a few days and it’s been a mostly pleasant experience.

However, I am also an avid ZFS user, and the current stable release of OpenZFS does not support Linux kernel 6.11 yet.

So, being the enterprising fellow that I am, I chose to build ZFS 2.3.0 RC3 from source using the official instructions

That went well, but one thing that tripped me up (and caused some issues for specific applications like syncthing, which keeps it’s local database in the user’s home directory) is that when you do this, unlike when you install zfsutils-linux on Ubuntu, it doesn’t set up the automatic import and mounting of your pool.

It took some digging around to find the correct process, because it seems to be a bit different between the official ZFS source and the one that ships with Ubuntu, but here’s what wound up working for me:

First, go through and enable the following services:

sudo systemctl enable zfs.target zfs-mount.target zfs-import.target zfs-import-cache.service

There are plenty of posts online telling you that you should create your cache file in /etc/zfs/zfs-list-cache/, but this is wrong (or at least outdated. I’m not sure if that location is Ubuntu specific?)

The ACTUAL location of the cache file (which you can see if you actually read through the systemd unit file for zfs-import-cache.service) is /usr/local/etc/zfs.

If the file doesn’t exist, create it.

sudo touch /usr/local/etc/zfs/zpool.cache

Import your pool.

sudo zpool import poolname

If you print out the cache file now it should no longer be empty.

cat /usr/local/etc/zfs/zpool.cache

Next time you reboot your pools should automatically mount and mount all filesystems.

And yes, I did cross post this over on the PracticalZFS forum.