Install Ubuntu Server with ZFS

[whitetiger-it]

I have this old server with all the disks in HBA mode.
I want to install ZFS on the two boot HDDs, either mirrored or on four in RAIDZ2.
However, I can’t find any options to do this.
I can only create a mirror with an md0 RAID and then in ext4.
Furthermore, by doing this (and I fear also with ZFS), I can’t define the RAID as the boot disk and am forced to use a USB stick partitioned as /boot.

Ubuntu Version:
Example: 25.10

Problem Description

Relevant System Information:
HP-DL380 G8 HDD SAS in HBA mode

[Actionparsnip]

25.10 doesn’t have much support. I can suggest you install with 26.04 which is LTS and has support for much longer

[Actionparsnip]

Does the hardware not support creating the array and then present a single, Raided, disk to the OS? Is this not an ability of the disk hardware in the case?

[aljames]

Installing Linux root (not just Ubuntu) on ZFS is not native to most installers due to licensing conflicts, and therefore not fully supported in the kernel. There are manual methods available that you can research to accomplish what you want. They are complex, and can get you a working system however you’ll need to be able to manage or fix your own boot when an update breaks it. All possible, but you’ll need to go elsewhere for this type of support.

Try the OpenZFS project
There is also ZFSBootMenu

Both well documented and cover this topic thoroughly.

I have a host that runs root on ZFS using a Mirror VDEV. It is maintainable, but there have been, and will be more, hitches along the way.

What I do mostly now is run root on ext4+LVM, and then just use ZFS datasets for virtual machines so that I can use snapshotting & send/recv for replications (backups), & rollbacks for VMs.

The reason I’m sticking with native file systems for Linux root now, is it’s quite trivial to reinstall the OS these days, and most of the hard work I do is in VMs. My use case may not be yours, so there is that.

[peterwhite23]

According to this spec document for the HP Smart Array P420 Controller, which comes with the DL380, IIUC, it has a battery-backed write cache, which should eliminate the dreaded “write hole”; being immune to the latter is one of the key selling points for ZFS. But, since there no write hole and because ZFS on Linux will forever be an out-of-tree add-on, you may want to reconsider your decision, as @Actionparsnip suggest.
BTRFS atop a native RAID-1 with battery backup could be a viable option. While the parity-based redundancy is still problematic, BTRFS is rather solid in simple non-redundant mode and “RAID” 1. I’d also recommend a separate /boot on ext4 to go with the BTRFS partition, which essentially means you can just use the bog standard Ubuntu installer to get it running. Just need to enable RAID-1 on your disk controller and hand the mirror device as install target to the installer.

[rubi1200]

@Actionparsnip

I can suggest you install with 26.04 which is LTS and has support for much longer

Moderator note to ALL users.

While, generally speaking, we do recommend installing LTS versions for security and stability, please bear in mind the following:

• 26.04 is currently still in Beta
• RC images have not been released yet
• Another rule of thumb is to wait for the first point release (usually in August) to allow for bugs and other kinks to get worked out

Installing 26.04 now might fix the issues and then again it might not since there is still much that can change over the coming days.

Proceed with caution and ensure you have solid backups at all time.

[arshiarasekhi]

Thanks for the pointers to OpenZFS and ZFSBootMenu; those are the right tools for this. One thing worth adding: if the disks are in HBA passthrough mode (which is what the OP mentioned), the ZFS native mirror VDEV approach works cleanly since ZFS gets direct access to the physical drives rather than going through a hardware RAID layer.

The workflow that tends to work well is: install Ubuntu minimal on a USB/small drive first, then use zpool create it to set up the mirror pool on the two boot HDDs, and finally use ZFSBootMenu to handle the EFI boot entries. It’s extra steps compared to the installer GUI, but it gives you full ZFS send/recv capability for snapshots and rollbacks on the root filesystem too, which is the real win.

The ubuntu-advantage ZFS support in 24.04 also makes kernel module updates a lot less painful than they used to be on earlier releases, so the “update breaks boot” risk James mentioned is somewhat reduced now.

[peterwhite23]

@arshiarasekhi

Thanks for the pointers to OpenZFS and ZFSBootMenu; those are the right tools for this. One thing worth adding: if the disks are in HBA passthrough mode (which is what the OP mentioned), the ZFS native mirror VDEV approach works cleanly since ZFS gets direct access to the physical drives rather than going through a hardware RAID layer.

Not with this:

The workflow that tends to work well is: install Ubuntu minimal on a USB/small drive first, then use zpool create it to set up the mirror pool on the two boot HDDs, and finally use ZFSBootMenu to handle the EFI boot entries. It’s extra steps compared to the installer GUI, but it gives you full ZFS send/recv capability for snapshots and rollbacks on the root filesystem too, which is the real win.

What ZFS wants is access to the raw disk devices, which is impossible with an EFI partition. And the gains are academic, IMHO.
There is also the problem with swapping, which doesn’t work too well when swap space is on ZFS. And before someone suggests doing away with swap altogether, think again. So a sane base config cannot be achieved with a pure ZFS setup anyway; one’ll need separate partitions for EFI and swap, anyway, so might as well just go with bog standard and supported setups.

I find it rather pointless to use a software mirror (ZFS) on a system that comes with rather expensive hardware for that. OP has payed for it, so might as well use it.

[arshiarasekhi]

Fair points, and thanks for the correction on the EFI limitation. You’re right that ZFS wanting raw block device access and the EFI partition requirement are fundamentally in tension, which is exactly why ZFSBootMenu exists as a workaround rather than a native solution. The swap issue is a real constraint, too.

On the hardware argument: the OP mentioned disks in HBA passthrough mode specifically, which suggests they may have intentionally bypassed the Smart Array controller to get raw disk access, in which case software-level redundancy via ZFS becomes the only option available. If the hardware RAID is active and reliable, then you’re absolutely right that ext4/BTRFS + hardware mirror It is the simpler and more supported path.

Either way, the conclusion is the same: separate EFI and swap partitions are unavoidable regardless of the filesystem choice, which makes a fully “pure ZFS” setup impractical, as you said.

[peterwhite23]

@arshiarasekhi

On the hardware argument: the OP mentioned disks in HBA passthrough mode specifically, which suggests they may have intentionally bypassed the Smart Array controller to get raw disk access, in which case software-level redundancy via ZFS becomes the only option available.

And he may have done so because everybody keeps suggesting to do that for ZFS purposes, which is getting it backwards all the time. The reason ZFS is the way it is, is because back when it was designed, such battery-backed RAID controllers were rare and very expensive, turning the idea of RAID (the “I” is for “inexpensive”!) on its head. So the Sun engineers came up with RAIDZ, to overcome the dreaded “write hole”, which only exists when there is no battery backup.

One can just as easily setup a non-redundant ZFS pool on an actual hardware RAID controller. And why wouldn’t one want to? If it checks all the boxes, which the one OP has seems to do, it’d be rather silly to reinvent it in software, aka RAIDZ. Also, said write hole is only relevant with parity-based RAID mode, i.e. RAID 5/6, so simple mirrors (RAID 1 or RAID 10) aren’t affected. And OP said something about a mirror.

Given all that, RAIDZ has not much to offer over hardware RAID. And if that’s the case, OP might want to reconsider, and simply (re-)enable the RAID mode of the disk controller. Especially when taking into account the drawbacks of ZoL in the actual real world, which is not exactly the same experience as native ZFS on Solaris, where it is at home among friends and family, and not an unwelcome step child as in Linux.

[arshiarasekhi]

That’s a really informative breakdown I wasn’t aware of the historical context around RAIDZ and battery-backed controllers, so thanks for that. You’re right that the write hole concern that originally motivated ZFS-over-HBA setups doesn’t apply here given the HP Smart Array has battery-backed write cache.

And the point about mirror vs RAIDZ is well taken since the OP specifically mentioned a mirror setup, RAIDZ’s parity protection advantages aren’t relevant anyway. Re-enabling the RAID controller and running a hardware mirror with a conventional filesystem does seem like the cleaner path for this hardware.

The ZoL caveat is worth keeping in mind too. Good to know for future reference I’ve been exploring ZFS mostly in VM/container contexts where the hardware RAID question doesn’t come up the same way.

[whitetiger-it]

My (refurbished) server is now in HBA mode and it’s not easy to switch back to controller-managed mode.
In any case, I doubt it’s possible to have two disks managed by the controller and the others in HBA mode; I think they’re all managed one way or another and if I want to manage ZFS then I’m forced to keep HBA.

The problem isn’t so much managing the operating system disks, but all the other ones that will be used for data.

Regarding OpenZFS, which I’m not familiar with, can I install it and then use it like I would with ZFS?
I need to create Samba folders and connect them with Nextcloud in SMB mode. Can I also see them as shared folders from my Windows and Xubuntu PCs?

Regarding ZFSBootMenu, thanks; I’ll try to find out more.

The 26.04 LTS will be released in a few days, but a month later than when I started, so I’m keeping 25.10 and will migrate when the new LTS is considered stable.

[whitetiger-it]

Sorry, but I don’t understand.
My ZFS version in Ubuntu Server 25.10 is 2.3.4-1ubuntu2 and this already appears to be native OpenZFS; furthermore, from what I understand, it’s installed by default.
So what are you recommending me regarding OpenZFS?

$ sudo dmesg | grep ZFS
[ 7.951429] ZFS: Loaded module v2.3.4-1ubuntu2, ZFS pool version 5000, ZFS filesystem version 5
[ 7.951434] ZFS: Using ZFS with kernel 6.17.0-22-generic is EXPERIMENTAL and SERIOUS DATA LOSS may occur!
[ 7.951436] ZFS: Please report your results at: ``https://github.com/openzfs/zfs/issues/new

[peterwhite23]

@whitetiger-it

My (refurbished) server is now in HBA mode and it’s not easy to switch back to controller-managed mode.

Can’t help you there, I’m afraid.

In any case, I doubt it’s possible to have two disks managed by the controller and the others in HBA mode;

The question is why you want that distinction to begin with.

I think they’re all managed one way or another and if I want to manage ZFS then I’m forced to keep HBA.

Why do you think that? ZFS will take any block device. A hardware-managed RAID mirror, or an actual parity-based array, for that matter, is just another (single) block device.

The problem isn’t so much managing the operating system disks, but all the other ones that will be used for data.

See above, why? If all disks are the same, you can just as easily have all of them in a single RAID block device and use that for a (Open)ZFS (on Linux) pool. The only difference being that the RAID controller handles all the redundancy and ZFS does the rest, or BTRFS, for that matter. Both filesystems do away with classic volume/partition management, so you can have as many datasets as you please – or subvolumes, in case of BTRFS --, all sharing the same storage pool; no more static partitioning/volumes, yay(!).
Don’t get me wrong, ZFS is a great filesystem, but it does come with some strings attached, when it is ZFS on Linux. Plus, Linux has come a long way since ZFS entered the scene. For instance, the ARC (Adaptive Replacement Cache) of ZFS once was one of its selling point, but with more recent developments in Linux, there really is no need for it, and it may even cause you some headaches, because, at least on Linux, it’s not as dynamically sized as advertised. Linux’s (multi-generational) LRU (least recently used) cache (file and anonymous memory) and shadow tracking are arguably better than ARC. The latter will, from my experience, not be able to shrink under memory pressure, or will do so too late, when the kernel has already had to start reclaiming memory to accommodate it.
Given all that, you may find that BTRFS combined with other Linux subsystems, can deliver a pretty good approximation of ZFS. But where BTRFS falls short is definitely, and will be for some time yet, parity RAID, i.e. modes 5/6. If that’s what you’re after and you must use HBA mode, then there is not much of an alternative to ZoL. But, since your controller does have battery backup, you could also just use Linux’s venerable md-raid for handling the redundancy and use BTRFS atop of it. It’s much more Unix-y that way and requires zero out-of-tree kernel modules.

Maybe tell us a bit more about the setup you have in mind, including all the disks/SSD’s involved.

Regarding OpenZFS, which I’m not familiar with, can I install it and then use it like I would with ZFS?

ZFS on Linux (ZoL) is OpenZFS; there is no other. But, as I’ve said, isn’t exactly all roses, given that it will never be mainlined.

I need to create Samba folders and connect them with Nextcloud in SMB mode. Can I also see them as shared folders from my Windows and Xubuntu PCs?

I don’t see a problem there.

The 26.04 LTS will be released in a few days, but a month later than when I started, so I’m keeping 25.10 and will migrate when the new LTS is considered stable.

Or you could seize the opportunity to experiment with the 25.10 and reinstall 26.04, when it’s released, which should be any day now. Waiting for the first point release is only recommended for release upgrades, because they are a bit trickier than fresh installations.

[whitetiger-it]

I’ll start from the bottom.
Sure; if I wait a little longer, 26.04 will be released, but the problem remains.
HBA vs RAID Controller: It’s pointless now because it’s configured as an HBA and to change it back to RAID I’d have to find some tools.
BTRFS vs ZFS: I want ZFS because I have an older system with older disks. Using ZFS, I can manage RAIDZ2 or RAIDZ3; if they bring one or two disks, I can continue working without having to rush to find a replacement disk.
In the meantime, I’m looking at BTRFS and maybe I’ll use it when RAID6 is more secure.
Anyway, the main issue I started the thread with isn’t how to use the other disks, but how to boot them because in HBA I’m now forced to use a USB stick dedicated to the /boot partition.
I haven’t had time to look at the suggested tool yet.

[peterwhite23]

@whitetiger-it

Sure; if I wait a little longer, 26.04 will be released, but the problem remains.

My point was, that this is your window of opportunity to try some things and don’t commit to anything, just yet. And by the time 26.04 gets released you’ll have gathered enough experience to install it from scratch with the lessons learned.

HBA vs RAID Controller: It’s pointless now because it’s configured as an HBA and to change it back to RAID I’d have to find some tools.

I don’t quite understand. Aren’t those tools included with the controller?

BTRFS vs ZFS: I want ZFS because I have an older system with older disks. Using ZFS, I can manage RAIDZ2 or RAIDZ3; if they bring one or two disks, I can continue working without having to rush to find a replacement disk.

I did not understand your intention, at first. Thanks for shedding some light on this. But, depending on how many disks you have, especially RAIDZ3 might be just a fancy way of doing RAID-10 with 6 disks for instance.
Suppose you have 6 disks in RAIDZ3. That’d only give you 3 actual data disks, just like RAID-10.

In the meantime, I’m looking at BTRFS and maybe I’ll use it when RAID6 is more secure.

Don’t hold your breath.

Anyway, the main issue I started the thread with isn’t how to use the other disks, but how to boot them because in HBA I’m now forced to use a USB stick dedicated to the /boot partition.

You might not need to, but it may also be good practice to have a separate /boot, because that’s what’s officially supported. This guide may offer you some pointers. Essentially that procedure has been incorporated into the current Ubuntu installer, IIRC. But more advanced setups, like an actual redundant pool, are out of scope.

I haven’t had time to look at the suggested tool yet.

Which one?

[whitetiger-it]

If I go into controller management, I should find an option to switch between RAID and HBA, but in my case, it’s not there, even though the firmware is up to date.
HPE support says this can happen on older machines like mine, and the only way to fix it is to use certain tools.
However, I repeat, from what I understand, it’s a controller function that doesn’t apply to individual disks, so either I have them all in HBA or I have them all in RAID.
With RAIDZ3 and RAID10, I waste the same amount of space, but with RAIDZ3, I can lose up to three disks and still work.
With RAID10, you only have to lose two disks in the same pair and you lose the entire volume.
ZFSBootMenu was suggested to me, but I haven’t had a chance to fully test its usefulness yet. From a first cursory reading, it seems like this is the solution I’m looking for.
With your link, I get the 404 error.

[peterwhite23]

@whitetiger-it

ZFSBootMenu was suggested to me, but I haven’t had a chance to fully test its usefulness yet. From a first cursory reading, it seems like this is the solution I’m looking for.

There once was official (experimental) support for that, but it wasn’t up to Canonical’s standards, so it was removed from main and not offered anymore as an installer option in newer versions, IIRC. I cannot remember any details, so just take that as a pointer for further research.

With your link, I get the 404 error.

Fun with the Discourse editor; it escaped the % in %20, again.
Fixed.