Enhancing our ZFS support on Ubuntu 19.10 - an introduction

Maybe I am wrong, but I took a snapshot of everything including the home directories in 18.04.3. I upgraded from 18.04.3 to 19.04 (as proposed by the Software Updater!) in the clean-up step they deleted over hundred files. I expect, there is a chance that files on which files in the snapshot depend also are deleted. I do not expect, that the 19.04 upgrade process knows anything about ZFS snapshots.
In the upgrade from 19.04 to the development edition through the CLI there is no clean-up step, so there is no snapshot issue.

My laptop OSes are now completely at 19.10 (1 x ext4 and 2 x zfs). In general I only detected minor issues and the more serious issues were related to my two phase update process from 18.04.3 to 19.04 to 19.10 of a relative complex multi boot PC. Iā€™m cleaning up my laptop now, deleting ā€œ1804ā€ from host-names and other important issues like that :slight_smile:
Next is the upgrade of my Ryzen desktop, but I probably wait for the installer. A clean install, because in April its disks including all installed systems were moved from a Phenom II X4 B97 to the Ryzen 3 2200G.

In general I am happy with the progress, Ubuntu is doing a good job with 19.10 and in my opinion you are back at the leading edge of Linux desktop innovation.

In the past I always did run ā€˜Ubuntu Cleanerā€™ or Synaptic to loose the deb files, old kernels, configs and unneeded packages. According to Ubuntu Cleaner I now have 1296 deb files, 3 old kernels, 2 package configs and 8 unneeded packages.
I think I can delete the 1 GB of deb files. For the others Iā€™m not sure.
I expect, if I want to loose the old kernels, package configs and unneeded packages, I have to destroy the older snapshots.
Are ā€˜Ubuntu Cleanerā€™, ā€˜apt autoremoveā€™, apt clean/autoclean and synaptic still safe to use or might I harm accidentally a snapshot?

In the boot menu all Ubuntu names are the same. In the past, I could distinguish them by partition id, do you intend to add instead of the partition e.g. the dataset name or host name?
The dataset name would be nice for ZFS, but the host name could be the future general solution for all file systems, also ext4, btrfs etc,

No, when you delete a file, a deb or cleaning for cleaning after upgrade, you are deleting them from the current dataset states. They are still present on the older snapshots (as long as you snapshotted all datasets). This is why the snashots will remain valid: if you revert all datasets to a particular snapshots, the files that you cleaned on latest states will be restored, because they were present when the snapshot was taken (which is the whole point of a snapshot).
So, the snapshots remains valid and you can revert to that point (again as long as you reverted all datasets ofc, including /boot for kernels). The only thing that is in a separate partition is /boot/grub (in our incoming new default installs), as there is only one valid per machine.

Installing side-by-side the same version of ubuntu twice is uncommon. We shows when you have multiple installations the version, so you will have:
Ubuntu 19.10
Ubuntu 18.04
for instance, to be able to differentiate them.

Even uncommon, showing the dataset can be helpful, when the exact same version is installed twice. Mind opening a bug against zsys as a reminder (not a high prio, but a nice to fix)?

I have been the whole day without electricity. That is the Caribbean; nice beaches, but sometimes somewhat too basic. Thanks for your reply. I think I start to understand the way the snapshots work and I remembered their explanation in one of the YouTube videos.
Just a remark about the same boot menu items. My two installations are ā€œUbuntu Mate 19.10ā€ and ā€œXubuntu 19.10ā€. If you display it like this, Iā€™m fine, but if you display both like Ubuntu 19.10 nothing is solved for me. To be sure I will write the bug report
I have some problems with both the history and advanced menu items, both did not work and I will file another bug report.

Better to start with desktop to test things out. Desktop has a lot of users and should there be any bugs it will be reported quickly and the damage will be pretty small. But if something goes wrong in a server farm lol, the damage is huge. So better to get things up and running on the desktop first :wink:
imo

1 Like

Hi,

I downloaded the nightly build of 19.10 to give zfs on root a try. The installer does not have zfs as an option, is there an easy way to try it out (I have used zfs for more than 10 years on servers.)

The installer option is still WIP (there is a merge proposal, but we revisit the design currently). Iā€™ll post here as soon as itā€™s on the daily ISO.

2 Likes

Thank you for the quick response. I will wait for your update.

Will the installer support to install ZFS rooted Ubuntu on a partition or will it require a full disk?

For now, it will require a full disk on the first iterations. (Then, you can install and extending, adding other disks on the pool).
Custom/advanced mode will probably be there in the future for a finer-grained control.

2 Likes

The installer limitation with respect to the use of partitions is new for me. For me partitions are important, since

  • I divide my desktop SSD between the boot datapool and L2ARC/LOG and
  • I have 2 datapools on my SSHD laptop, one for the system and Virtual Machines (VM) and one for data and large archives. I want to keep the systems and Virtual Machines in the begin of the SSHD, because the HDD throughput is often twice as high there!
    Booting from manually installed ZFS datapools on partitions worked fine during this year for 18.04, 19.04 and 19.10.
  1. Is the main limitation for the use of partitions in the installer?
  2. Can I install 19.10 on any existing zfs datapool? Or related, do you support dual and triple boots?

I can snapshot my system and rollback to such a snapshot using the terminal and boot again. So I can live with any limitations in the snapshot boot menu.

Iā€™m running FreeBSD on a 32-bits Pentium-4. That ancient PC is the backup-server (incremental send/receive) for especially my Ubuntu Virtualbox VMs. A question out of curiosity:

  1. Did you copy parts of the new software from FreeBSD-12, since I see the same limitations (partitions) and approach (datasets)?

  2. We are less then a week from the Beta-freeze and kernel-freeze, do you still expect to release it with 19.10?

When you do implement encryption, have you considered using zfs native encryption to encrypt individual home file systems with a key per user, replacing the functionality of ecryptfs?

1 Like

Indeed, itā€™s limited in the first installer implementation. However, nothing prevents you from doing your custom installation via debootstrap or any other form. The system itself handles whatever layout you wish. The guided/help installation (native to ubiquity) will be as described above though in its first round.

Those arenā€™t related.
As told, via a manual installation like debootstrap, you can install 19.10 on existing zfs datapool. However, I doubt your layout will be compatible with zsys and you lose all advanced features (basically, you have zfs as of today, and you need to handle it manually).

On the other questions, the advanced grub support we introduced in 19.10 have multiple system support, as you have already seen and discussed above.

No.

The installer option should be released once reviewed. We got final design last week (due to an issue in the ubiquity codebase preventing implementing previous design).

Zsys will stay in universe, meaning: not installed by default, but an option for those who wants to experiment with it.

There are 2 contestants in the room here:

  • lvm + luks + zfs. The underlying (lvm + luks) is already natively supported in ubuntu for TPM, so the goal would be to reuse the existing code. However, we plan for next week with the kernel team to have performance comparison so that we know where we stand at.
  • native zfs encryption, which means reimplementing the whole integration with various components (plymouth, grub, the initramfsā€¦), which is more work. Also the encryption is fresh from 0.8 and we need to evaluate how stable it is.
    Anyway, those items will be for the second phase, once zsys is a little bit more mature.

I think this maybe off-topic: ZFS On Linux 0.8.2 just released.

Since the Linux 5.3 is already in Eoan, will Eoan adapt this newer release as well?

1 Like

Just found the ZFS installer code got merged, congrats!

Looking forward to test it (in a VM first of course). :grin:

1 Like

That is good news. Anyone know how often they release a new installer iso.

Is it not released daily?

Good!! I already did give up on it, after I did see yesterday, that the grub boot from a ZFS snapshot had been removed from the grub menu.

thanks, I will use the desktop version. I was using the server install iso.