So no, ‘recovery mode’ is still using the dataset.
Then I guess booting a live cd and importing rpool for changes might be the way.
A previous system state is currently selected using the grub menu during boot. This doesn’t work as well for a headless server where there is no keyboard or monitor.
Dropbear SSH server can be used to run an unlock script to allow a password to be entered for encrypted installs remotely. Is there any way to do the same to revert a system state remotely in case a system becomes unbootable?
Is there something else that we can do so that zsys does not interfer with a persistent data set? (like flagging it with a property?)
In your article on layout (https://didrocks.fr/2020/06/16/zfs-focus-on-ubuntu-20.04-lts-zsys-dataset-layout/) you mention that we just need to be outside of /ROOT/, /BOOT/, or /USERDATA/ namespaces. What I have tried (naively?) on Ubuntu 20.04, is that have a dataset rpool/backups that receives (via zfs send/recv) backups from other machines. I expect it to be a persistent datasets that zsys does not need to consider because it is located at rpool/ level. Inside I’ll have a structure /rpool/backups/machineX/bpool/..., /rpool/backups/machineX/rpool/.... Something I realized is that when I install a package on my machine, I get a autozsys_xxx snapshot created within my backup datasets. Is it because there is a rpool/backups/machineX/rpool/USERDATA/username_uuid dataset which contains /USERDATA/ ?
Oh, this is interesting and not expected at all. You are right, /rpool/backups/ and any child (whatever their name or parent dataset name) are expected to be considered as persistent and thus, not handled by ZSys. I guess the regexp due to /USERDATA/ is to blame there.
Mind opening a bug against the github project? I will try to reproduce it next week and will follow up there.
1 Like
ZFSBootMenu offers remote access via SSH to unlock native ZFS encryption during boot, or to rollback the system to a previous state if needed after a bad upgrade.
I’ve created an install script here for those that need this functionality while waiting for zsys to offer it.
https://www.reddit.com/r/zfs/comments/mj4nfa/ubuntu_server_2104_native_encrypted_root_on_zfs/
I am stuck with ubuntu/zsys issue: https://github.com/ubuntu/zsys/issues/204
I need a work-around to get back to a normal system.
I am not sure what the implications of the system saying it is adding items to the grub menu, referring to snapshots that couldn’t be made.
Requesting to save current system state
ERROR couldn't save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.
Free space on pool "bpool" is 15%
...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
ZSys is adding automatic system snapshot to GRUB menu
Press Return to continue, 'q' followed by Return to quit.
Some basic stats that may help:
# zfs list bpool
NAME USED AVAIL REFER MOUNTPOINT
bpool 1.61G 145M 96K /boot
~# zsysctl -vv machine show
DEBUG /zsys.Zsys/MachineShow() call logged as [3ca665c6:cdfd449a]
DEBUG Check if grpc request peer is authorized
DEBUG Authorized as being administrator
INFO Retrieving information for machine rpool/ROOT/ubuntu_g3snm5
Name: rpool/ROOT/ubuntu_g3snm5
ZSys: true
Last Used: current
History:
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_rusfce
Created on: 2021-05-12 11:54:08
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_5mfxzy
Created on: 2021-05-12 11:54:08
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_cig1rh
Created on: 2021-05-07 14:34:07
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_yacith
Created on: 2021-05-06 15:16:41
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_6dy52p
Created on: 2021-05-06 15:16:04
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_7egdeg
Created on: 2021-05-06 15:02:17
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_pfcytp
Created on: 2021-05-06 14:57:19
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_lkc5ed
Created on: 2021-05-06 14:55:30
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_6u8mgj
Created on: 2021-05-06 10:33:13
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_345siz
Created on: 2021-05-04 16:59:07
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_ugotwg
Created on: 2021-05-04 12:18:48
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_3n659l
Created on: 2021-05-04 11:14:41
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_rn8yvd
Created on: 2021-03-27 19:09:06
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_nv89hg
Created on: 2021-03-27 15:32:05
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_nfcmdl
Created on: 2021-03-25 10:01:24
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_owb6wt
Created on: 2021-03-05 11:39:57
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_hf2bjy
Created on: 2021-03-05 11:09:34
- Name: rpool/ROOT/ubuntu_ttmtqb@autozsys_tk45hl
Created on: 2021-03-04 08:53:55
- Name: rpool/ROOT/ubuntu_ttmtqb
Created on: 2021-03-04 08:00:01
- Name: rpool/ROOT/ubuntu_ttmtqb@autozsys_h1ffq2
Created on: 2021-03-04 07:58:24
- Name: rpool/ROOT/ubuntu_g3snm5@autozsys_q6ixmj
Created on: 2021-03-04 07:27:42
Users:
- Name: peterg
History:
- rpool/USERDATA/peterg_npbufz@autozsys_7n8cvm (2021-05-06 09:46:00)
- rpool/USERDATA/peterg_npbufz (2021-05-06 09:45:04)
- rpool/USERDATA/peterg_npbufz@autozsys_345siz (2021-05-04 16:59:08)
- rpool/USERDATA/peterg_npbufz@autozsys_utlaph (2021-05-04 16:32:28)
- rpool/USERDATA/peterg_npbufz@autozsys_svb4r8 (2021-05-04 15:31:28)
- rpool/USERDATA/peterg_npbufz@autozsys_j1vf2z (2021-05-04 14:30:28)
- rpool/USERDATA/peterg_npbufz@autozsys_x8ckln (2021-05-04 13:29:28)
- rpool/USERDATA/peterg_npbufz@autozsys_e42xww (2021-05-04 12:28:28)
- rpool/USERDATA/peterg_npbufz@autozsys_ugotwg (2021-05-04 12:18:49)
- rpool/USERDATA/peterg_npbufz@autozsys_3xtek8 (2021-05-04 11:28:09)
- rpool/USERDATA/peterg_npbufz@autozsys_3n659l (2021-05-04 11:14:42)
- rpool/USERDATA/peterg_npbufz@autozsys_smxhmb (2021-05-04 11:08:37)
- Name: root
History:
- rpool/USERDATA/root_npbufz (2021-05-06 09:45:04)
- rpool/USERDATA/root_npbufz@autozsys_345siz (2021-05-04 16:59:08)
- rpool/USERDATA/root_npbufz@autozsys_ugotwg (2021-05-04 12:18:49)
- rpool/USERDATA/root_npbufz@autozsys_3n659l (2021-05-04 11:14:41)
- rpool/USERDATA/root_npbufz@autozsys_rn8yvd (2021-03-27 19:09:06)
- rpool/USERDATA/root_npbufz@autozsys_nv89hg (2021-03-27 15:32:06)
- rpool/USERDATA/root_npbufz@autozsys_nfcmdl (2021-03-25 10:01:25)
- rpool/USERDATA/root_npbufz@autozsys_owb6wt (2021-03-05 11:39:57)
- rpool/USERDATA/root_npbufz@autozsys_hf2bjy (2021-03-05 11:09:35)
- rpool/USERDATA/root_npbufz@autozsys_tk45hl (2021-03-04 08:53:55)
- rpool/USERDATA/root_npbufz@autozsys_h1ffq2 (2021-03-04 07:58:24)
- rpool/USERDATA/root_npbufz@autozsys_xql5mz (2021-03-04 07:44:35)
- rpool/USERDATA/root_npbufz@autozsys_qhcolg (2021-03-04 07:42:33)
- rpool/USERDATA/root_npbufz@autozsys_q6ixmj (2021-03-04 07:27:43)
- rpool/USERDATA/root_npbufz@autozsys_rb61xw (2021-02-23 09:24:12)
- rpool/USERDATA/root_npbufz@autozsys_wvlrd3 (2021-02-21 14:05:32)
- rpool/USERDATA/root_npbufz@autozsys_qn3fab (2021-02-21 14:05:31)
- rpool/USERDATA/root_npbufz@autozsys_sbippd (2021-02-18 08:16:29)
- rpool/USERDATA/root_npbufz@autozsys_bzadk6 (2021-02-12 09:58:01)
- rpool/USERDATA/root_npbufz@autozsys_92lz5x (2021-02-12 09:52:30)
This particular entry seems to be atypical: - rpool/USERDATA/peterg_npbufz (2021-05-06 09:45:04)
Thanks,
–Peter G
1 Like
What is actually the current state of Root on ZFS for Ubuntu? There’s a project specification posted on https://docs.google.com/document/d/1m9VWOjfmdbujV4AQWzWruiXYgb0W4Hu0D8uQC_BWKjk/edit but that document seems to never have evolved beyond the initial draft stage. Will there be an official specification for 22.04 LTS? For the Server edition?
1 Like
Is there a way to revert state without using grub ?
I’m thinking about the use case for headless machines, where the grub menu is not accessible to the user.
Ideally, I want to run zfs on my Odroid and/or Raspberry Pi single board computers in headless mode. I often experiment on those and mess them up in some way, but almost never to the point that they don’t boot. The backup/restore process is currently very painful - consisting of physically pulling the SD card, and using Win32diskimager on Windows to read/write the whole card. This will surely wear out the flash quickly.
The pre-built OS images for SBCs unfortunately don’t use ZFS, but ext4 instead. I’m wondering if it would be possible to have pre-built Ubuntu images that use ZFS instead, for installation on SD cards. And then use zsys to make snapshots, and restore them. The reliance on grub is unfortunately a non-starter for SBCs.
Hoping it’s just me doing something incorrect.
Reverting is failing for me.
What I’ve done:
- Install Minimal-Ubuntu from USB, choosing experimental ZFS on root.
- Booted, and saw that the system is working as expected.
- Saved manual state with ‘zsysctl state save --system’
- Rebooted into grub menu, choose to restore system and user dataset.
The boot fails, drops me into emergency mode.
using jourlantctl -xb there is a error after the ZFS module has successfully loaded:
ERROR: zfs-mount-generator failed and you requested a revert:
unknown: level=error msg="couldn't ensure boot: Mounted clone bootFS dataset created by initramfs doesn't have a valid _suffix (at least .*_<onechar>): \"rpool/ROOT/ubuntu_\""
unknown: You can reboot on current master dataset to fix the issue
systemd[857]: /usr/lib/system-generators/zfs-mount-generator failed with exit status 1.
a few lines after that, the Emergency Shell starts.
Listing with zfs list I noticed there is a new dataset named: rpool/ROOT/ubuntu_.
The dataset is mounted, with a canmount=noauto.
This dataset is also a source of contention if I try to reboot and attempt a revert again.
I need to destroy it and attempting another revert, brings me to the same situation.
Running on an NVMe drive in an Alienware desktop from 2016, UEFI boot mode, with Secure boot enabled and keys reset to factory default (that’s how I installed the OS).
Installed using the Ubuntu 20.10 Desktop ISO on an E2B USB key.
Questions I’ve already asked myself and answered:
- Same behavior for auto created states, not just manually created ones.
Any suggestions on troubleshooting this would be really appreciated.
Because the current state of my system is unusable, I installed Zorin OS instead (it’s based on Ubuntu 20.04) and the process above worked as expected, no issues.
So whatever it is it seems to be specific to Ubuntu 20.10 right now.
If anyone has any insight, I’d love to hear it.
Posted the following question in askubuntu, hoping someone here might have an answer.
I decided to try 20.04 with ZFS. I was happy with it till I shortly ran out of disk space. It seemed my workflow of processing large video files and installing packages caused ZFS to eat up my SSD space quickly. Had to delete all snapshots on /home to be able to work. In a similar vein I came to the place I couldn’t update grub because the tiny boot partition was full of snapshots.
Having so many partitions with set headroom creates multiple failure points.
So ZFS is adding a whole new layer to the problem of managing disk space.
At the risk of waking a sleeping topic, I’m wondering: in the case of a kernel upgrade, or something that triggers an initramfs update, how does zsys (or zfs for that matter) protect the non-zfs boot partitions? Things like /root/efi, aren’t items like ubuntu/grub.cfg interdependent with things in rpool, etc?
In the past at one point I had a failure (I think grub-customizer was to blame) where my ZoL system failed to boot; I tried get back to a previous system state via the grub history menu but it was a miserable failure. I suspected it was a mismatch between initramfs and rpool, it was dumping me to the recovery shell but things like selinux were preventing me from getting a bootable system back. At that point I had deadlines so I opted to wipe & reinstall the system so the details were lost but this has been a question in my mind since then.
I think zfs is great but in the zfs-on-linux case (which zsys covers generally pretty well) system boot recovery is a major unresolved issue. Even when I asked about ZoL recovery on the zfs on linux discussion forum, the experts generally responded by saying “we’re experts, we figure it out - there is no documented recovery procedure” - to me, in intermediate linux user, this lack of a documented way to rebuild partitions, efi, swap & bpool from a backed-up rpool concerns me. I had to reinstall & reconfigure everything from scratch and import my data from a backup; it worked but this is hardly a good solution for such an otherwise excellent filesystem.
I’ve been dealing with trying to recover this disaster (zfs root focal>jammy do-release-upgrade) for over a week now. I would not recommend the default ubuntu zfs setup to anybody. I’m contemplating giving up and trying to zfs recv into another distro or something.
https://askubuntu.com/questions/1435307/booting-to-emergency-mode-after-20-04-5-focal-to-22-04-jammy-upgrade
reddit dot com/r/Ubuntu/comments/y7k875/how_can_i_reinstall_grub_on_this_system/