Ubuntu 20.10 zfs: update-grub, zsysctl boot update-menu don't create new entries

I just recovered from a non-booting system. It was a default installation of ubuntu 20.10 zfs that had been working fine for a couple days, but I’m fairly certain what caused it to fail booting is update-grub is not creating new menu entries, and apt autoremove deleted initrd file grub.cfg was pointing to.

grub-mkconfig (update-grub) appears not to be able to look at bpool . I’m really confused about what’s going on, but it seems to be an ongoing issue, and I believe it originated with zsys .

I hope someone can help me. Anyone ever seen this before?

Here’s what happens when I run grub-mkconfig:

# update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Some pools couldn't be imported and will be ignored:
cannot import 'bpool': pool already exists    # I think this is the main issue (?)
cannot import 'rpool': pool already exists
Found linux image: vmlinuz-5.8.0-23-generic in rpool/ROOT/ubuntu_84vh5h
Found initrd image: initrd.img-5.8.0-23-generic in rpool/ROOT/ubuntu_84vh5h
Found linux image: vmlinuz-5.8.0-22-generic in rpool/ROOT/ubuntu_84vh5h
Found initrd image: initrd.img-5.8.0-22-generic in rpool/ROOT/ubuntu_84vh5h
Found linux image: vmlinuz-5.8.0-23-generic in rpool/ROOT/ubuntu_84vh5h@autozsys_6llbir

(truncated)

There’s a lot of moving parts, but here’s the reason I’m wondering if it has to do with zsys was because zsysctl boot commit failed, and zsysctl machine show does not show bpool

zsysctl boot update-menu APPEARS to work (which I performed after boot prepare and boot update commands), but then when I reboot there’s no changes reflected in the grub menu

# zsysctl -vv machine show 
DEBUG /zsys.Zsys/MachineShow() call logged as [c0e2688b:46dd7a84] 
DEBUG Check if grpc request peer is authorized     
DEBUG Authorized as being administrator            
INFO Retrieving information for machine rpool/ROOT/ubuntu_84vh5h 
Name:           rpool/ROOT/ubuntu_84vh5h
ZSys:           true
Last Used:      current
History:        
  - Name:       rpool/ROOT/ubuntu_84vh5h@autozsys_ay9l5x
    Created on: 2020-10-15 13:14:07
  - Name:       rpool/ROOT/ubuntu_84vh5h@autozsys_7j9b5k
    Created on: 2020-10-15 12:45:53
  - Name:       rpool/ROOT/ubuntu_84vh5h@autozsys_m2ibfv
    Created on: 2020-10-15 12:35:39
  - Name:       rpool/ROOT/ubuntu_84vh5h@autozsys_6llbir
    Created on: 2020-10-15 12:07:41
Users:
  - Name:    avery
    History: 
     - rpool/USERDATA/avery_5merai@autozsys_ay9l5x (2020-10-15 13:14:10)
     - rpool/USERDATA/avery_5merai@autozsys_8gyrt8 (2020-10-15 13:08:26)
     - rpool/USERDATA/avery_5merai@autozsys_7j9b5k (2020-10-15 12:45:57)
     - rpool/USERDATA/avery_5merai@autozsys_m2ibfv (2020-10-15 12:35:42)
     - rpool/USERDATA/avery_5merai@autozsys_9qrp3j (2020-10-15 12:07:59)
     - rpool/USERDATA/avery_5merai@autozsys_6llbir (2020-10-15 12:07:59)
  - Name:    root
    History: 
     - rpool/USERDATA/root_5merai@autozsys_ay9l5x (2020-10-15 13:14:11)
     - rpool/USERDATA/root_5merai@autozsys_7j9b5k (2020-10-15 12:45:57)
     - rpool/USERDATA/root_5merai@autozsys_m2ibfv (2020-10-15 12:35:42)
     - rpool/USERDATA/root_5merai@autozsys_6llbir (2020-10-15 12:07:59)

I also noticed this:

# zsysctl service -vv reload
DEBUG /zsys.Zsys/Reload() call logged as [53cc97a1:18b77189] 
DEBUG Check if grpc request peer is authorized     
DEBUG Authorized as being administrator            
INFO Reloading daemon configuration               
DEBUG couldn't find default configuration path, fallback to internal default

I
A couple things I am wondering about:

should zsys be seeing my bpool ?

should zsys have a default configuration path?

There’s no zfs inherit bpool/BOOT/ubuntu_84vh5h/grub dataset. From this openzfs team reference: https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS.html

They point to com.ubuntu.zsys:bootfs bpool/BOOT/ubuntu_UUID/grub . Is this a dataset that was destroyed somehow?

For now I’m rsyncing my user folder with another installation on a separate disk using ext4 so I can recover the machine immediately if it fails to boot on me again, but this whole thing has been a real pain. If anyone has any ideas, I’m all ears.

Thanks

For people interested: we are iterating on a bug rather than here: https://github.com/ubuntu/zsys/issues/168