Please test autoinstalls for 20.04!

Thanks for the feedback again. I could’t find your post with code. Found just this. Is that what you mean:

Any chance you can be more specific on how to do it? I’m sorry I’m new to this autoinstall thing and cloud-init in general.

I’ve looked at the source code, and I see that you’re using curtin’s merge_config to merge the configuration generated by subiquity with supplied user-data (if present).

merge_config is smart about merging dictionaries, but doesn’t do anything special about the lists. What if we merge lists as well, instead of overwriting them?

Please note that the patch below is just a suggestion, I haven’t tested it yet. I have a couple of concerns too:

  • Could we break something by merging lists? While beneficial for stuff like users and groups, it may perhaps corrupt other configuration items? This is curtin territory after all.
  • While subiquity can update itself during the installation, I imagine curtin can’t?

  1. In curtin, we add support for lists by extending them:

    diff --git a/curtin/config.py b/curtin/config.py
    index 2106b239..93695c87 100644
    --- a/curtin/config.py
    +++ b/curtin/config.py
    @@ -33,6 +33,8 @@ def merge_config(cfg, cfg2):
         for k, v in cfg2.items():
             if isinstance(v, dict) and isinstance(cfg.get(k, None), dict):
                 merge_config(cfg[k], v)
    +        elif isinstance(v, list) and isinstance(cfg.get(k, None), list):
    +            cfg[k].extend(v)
             else:
                 cfg[k] = v
    
    
  2. In subiquity, we merge user-data into subiquity’s config, not vice versa. This will ensure, for instance, that the default system user (if defined) will come first and have UID 1000. And minus one deep copy, which is also nice.

    diff --git a/subiquity/models/subiquity.py b/subiquity/models/subiquity.py
    index 43814794..ff6d5d7c 100644
    --- a/subiquity/models/subiquity.py
    +++ b/subiquity/models/subiquity.py
    @@ -193,9 +193,8 @@ class SubiquityModel:
                 config['snap'] = {
                     'commands': cmds,
                     }
    -        userdata = copy.deepcopy(self.userdata)
    -        merge_config(userdata, config)
    -        return userdata
    +        merge_config(config, self.userdata)
    +        return config
    
         def _cloud_init_files(self):
             # TODO, this should be moved to the in-target cloud-config seed so on
    

It seems that this bug has resurfaced. I can’t get user-data to work at all with the latest daily iso’s or the 20.04 server iso.
https://bugs.launchpad.net/subiquity/+bug/1880104

Hi,

I’m performing autoinstall with today’s daily-live focal-live-server-amd64.iso and in the user-data (autoinstall.yaml) file I’m refreshing subiquity from the edge (20.05.2+git63.4ef44df3). (I know, I like to live dangerously :smiley:).

The recent change to subiquitycore/controllers/network.py appears to ‘cause’ an installer crash. The git commit comment says “update wlan dialog when scan results come in

I’m confused by the error messages as I’m doing an autoinstall and my target server doesn’t have a wireless lan interface. Here are the final lines in the subiquity debug log:

2020-05-22 17:47:53,987 DEBUG probert.network:584 event for link_change: CHANGE {'ifindex': 2, 'flags': 69699, 'arptype': 1, 'family': 0, 'is_vlan': False, 'name': b'eno1'}
2020-05-22 17:47:53,987 DEBUG probert.network:584 event for addr_change: CHANGE {'ifindex': 2, 'flags': 128, 'family': 2, 'scope': 0, 'local': b'192.168.2.145/24'}
2020-05-22 17:47:53,987 DEBUG probert.network:671 link_change CHANGE {'ifindex': 2, 'flags': 69699, 'arptype': 1, 'family': 0, 'is_vlan': False, 'name': b'eno1'}
2020-05-22 17:47:53,987 DEBUG probert.network:716 addr_change CHANGE {'ifindex': 2, 'flags': 128, 'family': 2, 'scope': 0, 'local': b'192.168.2.145/24'}
2020-05-22 17:47:53,987 DEBUG subiquitycore.core:238 _exception_handler AttributeError("'ScrollBarListBox' object has no attribute '_w'",)
2020-05-22 17:47:53,989 ERROR subiquitycore.core:678 Exception in controller.run():
Traceback (most recent call last):
  File "/snap/subiquity/1907/lib/python3.6/site-packages/subiquitycore/core.py", line 676, in run
    self.urwid_loop.run()
  File "/snap/subiquity/1907/usr/lib/python3/dist-packages/urwid/main_loop.py", line 286, in run
    self._run()
  File "/snap/subiquity/1907/usr/lib/python3/dist-packages/urwid/main_loop.py", line 384, in _run
    self.event_loop.run()
  File "/snap/subiquity/1907/usr/lib/python3/dist-packages/urwid/main_loop.py", line 1484, in run
    reraise(*exc_info)
  File "/snap/subiquity/1907/usr/lib/python3/dist-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/snap/subiquity/1907/usr/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/snap/subiquity/1907/lib/python3.6/site-packages/subiquitycore/controllers/network.py", line 203, in _data_ready
    if isinstance(v._w, StretchyOverlay):
AttributeError: 'ScrollBarListBox' object has no attribute '_w'
2020-05-22 17:47:53,991 DEBUG subiquity.core:466 generating crash report
2020-05-22 17:47:53,992 INFO subiquity.core:483 saving crash report 'Installer UI crashed with AttributeError' to /var/crash/1590169673.992039204.ui.crash
2020-05-22 17:47:53,992 DEBUG root:39 start: subiquity/Error/1590169673.992039204.ui/add_info:
2020-05-22 17:48:00,700 DEBUG root:39 finish: subiquity/Error/1590169673.992039204.ui/add_info: SUCCESS: written to /var/crash/1590169673.992039204.ui.crash

Not sure why ScrollBarListBox is required in an autoinstall (I assume this is associated with manual installs). I note that the probert.network has just detected the static IP address change (for the wired ethernet eno1 interface). Is the installer reacting to the wrong event somehow?

Thanks,

Bob

Wanted to try out a change similar to the one I suggested above, but I can’t get the snap to build. Getting the following error:

Building languagelists
do not know native name for oc
Failed to run 'override-build': Exit code was 1.

The error comes from this script: scripts/make-language-lists. Not sure how to tackle this, to be honest. It’s not aware of Occitan for some reason?

There’s some more errors before that, for instance the lack of --single-version-externally-managed option in setup.py for apport and curtin.

Full log: paste.ubuntu.com/p/63FTqNfN5G.

Here’s how I build:

  • Start with a clean Ubuntu Server 20.04 VM
  • Install git, build-essential and snapcraft
  • Apply some changes I want to test
  • Run the snapcraft command from the readme

As far as I understand, the build process occurs in a Multipass virtual machine, so my environment shouldn’t really matter that much, should it?

Hey folks, I’m trying to install a physical computer lab with ubuntu 20.04, but I can’t figure out how to ONLY ask for the hostname during the process:

https://askubuntu.com/questions/1243679/ask-only-for-hostname-in-an-automated-server-install/

Also, It’s not clear to me how to set the config to select “Fill entire disk”, because it is not doing that when I do a fully automated install, even though that seems to be the default option? https://askubuntu.com/questions/1244293/how-to-autoinstall-config-fill-disk-option-on-ubuntu-20-04-automated-server-in?noredirect=1#comment2097575_1244293

Hello here,

I’m trying to autoinstall a VM with Ubuntu server 20.04 but I got stuck on the hard-drive partition step.

I can’t make GRUB to install itself. I tried every syntax and example provided here but each time the installation finishes with a crash and this error message: “autoinstall config did not create needed bootloader partition”

Please find the configuration I am using:
(please also note that re-using the configuration from a manual installation does the same error)

my conf

#cloud-config
autoinstall:
version: 1
refresh-installer:
update: yes
reporting:
hook:
type: webhook
endpoint:
keyboard:
layout: fr
variant: ‘’
locale: en_US

network:

ethernets:

enp0s3:

dhcp4: true

version: 2

identity:
hostname: workstation
password: password_here
realname: user
username: user
ssh:
allow-pw: true
authorized-keys: []
install-server: false
storage:
swap:
size: 0
grub:
reorder_uefi: false
install_devices:

  • /dev/sda1
    config:
  • type: disk
    id: disk-sda
    ptable: gpt
    path: /dev/sda
    preserve: false
    grub_device: false
    name: ‘system’
  • type: partition
    id: partition-efi
    device: disk-sda
    size: 512MB
    grub_device: true
    flag: boot
    number: 1
  • type: partition
    id: partition-boot
    device: disk-sda
    size: 1G
    grub_device: false
    flag: ‘’
    number: 2
  • type: partition
    id: partition-system
    device: disk-sda
    size: 100%
    grub_device: false
    flag: ‘’
    number: 3
  • type: dm_crypt
    id: dm_crypt-system
    volume: partition-system
    key: criteo
    preserve: false
  • type: lvm_volgroup
    id: lvm_volgroup-system
    name: ubuntu-vg
    devices: [dm_crypt-system]
    preserve: false
  • type: lvm_partition
    id: lvm_partition-system
    volgroup: lvm_volgroup-system
    name: ubuntu-lv
    size: 100%
    preserve: false
  • type: format
    id: format-efi
    volume: partition-efi
    fstype: fat32
    preserve: false
  • type: format
    id: format-boot
    volume: partition-boot
    fstype: ext4
    preserve: false
  • type: format
    id: format-system
    volume: lvm_partition-system
    fstype: ext4
    preserve: false
  • type: mount
    id: mount-system
    device: format-system
    path: /
  • type: mount
    id: mount-boot
    device: format-boot
    path: /boot
  • type: mount
    id: mount-efi
    device: format-efi
    path: /boot/efi

Hello did you finally solve your problem ?
I have the same. And even with the grub_device: true the installation fails with the same bootloader error

After much trial and error, I was able to netboot and autoinstall on my NUC. One issue I didn’t see talked about anywhere was the installer kept crashing while setting up the disks. I had an older Ubuntu install on this machine and it was running in legacy (BIOS) mode with MBR. I was switching it to EFI & GPT this go round and until I manually deleted the old dos partition table it would not successfully complete this step. I’m unsure if this is a bug or something missing in my config. Here is the storage section of my user-data

  storage:
    grub:
      reorder_uefi: False
    config:
      - id: disk0
        type: disk
        ptable: gpt
        serial: Samsung_SSD_850_EVO_250GB_S21NNSAFC10405E
        preserve: false
      - id: disk0-efi
        type: partition
        number: 1
        size: 512MB
        device: disk0
        flag: boot
        grub_device: true
        preserve: false
      - id: disk0-efi-fs
        type: format
        fstype: fat32
        volume: disk0-efi
        preserve: false
      - id: disk0-efi-mount
        type: mount
        path: /boot/efi
        device: disk0-efi-fs
        preserve: false
      - id: disk0-root
        type: partition
        number: 2
        size: -1
        device: disk0
        preserve: false
      - id: disk0-root-fs
        type: format
        fstype: ext4
        volume: disk0-root
        preserve: false
      - id: disk0-root-mount
        type: mount
        path: /
        device: disk0-root-fs
        options: 'defaults,noatime,discard,errors=remount-ro'
        preserve: false
1 Like

Hi, I want to give some feedback about the new autoinstaller for ubuntu 20.04 after spending all day trying to make it work:

Setting the ubuntu user password never worked:

#cloud-config
autoinstall:
  version: 1
  identity:
    hostname: ubuntu-server
    password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
    username: ubuntu

Even using the quickstart example verbatim never resulted in an installed system that I could log-in to with the ‘ubuntu’ user. I tried so many different ways to set the ubuntu user password, and nothing ever worked:

  • openssl passwd -6 -salt xyz ubuntu
  • echo ubuntu | mkpasswd -m sha512crypt --stdin
  • mkpasswd --method=SHA-512 --rounds=4096
  • simply using the plaintext ‘ubuntu’ as the value of the password field
  • quoting or not-quoting the value in the password field

I’m really curious if/how anyone ever got an autoinstalled system with a defined password that actually worked, or what specifically you need to do to generate a proper password and how to communicate it to the autoinstaller

ssh authorized keys configuration never worked

  ssh:
      install-server: yes
      authorized-keys:
          - ssh-rsa <some key infomation>
      allow-pw: yes

Because setting a password (see above) never worked, I next tried to set ssh authorized keys. Using the exact same format and data from a known working cloud-init configuration.

This did not work. I tried quoting and not-quoting the values passed via authorized-keys.

refresh-installer crashes with the May 31st focal server daily ISO:

  refresh-installer:
    update: yes

As I was desperately trying to make this work, I tried combinations of the GA release of ubuntu server 20.04 and the latest focal server dailies. I observed that attempting to set refresh-installer causes the installer to crash when using the May 31st version of the focal server daily ISO.

attempting to install some packages causes the autoinstaller to crash:

  packages:
    - curl
    - wget
    - htop
    - nfs-common

Attempting to install nfs-common caused the autoinstaller to crash. The only remedy was to remove that package from the packages: section.

networking configuration never worked:

I tried combinations of removing the nested networking: stanza and different interface names without ever having a successful network configuration set via autoinstaller. Because I couldn’t even log into the installed system (see the problems with the passwords above) it wasn’t possible to figure out what was being applied.

  network:
    network:
      version: 2
      ethernets:
          ens18:
            dhcp4: false
            optional: true
      vlans:
        vlan.20:
          dhcp4: true
          id: 20
          link: ens18

user-data setting never worked:

Thinking that autoinstaller is perhaps too problematic to use (based on the experiences above), I tried to make the ‘autoinstall’ portion as minimal as possible and inject a known working cloud-init configuration in the user-data section. This did not work.

I assume it’s a current bug based on this and I could never find a working combination of settings which resulted in settings defined in the user-data section of the autoinstall configuration to actually apply.

conclusion:

After dozens of installation attempts and trying combinations of attempts with both the official ubuntu 20.04 GA release server installer ISO, the May 31st ‘daily’ version of the focal server installer ISO, and refresh-installer (when it worked), it never resulted in a successful installation.

I know that some settings were being applied as I could set the hostname via the identity section and see the some of the packages were being installed (except the ones that caused it to crash); so I feel confident that it was properly consuming the referenced user-data (I also checked for tabs vs spaces in the user-data file).

Is the autoinstall approach still considered something to be tested/beta, or is it intended to be a GA usable installation approach for ubuntu server?

Are you perhaps booting your VM in legacy mode? (i.e. not UEFI) The config looks appropriate for a UEFI system on a quick glance.

There were some bugs in this area. They should be fixed in the release in the stable channel or on the latest dailies though, so if you’re still seeing failures I’d like to see the logs.

This sounds a bit as if the cloud-init config that should configure these things never ran on first boot. Can you fish out the /var/log/cloud-init* files from the installed system (by booting the installer again, or mounting the disk image if you installed a VM or something like that)?

I shall have to try this tomorrow. It certainly shouldn’t crash!

:frowning:

It’s intended to be usable but it’s new and some bugs are to be expected. You’ve definitely had a bad run though, sorry about that. I’ll try to reproduce your issues in the next day or so and see what I can figure out.

1 Like

Hello that was indeed the case :slight_smile:
I edited manualy the ISO file to add autoinstall menu and in broke GRUB. So my VM was booting ISOLINUX+BIOS instead of GRUB+EFI

Hello

Did somebody tried to execute multi-lines commands in the ‘late-commands’ section ?
I am trying to write a new file with text inside, but it fails.

Autoinstall file extract:

      late-commands:
        - cat <<EOT >> /target/lib/systemd/system/example.service
          [Unit]
          Wants=network-online.target
          [Service]
          Type=oneshot
          ExecStart=example.sh
          [Install]
          WantedBy=multi-user.target
          EOT
#cloud-config
autoinstall:
  version: 1
  identity:
    realname: UBUNTU_REMOVE_ME
    username: ubuntu
    password: $6$GSmxlvP1$K38Inpkh3FavGYA1oJGQciLYTBLNCqlqUSq6xeF7P.rk8GvRLg.OgcwP8cQBx/A1MZJ3sVs9PdC9GzYyqMTyo0
    hostname: changeme

Hostname gets done, user in fact does not exist in /etc/shadow

which just confirms what others have been saying. Identity user stuff is being ignored.

1 Like

Ah good.

      late-commands:
        - |
          cat <<EOT >> /target/lib/systemd/system/example.service
          [Unit]
          Wants=network-online.target
          [Service]
          Type=oneshot
          ExecStart=example.sh
          [Install]
          WantedBy=multi-user.target
          EOT

should work I think.

Can you get me /var/lib/cloud and /var/log/cloud-init* somehow?

2 Likes

Hi,

Should I report Please test autoinstalls for 20.04! on Launchpad?

I’m not sure as I’m using daily isos and the subiquity edge channel - which I know could be problematic.

Looks like edge has not been updated for a couple of weeks.

Thanks in advance.

Thanks! It works fine with a pipe at the beginning

Yes please, I’d forgotten about that one…

Done #1881887 :slightly_smiling_face:. I did try to install today to check the issue still exists, and it does.