Thanks for the feedback again. I could’t find your post with code. Found just this. Is that what you mean:
Any chance you can be more specific on how to do it? I’m sorry I’m new to this autoinstall thing and cloud-init in general.
I’ve looked at the source code, and I see that you’re using curtin’s merge_config to merge the configuration generated by subiquity with supplied user-data (if present).
merge_config is smart about merging dictionaries, but doesn’t do anything special about the lists. What if we merge lists as well, instead of overwriting them?
Please note that the patch below is just a suggestion, I haven’t tested it yet. I have a couple of concerns too:
- Could we break something by merging lists? While beneficial for stuff like users and groups, it may perhaps corrupt other configuration items? This is curtin territory after all.
- While subiquity can update itself during the installation, I imagine curtin can’t?
-
In curtin, we add support for lists by extending them:
diff --git a/curtin/config.py b/curtin/config.py index 2106b239..93695c87 100644 --- a/curtin/config.py +++ b/curtin/config.py @@ -33,6 +33,8 @@ def merge_config(cfg, cfg2): for k, v in cfg2.items(): if isinstance(v, dict) and isinstance(cfg.get(k, None), dict): merge_config(cfg[k], v) + elif isinstance(v, list) and isinstance(cfg.get(k, None), list): + cfg[k].extend(v) else: cfg[k] = v -
In subiquity, we merge user-data into subiquity’s config, not vice versa. This will ensure, for instance, that the default system user (if defined) will come first and have UID 1000. And minus one deep copy, which is also nice.
diff --git a/subiquity/models/subiquity.py b/subiquity/models/subiquity.py index 43814794..ff6d5d7c 100644 --- a/subiquity/models/subiquity.py +++ b/subiquity/models/subiquity.py @@ -193,9 +193,8 @@ class SubiquityModel: config['snap'] = { 'commands': cmds, } - userdata = copy.deepcopy(self.userdata) - merge_config(userdata, config) - return userdata + merge_config(config, self.userdata) + return config def _cloud_init_files(self): # TODO, this should be moved to the in-target cloud-config seed so on
It seems that this bug has resurfaced. I can’t get user-data to work at all with the latest daily iso’s or the 20.04 server iso.
https://bugs.launchpad.net/subiquity/+bug/1880104
Hi,
I’m performing autoinstall with today’s daily-live focal-live-server-amd64.iso and in the user-data (autoinstall.yaml) file I’m refreshing subiquity from the edge (20.05.2+git63.4ef44df3). (I know, I like to live dangerously 
).
The recent change to subiquitycore/controllers/network.py appears to ‘cause’ an installer crash. The git commit comment says “update wlan dialog when scan results come in”
I’m confused by the error messages as I’m doing an autoinstall and my target server doesn’t have a wireless lan interface. Here are the final lines in the subiquity debug log:
2020-05-22 17:47:53,987 DEBUG probert.network:584 event for link_change: CHANGE {'ifindex': 2, 'flags': 69699, 'arptype': 1, 'family': 0, 'is_vlan': False, 'name': b'eno1'}
2020-05-22 17:47:53,987 DEBUG probert.network:584 event for addr_change: CHANGE {'ifindex': 2, 'flags': 128, 'family': 2, 'scope': 0, 'local': b'192.168.2.145/24'}
2020-05-22 17:47:53,987 DEBUG probert.network:671 link_change CHANGE {'ifindex': 2, 'flags': 69699, 'arptype': 1, 'family': 0, 'is_vlan': False, 'name': b'eno1'}
2020-05-22 17:47:53,987 DEBUG probert.network:716 addr_change CHANGE {'ifindex': 2, 'flags': 128, 'family': 2, 'scope': 0, 'local': b'192.168.2.145/24'}
2020-05-22 17:47:53,987 DEBUG subiquitycore.core:238 _exception_handler AttributeError("'ScrollBarListBox' object has no attribute '_w'",)
2020-05-22 17:47:53,989 ERROR subiquitycore.core:678 Exception in controller.run():
Traceback (most recent call last):
File "/snap/subiquity/1907/lib/python3.6/site-packages/subiquitycore/core.py", line 676, in run
self.urwid_loop.run()
File "/snap/subiquity/1907/usr/lib/python3/dist-packages/urwid/main_loop.py", line 286, in run
self._run()
File "/snap/subiquity/1907/usr/lib/python3/dist-packages/urwid/main_loop.py", line 384, in _run
self.event_loop.run()
File "/snap/subiquity/1907/usr/lib/python3/dist-packages/urwid/main_loop.py", line 1484, in run
reraise(*exc_info)
File "/snap/subiquity/1907/usr/lib/python3/dist-packages/urwid/compat.py", line 58, in reraise
raise value
File "/snap/subiquity/1907/usr/lib/python3.6/asyncio/events.py", line 145, in _run
self._callback(*self._args)
File "/snap/subiquity/1907/lib/python3.6/site-packages/subiquitycore/controllers/network.py", line 203, in _data_ready
if isinstance(v._w, StretchyOverlay):
AttributeError: 'ScrollBarListBox' object has no attribute '_w'
2020-05-22 17:47:53,991 DEBUG subiquity.core:466 generating crash report
2020-05-22 17:47:53,992 INFO subiquity.core:483 saving crash report 'Installer UI crashed with AttributeError' to /var/crash/1590169673.992039204.ui.crash
2020-05-22 17:47:53,992 DEBUG root:39 start: subiquity/Error/1590169673.992039204.ui/add_info:
2020-05-22 17:48:00,700 DEBUG root:39 finish: subiquity/Error/1590169673.992039204.ui/add_info: SUCCESS: written to /var/crash/1590169673.992039204.ui.crash
Not sure why ScrollBarListBox is required in an autoinstall (I assume this is associated with manual installs). I note that the probert.network has just detected the static IP address change (for the wired ethernet eno1 interface). Is the installer reacting to the wrong event somehow?
Thanks,
Bob
Wanted to try out a change similar to the one I suggested above, but I can’t get the snap to build. Getting the following error:
Building languagelists
do not know native name for oc
Failed to run 'override-build': Exit code was 1.
The error comes from this script: scripts/make-language-lists. Not sure how to tackle this, to be honest. It’s not aware of Occitan for some reason?
There’s some more errors before that, for instance the lack of --single-version-externally-managed option in setup.py for apport and curtin.
Full log: paste.ubuntu.com/p/63FTqNfN5G.
Here’s how I build:
- Start with a clean Ubuntu Server 20.04 VM
- Install
git,build-essentialandsnapcraft - Apply some changes I want to test
- Run the
snapcraftcommand from the readme
As far as I understand, the build process occurs in a Multipass virtual machine, so my environment shouldn’t really matter that much, should it?
Hey folks, I’m trying to install a physical computer lab with ubuntu 20.04, but I can’t figure out how to ONLY ask for the hostname during the process:
https://askubuntu.com/questions/1243679/ask-only-for-hostname-in-an-automated-server-install/
Also, It’s not clear to me how to set the config to select “Fill entire disk”, because it is not doing that when I do a fully automated install, even though that seems to be the default option? https://askubuntu.com/questions/1244293/how-to-autoinstall-config-fill-disk-option-on-ubuntu-20-04-automated-server-in?noredirect=1#comment2097575_1244293
Hello here,
I’m trying to autoinstall a VM with Ubuntu server 20.04 but I got stuck on the hard-drive partition step.
I can’t make GRUB to install itself. I tried every syntax and example provided here but each time the installation finishes with a crash and this error message: “autoinstall config did not create needed bootloader partition”
Please find the configuration I am using:
(please also note that re-using the configuration from a manual installation does the same error)
my conf
#cloud-config
autoinstall:
version: 1
refresh-installer:
update: yes
reporting:
hook:
type: webhook
endpoint:
keyboard:
layout: fr
variant: ‘’
locale: en_US
network:
ethernets:
enp0s3:
dhcp4: true
version: 2
identity:
hostname: workstation
password: password_here
realname: user
username: user
ssh:
allow-pw: true
authorized-keys: []
install-server: false
storage:
swap:
size: 0
grub:
reorder_uefi: false
install_devices:
- /dev/sda1
config: - type: disk
id: disk-sda
ptable: gpt
path: /dev/sda
preserve: false
grub_device: false
name: ‘system’ - type: partition
id: partition-efi
device: disk-sda
size: 512MB
grub_device: true
flag: boot
number: 1 - type: partition
id: partition-boot
device: disk-sda
size: 1G
grub_device: false
flag: ‘’
number: 2 - type: partition
id: partition-system
device: disk-sda
size: 100%
grub_device: false
flag: ‘’
number: 3 - type: dm_crypt
id: dm_crypt-system
volume: partition-system
key: criteo
preserve: false - type: lvm_volgroup
id: lvm_volgroup-system
name: ubuntu-vg
devices: [dm_crypt-system]
preserve: false - type: lvm_partition
id: lvm_partition-system
volgroup: lvm_volgroup-system
name: ubuntu-lv
size: 100%
preserve: false - type: format
id: format-efi
volume: partition-efi
fstype: fat32
preserve: false - type: format
id: format-boot
volume: partition-boot
fstype: ext4
preserve: false - type: format
id: format-system
volume: lvm_partition-system
fstype: ext4
preserve: false - type: mount
id: mount-system
device: format-system
path: / - type: mount
id: mount-boot
device: format-boot
path: /boot - type: mount
id: mount-efi
device: format-efi
path: /boot/efi
Hello did you finally solve your problem ?
I have the same. And even with the grub_device: true the installation fails with the same bootloader error
After much trial and error, I was able to netboot and autoinstall on my NUC. One issue I didn’t see talked about anywhere was the installer kept crashing while setting up the disks. I had an older Ubuntu install on this machine and it was running in legacy (BIOS) mode with MBR. I was switching it to EFI & GPT this go round and until I manually deleted the old dos partition table it would not successfully complete this step. I’m unsure if this is a bug or something missing in my config. Here is the storage section of my user-data
storage:
grub:
reorder_uefi: False
config:
- id: disk0
type: disk
ptable: gpt
serial: Samsung_SSD_850_EVO_250GB_S21NNSAFC10405E
preserve: false
- id: disk0-efi
type: partition
number: 1
size: 512MB
device: disk0
flag: boot
grub_device: true
preserve: false
- id: disk0-efi-fs
type: format
fstype: fat32
volume: disk0-efi
preserve: false
- id: disk0-efi-mount
type: mount
path: /boot/efi
device: disk0-efi-fs
preserve: false
- id: disk0-root
type: partition
number: 2
size: -1
device: disk0
preserve: false
- id: disk0-root-fs
type: format
fstype: ext4
volume: disk0-root
preserve: false
- id: disk0-root-mount
type: mount
path: /
device: disk0-root-fs
options: 'defaults,noatime,discard,errors=remount-ro'
preserve: false
1 Like
Hi, I want to give some feedback about the new autoinstaller for ubuntu 20.04 after spending all day trying to make it work:
Setting the ubuntu user password never worked:
#cloud-config
autoinstall:
version: 1
identity:
hostname: ubuntu-server
password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
username: ubuntu
Even using the quickstart example verbatim never resulted in an installed system that I could log-in to with the ‘ubuntu’ user. I tried so many different ways to set the ubuntu user password, and nothing ever worked:
openssl passwd -6 -salt xyz ubuntuecho ubuntu | mkpasswd -m sha512crypt --stdinmkpasswd --method=SHA-512 --rounds=4096- simply using the plaintext ‘ubuntu’ as the value of the password field
- quoting or not-quoting the value in the
passwordfield
I’m really curious if/how anyone ever got an autoinstalled system with a defined password that actually worked, or what specifically you need to do to generate a proper password and how to communicate it to the autoinstaller
ssh authorized keys configuration never worked
ssh:
install-server: yes
authorized-keys:
- ssh-rsa <some key infomation>
allow-pw: yes
Because setting a password (see above) never worked, I next tried to set ssh authorized keys. Using the exact same format and data from a known working cloud-init configuration.
This did not work. I tried quoting and not-quoting the values passed via authorized-keys.
refresh-installer crashes with the May 31st focal server daily ISO:
refresh-installer:
update: yes
As I was desperately trying to make this work, I tried combinations of the GA release of ubuntu server 20.04 and the latest focal server dailies. I observed that attempting to set refresh-installer causes the installer to crash when using the May 31st version of the focal server daily ISO.
attempting to install some packages causes the autoinstaller to crash:
packages:
- curl
- wget
- htop
- nfs-common
Attempting to install nfs-common caused the autoinstaller to crash. The only remedy was to remove that package from the packages: section.
networking configuration never worked:
I tried combinations of removing the nested networking: stanza and different interface names without ever having a successful network configuration set via autoinstaller. Because I couldn’t even log into the installed system (see the problems with the passwords above) it wasn’t possible to figure out what was being applied.
network:
network:
version: 2
ethernets:
ens18:
dhcp4: false
optional: true
vlans:
vlan.20:
dhcp4: true
id: 20
link: ens18
user-data setting never worked:
Thinking that autoinstaller is perhaps too problematic to use (based on the experiences above), I tried to make the ‘autoinstall’ portion as minimal as possible and inject a known working cloud-init configuration in the user-data section. This did not work.
I assume it’s a current bug based on this and I could never find a working combination of settings which resulted in settings defined in the user-data section of the autoinstall configuration to actually apply.
conclusion:
After dozens of installation attempts and trying combinations of attempts with both the official ubuntu 20.04 GA release server installer ISO, the May 31st ‘daily’ version of the focal server installer ISO, and refresh-installer (when it worked), it never resulted in a successful installation.
I know that some settings were being applied as I could set the hostname via the identity section and see the some of the packages were being installed (except the ones that caused it to crash); so I feel confident that it was properly consuming the referenced user-data (I also checked for tabs vs spaces in the user-data file).
Is the autoinstall approach still considered something to be tested/beta, or is it intended to be a GA usable installation approach for ubuntu server?
Are you perhaps booting your VM in legacy mode? (i.e. not UEFI) The config looks appropriate for a UEFI system on a quick glance.
There were some bugs in this area. They should be fixed in the release in the stable channel or on the latest dailies though, so if you’re still seeing failures I’d like to see the logs.
This sounds a bit as if the cloud-init config that should configure these things never ran on first boot. Can you fish out the /var/log/cloud-init* files from the installed system (by booting the installer again, or mounting the disk image if you installed a VM or something like that)?
I shall have to try this tomorrow. It certainly shouldn’t crash!
It’s intended to be usable but it’s new and some bugs are to be expected. You’ve definitely had a bad run though, sorry about that. I’ll try to reproduce your issues in the next day or so and see what I can figure out.
1 Like
Hello that was indeed the case 
I edited manualy the ISO file to add autoinstall menu and in broke GRUB. So my VM was booting ISOLINUX+BIOS instead of GRUB+EFI
Hello
Did somebody tried to execute multi-lines commands in the ‘late-commands’ section ?
I am trying to write a new file with text inside, but it fails.
Autoinstall file extract:
late-commands:
- cat <<EOT >> /target/lib/systemd/system/example.service
[Unit]
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=example.sh
[Install]
WantedBy=multi-user.target
EOT
#cloud-config
autoinstall:
version: 1
identity:
realname: UBUNTU_REMOVE_ME
username: ubuntu
password: $6$GSmxlvP1$K38Inpkh3FavGYA1oJGQciLYTBLNCqlqUSq6xeF7P.rk8GvRLg.OgcwP8cQBx/A1MZJ3sVs9PdC9GzYyqMTyo0
hostname: changeme
Hostname gets done, user in fact does not exist in /etc/shadow
which just confirms what others have been saying. Identity user stuff is being ignored.
1 Like
Ah good.
late-commands:
- |
cat <<EOT >> /target/lib/systemd/system/example.service
[Unit]
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=example.sh
[Install]
WantedBy=multi-user.target
EOT
should work I think.
Can you get me /var/lib/cloud and /var/log/cloud-init* somehow?
2 Likes
Hi,
Should I report Please test autoinstalls for 20.04! on Launchpad?
I’m not sure as I’m using daily isos and the subiquity edge channel - which I know could be problematic.
Looks like edge has not been updated for a couple of weeks.
Thanks in advance.
Thanks! It works fine with a pipe at the beginning
Yes please, I’d forgotten about that one…
Done #1881887 
. I did try to install today to check the issue still exists, and it does.