Automated Server installation

Server Guide
21

Hi, here are the logs again:
https://filebin.net/sqqwo5gi5v478nf3/logs-crash-cloud-install.tgz?t=g787z8s3

22

Thanks for this. The failure is apt-get update failing like this:

Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Get:1 file:/cdrom focal InRelease
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Ign:1 file:/cdrom focal InRelease
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Get:2 file:/cdrom focal Release [1486 B]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Get:2 file:/cdrom focal Release [1486 B]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Get:3 file:/cdrom focal Release.gpg [833 B]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Get:3 file:/cdrom focal Release.gpg [833 B]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Ign:4 http://ubuntu.mirror.lrz.de focal InRelease
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Ign:5 http://ubuntu.mirror.lrz.de focal-updates InRelease
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Ign:6 http://ubuntu.mirror.lrz.de focal-backports InRelease
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Ign:7 http://ubuntu.mirror.lrz.de focal-security InRelease
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Err:8 http://ubuntu.mirror.lrz.de focal Release
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:           404  Not Found [IP: 2001:4ca0:0:101:0:80:21:1 80]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Err:9 http://ubuntu.mirror.lrz.de focal-updates Release
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:           404  Not Found [IP: 2001:4ca0:0:101:0:80:21:1 80]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Err:10 http://ubuntu.mirror.lrz.de focal-backports Release
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:           404  Not Found [IP: 2001:4ca0:0:101:0:80:21:1 80]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Err:11 http://ubuntu.mirror.lrz.de focal-security Release
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:           404  Not Found [IP: 2001:4ca0:0:101:0:80:21:1 80]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Get:12 file:/cdrom focal/main amd64 Packages [18.7 kB]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Ign:12 file:/cdrom focal/main amd64 Packages
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Get:12 file:/cdrom focal/main amd64 Packages [18.7 kB]
Aug 12 20:00:43 ubuntu-server curtin_log.3318[9276]:         Reading package lists...

It looks like your mirror is wrong or incorrectly configured.

23

Argh, there was an “/ubuntu” missing there, thanks, should have spotted this myself. I never thought to look in this direction… Maybe the installer could throw an error message in a more visible location… ?

24

Yes, we should probably fetch at least the Release file before starting the install.

25

The documentation intersperses statements that “the system is endlessly flexible” with things like “format can be yaml”, leaving one to think that this is an ingenious system that really doesn’t care too much how you format or organize things. This seems to tie in with how vague and inconsistent the documentation is.

For instance, you start off by demonstrating the

# cloud-config
autoinstall:
    version: 1

but this is missing from each subsequent example, which are also backdented.

This difference in expectations makes it unduly painful dealing autoinstall. It’s yaml, so it has a schema, and you need to ensure you write a valid document. But the only way to confirm your document is valid is to run an install and see if it fails.

There’s nothing new or bad about that, unless you’ve been given the impression that the system can pretty much handle any kind of input you care to throw at it.

It would be better to refactor the documentation around the baseline requisites to get the thing working and out-of-line the annotations about alternate and flexible formats. I’m not even sure what formats other than YAML it supports, given that YAML is a superset of json and the only other examples I’ve seen were yaml-compliant json that complied precisely with the yaml schema?

Two smaller points of feedback:

1- There doesn’t appear to be a way to disable the unattended system upgrade, and for some reason the 20.04.1-live-server insists on doing a truly huge upgrade that slows the process down; I cannot find a documented way to disable it,

2- There doesn’t appear to be a recommended way to implement something like “postinstall”, nor sufficient documentation to cobble together the steps. I’ve tried combinations of write-files and late-commands that seem to work elsewhere; I’ve been trying to do:
late-commands:
- cp /cdrom/nocloud/postinstall.sh /run (I am putting postinstall.sh in /nocloud in the iso),
- curtin intarget --target /target /run/postinstall.sh
but I suspect the yaml is failing to validate because I simply end up with a log line about fallback data source and an option to login as ubuntu (which is not the user I’m defining).

2 Likes
26

Current documentation makes it very ambiguous whether autoinstall coexists with pre-20.04 cloud-config user data, whether it replaces it or if there is some third option. Can you, for instance:

#cloud-config
# vim: syntax=yaml

write-files:
- path: /etc/motd
  content: "configure me"

autoinstall: 
  version: 1
  refresh-installer:
    update: no

This is perhaps caused in part because of a lack of distinction between “user-data” (the file), user-data (the content) and "user-data" the key…

$ cat /cdrom/inst/userdata
#cloud-config
autoinstall:
  version: 1
  user-data:
    .. uh, but isn't autoinstall user-data? ..

Is user-data recursive?

#cloud-config
autoinstall:
  version: 1
  user-data:
    #cloud-config
    autoinstall:
      version: 1
      user-data:
        #cloud-config
        ....

Perhaps the following would be a useful lead for the page?

autoinstall is an extension for cloud-config that enables pre-configured responses to the Subiquity installers prompts, to facilitate unattended or partially-attended installations.

(If autoinstall replaces user-data:*

It replaces the content of the cloud-config file typically called user-data with a YAML dictionary with a top-level key of autoinstall. Legacy user-data settings can be populated by a sub-key, user-data

before:

#cloud-config
apt:
  proxy: http://apt-cacher-ng.lan:3142/
hostname: new-unconfigured-host
...

after:

#cloud-config
autoinstall:
  version: 1
  proxy: http://apt-cacher-ng.lan:3142/
  identity:
      hostname: new-unconfigured-host
      ...

If it coexists:

autoinstall can be inline of existing user-data cloud-configs simply by adding a top-level autoinstall key

#cloud-config
autoinstall:
  version: 1
  user-data:
    .. legacy top-level user-data options ..

A lot could be clarified by providing an example that creates an /etc/motd file, since this requires some additional order-of-operations explanation. c.f where do write-files entries go, when are they executed in relation to other actions, how do you ensure directory creation, etc?

Popular example for this might be configuring a system with a couple of apache config files – one that is installed by the package, one that you want to add. If write-files occurs early, the first file would be overwritten by the package installer.

27

The snap autoinstall-editor talked about here does not exist. Where may I find it? I do realize that the Debian pressed is a bit outdated, but should we really abandon it when the replacement is not fully there yet? Also, the documentation is not easy, for me at least, to follow… YAML is very picky since it will fail if there is a single space out of place. But it is a new way… so some tool would be very helpful.

2 Likes
28

I’m trying to perform a simple bare-metal install using the new 20.04 Automated Server Installs method.

I have found that this medium article gave my first end-to-end automated working example. And even this example has bugs and doesn’t work right.

Looking at the documentation, it feels that Ubuntu has dumped a 1500 piece puzzle over the fence and just let us figure it all out on our own.

I would love to obtain a 100% working end-to-end user-data I can just try in a VM. Then I can add parameters and build it out, but use that initial example as a working startingpoint.

  • The install mechanism requires that the computer has 3.2 GB of memory installed
  • The default install for a server is 5 GB --> I see no reason why a vanilla server install needs to be any larger than ~2GB max
  • A preseeded net-boot-based box is immediately up-to-date. With this solution I have to run apt upgrade -y and that takes another few minutes. I have not figured out if this process can be part of the automated install, but I doubt it.

I have no problems at all with new things. But new things should be:

  1. Solving real problems
  2. Making things better (simpler, faster, not worse
  3. well-documented!
  • The 5.5! install size seems due to a 3.2GB ‘swap.img’ file in the root. Well… ok.
29

I just want to add that I was able to PXE boot 18.04 LTS on a VM with just 512 MB of RAM. I think requiring 3+ GB of memory in a system for it to boot an Ubuntu installer is not OK.

Windows 2019 server requires only 512 MB of memory for crying out loud

30

Regarding my remarks about a working default out-of-the-box config: that was wrong. The simple example providet by Ubuntu works fine, I overlooked some steps, which is my fault.

The memory requirement still seems to stand.

My remarks about the upgrade process are also wrong: this is part of the automated installer.

I still think that the documentation is fragmented, but it is good enough to get started.

31

Ah good, in a sense. I did actually try the steps when I wrote that documentation…

Yes, this is unfortunate. It’s because netbooting works by downloading the ISO into RAM before mounting it. casper (the thing that does all the initramfs magic) does have code to support mounting via CIFS or NFS but I’ve not tested these in a long time and clearly requires a lot more setup on the server end.

32

Thanks for your reply. I think I read that others tried but could not make NFS to work. I notice that Casper only supports NFS or CIfs so the old behaviour can’t be replicated (bootstrapping everything over HTTP). Let’s hope that some day priority is given to - at least - make NFS work. That would help.

33

I’ve documented my experience with the installer in this personal, non-commercial blogpost.

Maybe it helps somebody else.

1 Like
34

I had to do something similar for this with debian-installer for ubuntu precise-bionic and debian stretch when installing via pxeboot/preseed. Newer hardware had the interfaces detected in the wrong order (1G interfaces got named before 10G interfaces, even though they were on later buses and had higher MAC addresses), and thanks to a bug in netcfg, it would only attempt dhcp over the first interface it found, even if the 2nd, 3rd, etc interface was the only active one. One that failed, it would give up and say that dhcp was impossible. Not a good behavior when you’re trying to manage 2000+ machines with random hardware configurations!
Solution?
Inject a script into the initrd init script to query for a dhcp lease, then use the dhclient.leases file generated to write a preseed.cfg file under / containing the interface to use, the IP address, hostname, dns servers, etc, which got autoloaded by debian-installer and executed before the one specified in the PXEBOOT file.

35

((yes/no Boolean expression 1))
ssh:
install-server: yes
authorized-keys:
- $key
allow-pw: no

((yes/no Boolean expression 2))
network:
network:
version: 2
ethernets:
enp0s25:
dhcp4: yes
enp3s0: {}
enp4s0: {}
bonds:
bond0:
dhcp4: yes
interfaces:
- enp3s0
- enp4s0
parameters:
mode: active-backup
primary: enp3s0

36

I think the values one “should” use for booleans are a bit of a matter of style aren’t they? I guess I’m not against making them match yamllint’s defaults but, well, YAML is YAML …

37

The example keyboard settings given are invalid.

    layout: en
    variant: uk

Should be the following

    layout: gb
    variant: extd
1 Like
38

Oh huh yes I obviously didn’t have my brain in gear there. Fixed, thanks.

1 Like
39

I’m trying to create an autoinstall config that only enables first ethernet card and leaves all other interfaces as disabled. It’s supposed to act this way on any hardware model so I have no idea what would be the device name of any NIC. On my testing VM they are called ens33, ens37 and ens38 so I couldn’t think about any possible rule to use match: in order to accomplish it.

Is it something possible to do?

Thanks!

40

Yeah, I don’t think that netplan supports this natively (what do you mean by “first” NIC?). You could presumably do this with an early-command though.