SSHd now uses socket-based activation (Ubuntu 22.10 and later)

As of version 1:9.0p1-1ubuntu1 of openssh-server in Kinetic Kudu (Ubuntu 22.10), OpenSSH in Ubuntu is configured by default to use systemd socket activation. This means that sshd will not be started until an incoming connection request is received. This has been done to reduce the memory consumed by Ubuntu Server instances by default, which is of particular interest with Ubuntu running in VMs or LXD containers: by not running sshd when it is not used, we save at least 3MiB of memory in each instance, representing a savings of roughly 5% on an idle, pristine kinetic container.

At Canonical we care about making Ubuntu as efficient as possible on your hardware and in the cloud, which is why this change has been landed as part of a larger effort to reduce the default memory footprint of our images. A default Ubuntu 22.04 LXD image at release time used 65MiB of RAM, which in kinetic now uses 58MiB after this OpenSSH change; and more improvements are in progress, with the intention of backporting the safer changes to our Ubuntu 22.04 images to improve memory usage for the greatest number of users.

On new installs of Ubuntu 22.10 or later, the OpenSSH change in behavior should be completely transparent to users.

On upgrades from Ubuntu 22.04 LTS, users who had configured Port settings or a ListenAddress setting in /etc/ssh/sshd_config will find these settings migrated to /etc/systemd/system/ssh.socket.d/addresses.conf. As an exception, if more than one ListenAddress setting is declared, the configuration is not migrated because systemd’s ListenStream has different semantics: any address configured which is not present at boot time would cause the ssh.socket unit to not start. Because it is not possible to reliably determine at upgrade time whether ssh.socket could fail to start on reboot, if you have more than one ListenAddress configured, your system will not be migrated to socket-based activation but instead the daemon will be started on boot as before.

Socket activation is recommended wherever possible, but if for any reason you find after migration that this is incompatible with your configuration, it is still possible to revert to the previous non-socket-activated behavior. See /usr/share/doc/openssh-server/README.Debian.gz for details.

7 Likes

Works well for me. One thing I verified is that the hosts_access(5) (libwrap) remains the same: it does. I do have a couple of suggestions:

  • We still ship sshd_config with #ListenAddress and #Port lines, with no warning about them not being honored anymore by default. Users editing those will be surprised that they have no effect. I think they should be replaced or complemented by a comment explaining the new way for specifying the listen address and port (maybe by pointing to openssh-server.README.Debian, see below). I see that this requires patching upstream’s sshd_config, which is a bit annoying.
  • There’s useful info about socket-based activation in README.Debian, but I initially failed to find it because it gets installed by the openssh-client package, as it’s the first package specified in d/control. Consider moving the socket activation bits to d/openssh-server.README.Debian in the source package.

TIL about sd_listen_fds(3)!

1 Like

Thanks for the feedback. It was intended to document in the default sshd_config that these options are unused, but that didn’t get done yet. I’ll make sure that’s handled in the next upload.

As far as finding README.Debian, openssh-server has a strict versioned dependency on openssh-client because /usr/share/doc/openssh-server is a symlink to /usr/share/doc/openssh-client - so I’m not sure why this prevented you from finding it?

1 Like

As I was inspecting the package more in general I had a look at the files installed by openssh-server and didn’t see README.Debian. This is not what users will (and should) normally do, I think the current approach is good, thanks for pointing me to it.

1 Like

What will happen if there are Port or ListenAddress entries in an Include file eg in /etc/ssh/sshd_config.d? Thanks

the upgrade code understands and follows Include directives.

2 Likes

Hello,
So in simple terms, how can we change the ssh port on a new install of Ubuntu 22.10?

(I cannot find the right way to do it, after editing the port in /lib/systemd/system/ssh.socket, the port ssh.socket is listening to changed as excepted and it starts successfully, however whenever I try to SSH, ssh.socket on the server side will fail immediately and refuse connections.)

in README.Debian you write

mkdir -p /etc/systemd/system/ssh.socket.d
cat >/etc/systemd/system/ssh.socket.d/listen.conf <<EOF
[Socket]
ListenStream=2222
EOF

this makes sshd listen on port 2222 and port 22.
I think you should mention that for sshd to listen n 2222 only you need to do this:

mkdir -p /etc/systemd/system/ssh.socket.d
cat >/etc/systemd/system/ssh.socket.d/listen.conf <<EOF
[Socket]
ListenStream=
ListenStream=2222
EOF

quote from man systemd.socket

These options may be specified more than once, in which case incoming traffic on any of the sockets will trigger service activation, and all listed sockets will be passed to the service,
regardless of whether there is incoming traffic on them or not. If the empty string is assigned to any of these options, the list of addresses to listen on is reset, all prior uses of any
of these options will have no effect.
2 Likes

Long story short - for all those who do not like this change:

systemctl disable --now ssh.socket
systemctl enable --now ssh.service

As per: /usr/share/doc/openssh-server/README.Debian.gz
Then the /etc/ssh/sshd_config works again with Ports and Addresses settings.

3 Likes

in some cases when updating /etc/systemd/system/ssh.service.d/00-socket.conf is created.
This file forces the use of ssh.socket. You might also check the existence of this file and delete it followed by a systemctl daemon-reload

1 Like

This solution works fine, thanks.