SSHd now uses socket-based activation (Ubuntu 22.10 and later)

paride · August 26, 2022, 9:44am

As I was inspecting the package more in general I had a look at the files installed by openssh-server and didn’t see README.Debian. This is not what users will (and should) normally do, I think the current approach is good, thanks for pointing me to it.

phil-hughes · October 21, 2022, 1:47pm

What will happen if there are Port or ListenAddress entries in an Include file eg in /etc/ssh/sshd_config.d? Thanks

vorlon · October 21, 2022, 2:45pm

the upgrade code understands and follows Include directives.

yanchenksea · October 22, 2022, 9:44am

Hello,
So in simple terms, how can we change the ssh port on a new install of Ubuntu 22.10?

(I cannot find the right way to do it, after editing the port in /lib/systemd/system/ssh.socket, the port ssh.socket is listening to changed as excepted and it starts successfully, however whenever I try to SSH, ssh.socket on the server side will fail immediately and refuse connections.)

saxl · October 22, 2022, 1:35pm

in README.Debian you write

mkdir -p /etc/systemd/system/ssh.socket.d
cat >/etc/systemd/system/ssh.socket.d/listen.conf <<EOF
[Socket]
ListenStream=2222
EOF

this makes sshd listen on port 2222 and port 22.
I think you should mention that for sshd to listen n 2222 only you need to do this:

mkdir -p /etc/systemd/system/ssh.socket.d
cat >/etc/systemd/system/ssh.socket.d/listen.conf <<EOF
[Socket]
ListenStream=
ListenStream=2222
EOF

quote from man systemd.socket

These options may be specified more than once, in which case incoming traffic on any of the sockets will trigger service activation, and all listed sockets will be passed to the service,
regardless of whether there is incoming traffic on them or not. If the empty string is assigned to any of these options, the list of addresses to listen on is reset, all prior uses of any
of these options will have no effect.

twinsen · October 23, 2022, 9:52pm

Long story short - for all those who do not like this change:

systemctl disable --now ssh.socket
systemctl enable --now ssh.service

As per: /usr/share/doc/openssh-server/README.Debian.gz
Then the /etc/ssh/sshd_config works again with Ports and Addresses settings.

saxl · October 24, 2022, 6:38am

in some cases when updating /etc/systemd/system/ssh.service.d/00-socket.conf is created.
This file forces the use of ssh.socket. You might also check the existence of this file and delete it followed by a systemctl daemon-reload

nebulabox · January 12, 2023, 4:51am

This solution works fine, thanks.

michael.utech · January 28, 2023, 6:17pm

I’m trying to bind to an IP address assigned by DHCP by mac and that fails, apparently the socket is created before the IP is set up.

I want to exclude all but this interface from being served sshd. How am I going about that with ssh.socket?

What is the benefit of ssh.socket? The price is that ssh configuration is now split between the actual ssh daemon and systemd. I rarely read ssh documentation or even the comments in sshd_config anymore and was just baffled first that my settings are ignored, then that there was not /etc/systemd/system/ssh.socket.d but an ssh.service.d that apparently was not used. ssh.socket.d does of course not use sshd_config syntax. The package is named ssh, the documentation is named openssh. I had to find out that the actual documentation is in systemd.socket. All that was a bit annoying, since I had something else to accomplish and just wanted to log into that box.

It sounds like in the near future, ssh.service will be deprecated or somehow disabled, so I’m not sure if it’s a good idea to just ignore it and revert to ssh.service.

There must be some substantial benefit of using the socket mechanism? I didn’t find any mention anywhere what that in the case of sshd might be…

piradix · April 5, 2023, 3:11pm

The problem with not using “/etc/ssh/sshd_config” is that if you stop sshd (sshd stop service), and start it (sshd start service) it takes the port from the “/etc/ssh/sshd_config” configuration file, and not from “/etc/systemd/system/ssh.socket.d/listen.conf”!
For me this is not a very good solution because it generates human errors!
A small comment in the config file on “#Port” would be a minimum and welcome
If you like a comment … to avoid the admin to waste their time

hrudy · April 19, 2023, 8:20pm

I am still wondering why Ubuntu would change something that has been working since the last century. For me the fixes didn’t work until I reverted back to the old configuration. So for me, this is 3 strikes.
1 strike for changing it in the first place.
2nd for the socket-based fix not working
3rd strike for inadequate documentation.
I did a search as to why socket based activation is “superior”. Supposedly it is there to minimize memory usage. Allegedly this reduces memory from 65MiB to 58Mib. I guess if this makes the ppl at Canonical happy but even a Raspberry PI 3B has 1 GB of memory.

dsembry · April 23, 2023, 4:02am

I’m having similar issues with sshd on my ubuntu 22.10 installation. I have followed the instructions in /usr/share/doc/openssh-server/README.Debian.gz:

  systemctl disable --now ssh.socket
  systemctl enable --now ssh.service

however, as the outputs of a few different commands I’ve typed show, ssh.socket is still running and systemd is still listening to port 22.

root@bournemouth:/usr/share/doc/openssh-client# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/ssh.service.d
             └─00-socket.conf
     Active: active (running) since Sat 2023-04-22 23:42:40 EDT; 10min ago
TriggeredBy: ● ssh.socket
...

root@bournemouth:/usr/share/doc/openssh-client# systemctl status ssh.socket
● ssh.socket - OpenBSD Secure Shell server socket
     Loaded: loaded (/lib/systemd/system/ssh.socket; disabled; preset: enabled)
     Active: active (running) since Sat 2023-04-22 23:42:40 EDT; 10min ago
      Until: Sat 2023-04-22 23:42:40 EDT; 10min ago
   Triggers: ● ssh.service
     Listen: [::]:22 (Stream)
      Tasks: 0 (limit: 9365)
     Memory: 4.0K
        CPU: 1ms
     CGroup: /system.slice/ssh.socket
...

root@bournemouth:/usr/share/doc/openssh-client# lsof -iTCP -sTCP:LISTEN 
COMMAND     PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd       1            root   60u  IPv6 214824      0t0  TCP *:ssh (LISTEN)
systemd       1            root  345u  IPv4  17890      0t0  TCP *:sunrpc (LISTEN)
systemd       1            root  347u  IPv6  17891      0t0  TCP *:sunrpc (LISTEN)
rpcbind     685            _rpc    4u  IPv4  17890      0t0  TCP *:sunrpc (LISTEN)
rpcbind     685            _rpc    6u  IPv6  17891      0t0  TCP *:sunrpc (LISTEN)
systemd-r   687 systemd-resolve   14u  IPv4  24694      0t0  TCP localhost:domain (LISTEN)
systemd-r   687 systemd-resolve   16u  IPv4  24696      0t0  TCP localhost:domain (LISTEN)
cupsd      5583            root    7u  IPv6  70007      0t0  TCP ip6-localhost:ipp (LISTEN)
cupsd      5583            root    8u  IPv4  70008      0t0  TCP localhost:ipp (LISTEN)
sshd      12927            root    3u  IPv6 214824      0t0  TCP *:ssh (LISTEN)

vorlon · April 23, 2023, 4:06pm

Sorry, it appears the documenation became inaccurate once we added /etc/systemd/system/ssh.service.d/00-socket.conf. If you remove this file (or rename it to be inactive) and then run the other two commands, it works as expected. Opened https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2017434 to track resolution of the documentation bug.

jpf · July 5, 2023, 6:51am

Sorry if I’m missing the obvious, but documentation in Ubuntu in general is so terrible to find…

After changing the port in /etc/systemd/system/ssh.socket.d, what do I need to do to make it take effect ? (without rebooting the machine that is)

jpf · July 5, 2023, 7:07am

According to /usr/share/doc/openssh-server/README.Debian.gz (again, documetnation is spread everywhere, it’s hours to find anything), to make it pick up on the new address should require systemctl daemon-reload.

Except it doesn’t work … nothing I do seems to make it listen to my selected port

Disabling the system socket thing makes it work:

systemctl disable --now ssh.socket
systemctl enable --now ssh.service

Now it is listening to the new port.

jpf · July 5, 2023, 7:22am

A bit more searching and this stack overflow question answered it:

I needed to add and extra “ListenAddress=” line, with an empty address !? Yes … those undocumented mysteries … But then it works.

geenat · July 17, 2023, 12:05pm

Breaks SSH access for anyone using /etc/ssh/sshd_config on Ubuntu 22.10, 23.04, 23.10…

RIP those who block Port 22. Hope ya’ll had backups or physical access.

@vorlon
NEEDS to be added to the OP… make this google-able.

To go back to previous SSH behavior:

sudo rm /etc/systemd/system/ssh.service.d/00-socket.conf
systemctl disable --now ssh.socket
systemctl enable --now ssh.service
sudo systemctl daemon-reload
sudo systemctl restart ssh

Thanks

clopi · August 1, 2023, 3:10pm

Thanks @geenat for your comment.
It took me like an hour trying to figure out why I couldn’t set a custom SSH port, then read about the “issue” and then get the official revert commands to work. The official documentation in regards to reverting is pretty much worthless, as it only mentions 3 out of 5 steps, and have I not found this thread I’d still be wondering what I’m doing wrong.

mrgrymreaper · August 4, 2023, 10:52am

How does SSHd on socket respond with Cockpit installed? As remote server connections with Cockpit are handled by SSHd, cause if Cockpit is expecting it as a service then well, YIKES if it doesn’t start in time by socket.

I hope fixes are in place upstream for Cockpit as this is software which can likely be irritated by this change.

ajh · August 4, 2023, 10:21pm

I can confirm this does NOT work when running in a container. The transient directory /run/sshd is NOT getting created in the start process and therefore sshd dies on boot.