Ubuntu 22.04.3 LTS hit a roadblock on apt upgrade. "Operation not permitted" on systemd and many suspicious about Apparmor

super-ditto · May 10, 2025, 9:08am

Hello guys, I’m stuck on ‘apt upgrade’ when upgrading our production Ubuntu server. Here are some related commands:

Ubuntu Version:
Ubuntu 22.04.05 LTS

Problem Description:

$ apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libnss-systemd : Depends: systemd (= 249.11-0ubuntu3.15) but 249.11-0ubuntu3.12 is installed
 libpam-systemd : Depends: systemd (= 249.11-0ubuntu3.15) but 249.11-0ubuntu3.12 is installed
 systemd : Depends: libsystemd0 (= 249.11-0ubuntu3.12) but 249.11-0ubuntu3.15 is installed
 systemd-sysv : Depends: systemd (= 249.11-0ubuntu3.15) but 249.11-0ubuntu3.12 is installed
 udev : Breaks: systemd (< 249.11-0ubuntu3.15) but 249.11-0ubuntu3.12 is installed
        Recommends: systemd-hwe-hwdb but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).



$ apt --fix-broken install

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  systemd
Suggested packages:
  systemd-container libtss2-rc0
The following packages will be upgraded:
  systemd
1 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
4 not fully installed or removed.
Need to get 0 B/4,581 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
(Reading database ... 138971 files and directories currently installed.)
Preparing to unpack .../systemd_249.11-0ubuntu3.15_amd64.deb ...
Unpacking systemd (249.11-0ubuntu3.15) over (249.11-0ubuntu3.12) ...
dpkg: error processing archive /var/cache/apt/archives/systemd_249.11-0ubuntu3.15_amd64.deb (--unpack):
 unable to make backup link of './lib/systemd/systemd-shutdown' before installing new version: Operation not permitted
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/systemd_249.11-0ubuntu3.15_amd64.deb
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Well I think the key line is unable to make backup link of ‘./lib/systemd/systemd-shutdown’ before installing new version: Operation not permitted

Relevant System Information:

$ cat /etc/os-release 

PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -r
5.15.0-91-generic

What I’ve Tried:
At first, I did some research and eliminated one possibility regarding a file/folder permission issue(immutable tag).

Then the log hits the problem around AppArmor, I tried hard to disable it temporary but all failed.

After failure on apt --fix-broken install I try to immediately run

$ sudo grep -i "apparmor.*denied" /var/log/syslog /var/log/kern.log | tail -n 10

/var/log/kern.log:May 10 08:17:00 node4 kernel: [5067947.095355] audit: type=1400 audit(1746836220.203:276): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/dpkg" pid=2006043 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
/var/log/kern.log:May 10 08:17:00 node4 kernel: [5067947.096986] audit: type=1400 audit(1746836220.203:277): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/dpkg" pid=2006047 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
/var/log/kern.log:May 10 08:33:49 node4 kernel: [  268.065807] audit: type=1400 audit(1746837229.082:45): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/ubuntu-distro-info" pid=20565 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
/var/log/kern.log:May 10 08:33:49 node4 kernel: [  268.109429] audit: type=1400 audit(1746837229.126:46): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/ubuntu-distro-info" pid=20615 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
/var/log/kern.log:May 10 08:33:49 node4 kernel: [  268.111614] audit: type=1400 audit(1746837229.130:47): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/dpkg" pid=20618 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
/var/log/kern.log:May 10 08:33:49 node4 kernel: [  268.113189] audit: type=1400 audit(1746837229.130:48): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/dpkg" pid=20619 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
/var/log/kern.log:May 10 16:24:48 node4 kernel: [28527.119412] audit: type=1400 audit(1746865488.301:49): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/ubuntu-distro-info" pid=401252 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
/var/log/kern.log:May 10 16:24:48 node4 kernel: [28527.131312] audit: type=1400 audit(1746865488.313:50): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/ubuntu-distro-info" pid=401271 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
/var/log/kern.log:May 10 16:24:48 node4 kernel: [28527.133795] audit: type=1400 audit(1746865488.317:51): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/dpkg" pid=401281 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
/var/log/kern.log:May 10 16:24:48 node4 kernel: [28527.136301] audit: type=1400 audit(1746865488.321:52): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache" name="/usr/bin/dpkg" pid=401285 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

So I think the problem may be related to AppArmor. I try to disable it, even temporarily.
I tried all below and all failed:

systemctl stop/disable AppArmor
edit /etc/default/grub change a line to GRUB_CMDLINE_LINUX_DEFAULT="apparmor=0 security=none" and reboot
Move the suspicious profile away (mv /etc/apparmor.d/local/ubuntu_pro_esm_cache to somewhere else ) and try to reload AppArmor.

$ mkdir /etc/apparmor.d/backup/

$ mv  /etc/apparmor.d/local/ubuntu_pro_esm_cache /etc/apparmor.d/backup/ubuntu_pro_esm_cache

$ systemctl reload apparmor

$ sudo aa-status 

43 profiles are in enforce mode.
   ...
   ubuntu_pro_esm_cache
   ubuntu_pro_esm_cache//apt_methods
   ubuntu_pro_esm_cache//apt_methods_gpgv
   ubuntu_pro_esm_cache//cloud_id
   ubuntu_pro_esm_cache//dpkg
   ubuntu_pro_esm_cache//ps
   ubuntu_pro_esm_cache//ubuntu_distro_info
   ubuntu_pro_esm_cache_systemctl
   ubuntu_pro_esm_cache_systemd_detect_virt

jaime-cruz · May 10, 2025, 11:30am

The only question I’d ask is where did you get 24.04.5LTS? Canonical hasn’t released 24.04.3LTS yet…

Oh, disregard. Your SUBJECT line says 24.04.5LTS but I see in the body of your message you said 22.04.5LTS…

super-ditto · May 10, 2025, 11:33am

Sorry It’s my typo on title. Should be 22.04.5LTS

rubi1200 · May 13, 2025, 7:11am

A post was split to a new topic: Ubuntu Update Issues

super-ditto · May 13, 2025, 10:10am

Some dig using strace:

strace -o dpkg_strace.log -ff -s 256 -v dpkg --unpack /var/cache/apt/archives/systemd_249.11-0ubuntu3.15_amd64.deb
grep -rwn ./dpkg_strace.log.* -e 'EPERM'
cat dpkg_strace.log.3676290 | grep EPERM -A 5 -B 5

it shows:

close(13)                               = 0
utimensat(AT_FDCWD, "/lib/systemd/systemd-shutdown.dpkg-new", [{tv_sec=1747130619, tv_nsec=0} /* 2025-05-13T18:03:39+0800 */, {tv_sec=1740057842, tv_nsec=0} /* 2025-02-20T21:24:02+0800 */], 0) = 0
link("/lib/systemd/systemd-shutdown", "/lib/systemd/systemd-shutdown.dpkg-tmp") = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)