Do-release-upgrade Ubuntu 22.04.5 LTS to Ubuntu 26.04 LTS

Support and Help
1

Ubuntu Version:
Ubuntu 22.04.5 LTS

Desktop Environment (if applicable):
cli

Problem Description:
I am trying to upgrade Ubuntu 22.04.5 LTS to Ubuntu 26.04 LTS

when I try do-release-upgrade, it wants to upgrade to Ubuntu 24.04 LTS – this also fails because

Cannot upgrade system with unmerged /usr

Please install the usrmerge package to fix this, and then try theupgrade again.

When I install usrmerge:

Smartmatch is experimental at /usr/lib/usrmerge/convert-usrmerge line 172.

FATAL ERROR:Both /lib/x86_64-linux-gnu/libnss_files.so.2 and /usr/lib/x86_64-linux-gnu/libnss_files.so.2 exist

These are the two files:

ls -l /lib/x86_64-linux-gnu/libnss_files.so.2 /usr/lib/x86_64-linux-gnu/libnss_files.so.2
-rw-r–r-- 1 root root 14352 Jan 30 08:20 /lib/x86_64-linux-gnu/libnss_files.so.2
-rw-r–r-- 1 root root 14336 Mar 30 2024 /usr/lib/x86_64-linux-gnu/libnss_files.so.2

I have tried following various solutions to the above, involving creating a link, these either fail because there’s no loader any more, so I have to use /bin/busybox to get them back, or, when I do create a link it is overwritten again, as above.

What I’ve Tried:
I have, as I say above, tried all the recommendations I can find.

2

There is no direct upgrade path. One must go through consecutive LTS releases.

What’s the state of the system with regards to 3rd party packages[1][2]? Did you run apt upgrade --update and rebooted prior to do-release-upgrade

Please tell us exactly which recommendations. We can’t know all the ones floating around the internet. Plus, it’s highly dependent on the way you search, what you get as solution suggestions.

  1. ↩︎

  2. ↩︎

1 Like
3

I’m afraid that I didn’t keep the links. The recommendations included deleting one of the two files and replacing it with a link to the other. Deleting the file, of course, stops programs loading (either of the two), and only /bin/busybox enables you to get things running again.

4

Online upgrades can fail because the upgrade scripts cannot take into account modifications we have done to the OS or the third party applications we have installed.

The OS also changes from one LTS to the next. And usrmerge is one such changes. Could it be that you have a third party application that cannot be converted? Removing the application may be necessary. Then reinstall it again.

And explanation of usrmerge

Regards

1 Like
5

What about my other questions? usrmerge is a pretty straight forward process, on a standard system. With 3rd party packages present, nobody knows.

Also, deleting the file in /usr/lib (after making a backup) shouldn’t hurt programs, because the linker always checks /lib too. But doing it manually shouldn’t be necessary to begin with. The one in /usr/lib could well be a remnant of a half-completed previous attempt of usrmerge.

6

Thank you, that’s a useful link. I’m not aware of any 3rd Party installations that could be doing this. I’d happily remove them if I knew what they were. It’s not clear to me why removing either, causes the same failure.

I’m not even clear how I’d go about finding what is responsible.

The strategy of getting a list of everything installed by apt and removing it seems extreme, and if something has been installed by curl, it may not work anyway.

Performing a search of all binaries on the system to find if any have hard-coded links to the wrong file is quite a big project.

I have been hoping that somebody might have met this before, and found that package X does this - but I’ve searched for quite some time and not found a reference to this.

What do you think? Is it worth writing something that finds every executable, shellscript, library, binary on the system and searching each one for the wrong file reference? Would this even be certain to work?

I have also tried doing a fresh install, then trying to updated it from the installed list, but lots of configurations break, and it left everything very messy.

That’s after all, why we have upgrade scripts, rather than re-install - migrate, as the standard method.

7

The point is that such changes were only tested with standard systems, no 3rd party packages, and nobody can give you an exhaustive list of what packages might be interferring. Remove all of them and downgrade the one overriding official ones. See the first link, I provided above.

8

OK. I understand that. My question is how to find all the possible 3rd party packages that may have been installed with curl over the past several years.

I know, for example, that I installed fpc maybe from the site directly. I don’t know, though.

9

Those you won’t find and curltobash is the worst way to install software — I knew that link would come in handy. :stuck_out_tongue:
The point of package management is precisely to keep track of everything installed to easily get rid of, if that’s the desired action. Installing things isn’t the hard problem, as curltobash demonstrates.

BTW, these two are different:

:exclamation_question_mark:
And looking at my own 24.04, the one in /usr/lib must be the correct one, because it’s exactly the same size as here. So it looks like limbo, because usrmerge has obviously done some things already.

1 Like
10

Yes, absolutely! You, and Mark Shuttleworth are preaching to the converted. I agree completely.

I’m looking, as you know, to find a way of recovering from mistakes I have made, and forgotten, over the past several years. To fix the resultant mess.

11

Yes, too, I know the two files are different - that’s why I posted the listing. If they were being accessed by currently running programs, it’s be easy to use lsof, find what they are, kill and uninstall them… sadly, that isn’t the problem I’ve got.

12

Trying to get rid of one:

20:48 root@ganesh:~#ls -l /lib/x86_64-linux-gnu/libresolv.so.2 /usr/lib/x86_64-linux-gnu/libresolv.so.2
-rw-r–r-- 1 root root 68552 Jan 30 08:20 /lib/x86_64-linux-gnu/libresolv.so.2
-rw-r–r-- 1 root root 68104 Mar 30 2024 /usr/lib/x86_64-linux-gnu/libresolv.so.2
20:49 root@ganesh:~#ls -l /lib/x86_64-linux-gnu/libresolv.so.2* /usr/lib/x86_64-linux-gnu/libresolv.so.2*
-rw-r–r-- 1 root root 68552 Jan 30 08:20 /lib/x86_64-linux-gnu/libresolv.so.2
-rw-r–r-- 1 root root 68104 Mar 30 2024 /usr/lib/x86_64-linux-gnu/libresolv.so.2
20:49 root@ganesh:~#mv /usr/lib/x86_64-linux-gnu/libnss_files.so.2 /usr/lib/x86_64-linux-gnu/libnss_files.so.2.old
20:49 root@ganesh:~#ls -l /lib/x86_64-linux-gnu/libresolv.so.2* /usr/lib/x86_64-linux-gnu/libresolv.so.2*
-rw-r–r-- 1 root root 68552 Jan 30 08:20 /lib/x86_64-linux-gnu/libresolv.so.2
-rw-r–r-- 1 root root 68104 Mar 30 2024 /usr/lib/x86_64-linux-gnu/libresolv.so.2

Using root to make it clearer..

13

I think that’s a tall order. The problem is with files that are outside of APT’s radar, and all too many curltobash workflows (over)write files in /usr, instead of /usr/local or /opt[1]. I suspect that’s how the stray one in /lib got there. And it’s probably the first of many such errors to follow.

If you really want to clean house, buckle up. :wink:
First you’ll have to do what can be done with APT, i.e. work your way through the linked thread. Afterwards there must not be any 3rd party packages installed; no orphaned one (no source available), no newer versions of official ones (hence my downgrade suggestion at the bottom). Then you can install debsums and do a full system sweep:

sudo debsums -c | tee ~/debsums-changed.log

That will get you a list of packages with corrupted files; those you need to sudo apt reinstall.

Then you can try again with usrmerge. But files that aren’t provided by any packages may still block the merge, you may need to take manual action, regardless. You can find out which of two files is supposed to be there running dpkg -S /full/path/to/file. If that turns up empty, remove the file; must have been curltobash’d there. :wink:

Or, this just came to me while writing this, you could list all files dpkg knows about:

dpkg -L $(dpkg --get-selections | awk '{print $1}')

and crosscheck with some find magic. What isn’t in /usr/local, /var, /home or /opt should probably not be there. That list is not exhaustive, consult FHS (linked above).

On balance, I think, you’ll find that wiping the slate clean is probably the best option; file it away under paying your dues. :wink: Plus, you can install 26.04 from scratch, without going through 24.04 and having to wait until 26.04.1 is released and LTS release upgrades will be enabled. And afterwards read the first document linked in the footnotes of my first comment.

If you want to compile locally and still keep track of the files, checkinstall might be able to help; it’s a package that helps with the make install — just do checkinstall instead. It’s an old package and perhaps obsolete, but should get the job done; I haven’t used it in a long time, though. If it doesn’t work, you may want to read the packaging guide, or perhaps Debian’s maintainer guide.

  1. see FHS ↩︎

1 Like
14

Addendum

One of the points made in favor of the /usr merge is, that it facilitates mounting it read-only. With some additional configuration, apt can remount it read-write for upgrades, but in normal operation it’s locked, so to speak. That should save you from the worst of curl ... | bash.

1 Like
15

Just a quick bump to keep the topic alive just a tad longer, for I may have some suggestions to make. This is indeed an intriguing topic, after all.

It came to me that the coreutils keep on giving. So I’m thinking that there should be a way to leverage the comm utility for this. This may not be the final solution but here goes:

comm -3 <(dpkg -L $(dpkg --get-selections | awk '{print $1}') | sed -E 's/^(diverted by [0-9a-z.\-]+|package diverts others) to: //' | sort -u) <(find /lib /usr /bin /sbin /var -not -path /usr/local\* | sort -u)

(For brevity this uses Bash extensions of the POSIX standard; <() is like $(), only that the output is piped to a file descriptor.)

That could be all it takes to find files that aren’t managed by APT/Dpkg, but, since I tend to overlook things, I’ll throw this out there for review/thoughts. /usr/local is deliberately excluded, because that’s where locally built and managed software is supposed to go. It might still be a good idea to move it aside during an upgrade, so as to not have some accidental overrides from there, e.g. /usr/local/bin/sudo.

16

The original poster marked this as Solved.

Thanks to everyone who participated in this fascinating topic.

Closed.

1 Like