Do use persistent device names

[peterwhite23]

Why

See this problem:

@corradoventu

Problem Description:
I have 2 nvme volumes, let’s say A and B. If I boot from an OS installed on A I see volume A as nvme0 and B as nvme1. If instead I boot from an OS installed on B I see volume B as nvme0 and A as nvme1. How can I avoid this confusion?

and the solution^[1]:

@thingizkhan

nvme0 and nvme1 are assigned in the order the firmware hands the
controller IDs to the kernel. When you boot from drive A, the firmware
enumerates A first, so it becomes nvme0 and B becomes nvme1; when
you boot from B the order flips. The names are not meant to be stable.

Use the identifiers that never change

While the initial problem may look somewhat niche, the underlying mechanisms can hit anyone off guard and lead them on a wild goose chase^[2]:

@peterwhite23

@pjw9779

/dev/sdb1 /media/pjw/ ntfs users 0 0

Why is there even a line pointing directly to /dev/sdb1? That alone can exhibit undefined behavior

As you can see, the real issue wasn’t even close to being caused by the “Usual Suspect”.

Fear the Switcheroo

Or picture this rather dangerous example, which can really really hurt. Suppose you’ve decided to clone your whole external backup device to an identical one — can never have enough backups, right? You’ve done your due diligence of identifying the source as /dev/sda and destination as /dev/sdb; or, so you thought. Then, however, you get a notification that a reboot is required after some system updates in the background and you follow suit and reboot before proceeding with the cloning. Now you run^[3]:

sudo cp /dev/sda /dev/sdb

Of course, you check if everything indeed was copied correctly, afterwards; there was no error message, so that’s a good start, right? Except, both devices are now empty!

What happened?! When you rebooted, the external devices were enumerated in the order they appeared, just like they were before the restart; the crucial difference being, the order of plugging them in dictated the latter. So, in fact, source and destination had been switched by the restart, because the destination device happened to come up quicker. It is the newer device, after all, and harddisk drives, having motor bearings under constant wear and tear, tend to get ever so slightly slower spinning up with age^[4], so that could be an actual reason for the switcheroo, even though this example may seem a little contrived, which it admittedly is, but I feel like it’s a necessary evil to put the fear of Saint IGNUcius in you, dear reader.

Do use persistent identifiers everywhere

It cannot be stressed enough that there is an inherent raciness built into the way hardware is being discovered an enumerated. So whenever you edit /etc/fstab do use one of those convenient symbolic links in, e.g. /dev/disk/by-id. One way to find a suitable one to a given “legacy” device node, e.g. /dev/sda1 is this:

find -L /dev/disk/by-id -samefile /dev/sda1

This is what I get:

/dev/disk/by-id/scsi-3500a0751e4b94dc3-part1
/dev/disk/by-id/wwn-0x500a0751e4b94dc3-part1
/dev/disk/by-id/scsi-0ATA_CT1000MX500SSD1_2042E4B94DC3-part1
/dev/disk/by-id/ata-CT1000MX500SSD1_2042E4B94DC3-part1
/dev/disk/by-id/scsi-1ATA_CT1000MX500SSD1_2042E4B94DC3-part1
/dev/disk/by-id/scsi-SATA_CT1000MX500SSD1_2042E4B94DC3-part1

Take your pick; they’re all pointing to the same device and everything is unique between the initial interface/bus id and the final partition name. I tend to like the ones which have the model id and the serial number in the name, sometimes enriched with the vendor name. BTW, the Ubuntu installer likes those too, take a look at this excerpt from /etc/fstab of a freshly installed 26.04 VM:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-qtm3uyivYhdcVrdm0VfeBYmUOKJdHjW0RdPvdnMcwjaJXDhGVqBt5P2eHOLQYldm / ext4 defaults 0 1
# /boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/85f27c35-0056-4d79-9730-aa3a70627eef /boot ext4 defaults 0 1
# /boot/efi was on /dev/sda1 during curtin installation
/dev/disk/by-uuid/3864-79FA /boot/efi vfat defaults 0 1
/swap.img	none	swap	sw	0	0

It used to be that the first column was more like UUID=, which never sat well with my mild case of OCD, for it must be an absolute path.

Linux Archeology

Once upon a time, in the aftermath of the cataclysm which wiped the dinosaurs from the face of the computing world and Saint IGNUcius’ blessings had only just begun to make the rounds, thanks to the Almighty Linus, there was a real determinism to these device node names, which probably explains why so many HowTos on the internet neglect to mention the new realities, which even predate udev, IIRC. While such HowTos are not wrong, per se, they never tell you that these device names are only valid until the system shuts down, because the deck gets reshuffled on every system start.

Now, here is where it gets really nasty, because it may seem like those device nodes are consistent across restarts; in case of our two external drives, doing another restart will probably result in the same race, which the newer device is going to win. Don’t be too hung up on these being external devices, BTW, because there it’s somewhat more obvious that you might be tickling the dragon — duh! So everything is smooth sailing for a long time and across every single restart and life is good; until it isn’t, for instance, because the unchallenged winner of all those past races gets a little poorly and takes just a tad longer to get out of bed that fateful morning, when your next boot drops you to the emergency shell, because your local filesystem doesn’t come up, with all kinds of strange errors in the log, which don’t seem to make any sense.

Plus, simply plugging devices in and out can shuffle the names even while the system is still running! You can confirm this yourself, by running this command in a terminal:

journalctl --follow

It follows the log in realtime so you can observe what happens when you plug in devices in different order. For instance, the following illustrates how you can switch “source” and “destination” from our above example, without even realizing:

Kernel log of USB switcheroo

Observe how my Sandisk USB stick starts as sdb, another gets sdc, then both are unplugged and the latter gets plugged in again and gets the recycled sdb.

usb 1-4: USB disconnect, device number 6
usb 1-4: new high-speed USB device number 8 using xhci_hcd
usb 1-4: New USB device found, idVendor=0781, idProduct=5581, bcdDevice= 1.00
usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-4: Product: Ultra
usb 1-4: Manufacturer: SanDisk
usb 1-4: SerialNumber: 4C530001200708109334
usb-storage 1-4:1.0: USB Mass Storage device detected
scsi host1: usb-storage 1-4:1.0
scsi 1:0:0:0: Direct-Access     SanDisk  Ultra            1.00 PQ: 0 ANSI: 6
sd 1:0:0:0: Attached scsi generic sg1 type 0
sd 1:0:0:0: [sdb] 60062500 512-byte logical blocks: (30.8 GB/28.6 GiB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 43 00 00 00
sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
 sdb: sdb1 sdb2 sdb3 sdb4
sd 1:0:0:0: [sdb] Attached SCSI removable disk
usb 1-3: USB disconnect, device number 7
usb 1-3: new high-speed USB device number 9 using xhci_hcd
usb 1-3: New USB device found, idVendor=058f, idProduct=6387, bcdDevice= 1.02
usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-3: Product: Mass Storage
usb 1-3: Manufacturer: Generic
usb 1-3: SerialNumber: 8F46C957
usb-storage 1-3:1.0: USB Mass Storage device detected
scsi host2: usb-storage 1-3:1.0
scsi 2:0:0:0: Direct-Access     Generic  Flash Disk       8.07 PQ: 0 ANSI: 4
sd 2:0:0:0: Attached scsi generic sg2 type 0
sd 2:0:0:0: [sdc] 7864320 512-byte logical blocks: (4.03 GB/3.75 GiB)
sd 2:0:0:0: [sdc] Write Protect is off
sd 2:0:0:0: [sdc] Mode Sense: 23 00 00 00
sd 2:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sdc] Attached SCSI removable disk
usb 1-4: USB disconnect, device number 8
usb 1-3: USB disconnect, device number 9
usb 1-3: new high-speed USB device number 10 using xhci_hcd
usb 1-3: New USB device found, idVendor=058f, idProduct=6387, bcdDevice= 1.02
usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-3: Product: Mass Storage
usb 1-3: Manufacturer: Generic
usb 1-3: SerialNumber: 8F46C957
usb-storage 1-3:1.0: USB Mass Storage device detected
scsi host1: usb-storage 1-3:1.0
scsi 1:0:0:0: Direct-Access     Generic  Flash Disk       8.07 PQ: 0 ANSI: 4
sd 1:0:0:0: Attached scsi generic sg1 type 0
sd 1:0:0:0: [sdb] 7864320 512-byte logical blocks: (4.03 GB/3.75 GiB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 23 00 00 00
sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
sd 1:0:0:0: [sdb] Attached SCSI removable disk

---

1. emphasis added ↩︎

2. Look at the initial suspect! ↩︎

3. Yes, that does work, for everything is a file; silly how other OS’s need “cloning” tools, right? ↩︎

4. Spin-up time is an early failure indicator in S.M.A.R.T. ↩︎

[paddylandau]

There’s nothing wrong with using UUID — unless you manually make one device’s UUID equal to another’s, which is truly risky!

@peterwhite23

it must be an absolute path.

Why does it have to be an absolute path? It needs only to be guaranteed to be unique.

[peterwhite23]

@paddylandau

There’s nothing wrong with using UUID

It’s a semantics issue. /etc/fstab traditionally required an actual device path in the first column. As I said, (mild) OCD. It isn’t wrong, per se, but also totally unnecessary, for there’s always /dev/disk/by-uuid to get the same outcome. Take a look again at that excerpt from the fstab generated by the 26.04 Ubuntu installer. Apparently, someone in foundations agrees. Not to toot my own horn, but, because I never liked the unwieldy UUID’s, I either label my devices and use /dev/disk/by-*label/ or just use /dev/disk/by-id/. The latter has the benefit that I can find the physical device in question by looking at its label sticker, since Model ID and Serial Number are always somewhere in the name of native partitions — device mapper (e.g. LVM) is another matter but also doesn’t suffer from the ambiguity to begin with.

@peterwhite23

it must be an absolute path.

Why does it have to be an absolute path? It needs only to be guaranteed to be unique.

Consider it a matter of style, just like I must wear my sunglasses at night.
https://www.youtube.com/watch?v=X2LTL8KgKv8

---

Essentially, I consider UUID= a hack that stuck around too long, for there may have been a brief moment in time, when the /dev/disk/by-* hierarchies hadn’t existed yet; pre-udev that might have been, IIRC.

---

Oh, and there was this one time when the UUID of my /boot did change for no apparent reason after a release upgrade; that was a “fun” afternoon.

[paddylandau]

Thank you, Peter. That makes tremendous sense, and I’ve updated my notes to reflect this. I like the idea of using the device’s serial number. Looking at my scripts for automating my backups, I see that I’ve actually used it without remembering having done so!