Where to begin?
I’m in hopes this is in a appropriate area as truly it maybe technical, have questions but not asking for assistance rather for a conversation / discussion on ZFS on Ubuntu in a headless File Server role. Nothing more, nothing less, it serves no purpose other that serve files, mostly to other Ubuntu systems i.e. media servers are using Jellyfin and Plex. All the media is on the NFS, the media servers are bare and rely on the NFS.
@anon36188615 and I had a discussion today about using the lounge for this purpose and to keep the clutter /noise down on the support side. Plus the drives that I mentioned to him have arrived. currently the system is in scrub (should be completed in 22 minutes or so).
In that thread I posed a question of drive width for a RAIDz2. And really outside Rick and I the question really gained no traction.
So Moderators if this really belongs somewhere else or is just inappropriate please advise or move. As I mentioned in the support side I view myself as a guest on this site. So thank you in advance.
Looking at the history of the system I went with RAIDZ2 x 9 drives width for the vdev for a pool named mediapool1.
History for 'mediapool1':
2024-10-22.09:48:04 zpool create -f mediapool1 raidz2 /dev/disk/by-partuuid/11a27fcc-ebbd-4864-9bc9-5cc7f01dc785 1ccc6753-a8af-41a9-8a3e-3ee420d90f81 507cbe23-0408-48b3-bfbe-e589d19ef8fe 15bf2071-4741-4bde-9b95-15aa69e50c61 169b4665-7aa6-4a6e-bec0-f7364f7097c4 72d42ad6-ebeb-4fb7-a276-51285483bfba 714d75bb-9b2e-43c2-b38b-e4068b21e105 ca189b0c-4e82-49c0-80fe-aa6f9243f01c 9431154c-4449-4716-9c6e-b51e436721d3
2024-10-22.09:53:48 zfs set compress=lz4 mediapool1
2024-10-22.09:55:57 zpool set autoexpand=on mediapool1
2024-10-22.21:21:13 zpool export mediapool1
2024-10-22.21:21:32 zpool import -d /dev/disk/by-id mediapool1
2024-10-22.21:22:22 zpool export mediapool1
2024-10-22.21:23:19 zpool import -d /dev/disk/by-partuuid mediapool1
2024-10-22.21:24:47 zpool export mediapool1
2024-10-22.21:25:03 zpool import -d /dev/disk/by-partuuid mediapool1
2024-10-23.23:32:24 zpool export mediapool1
2024-10-23.23:34:50 zpool import -d /dev/disk/by-id mediapool1
2024-10-23.23:41:01 zpool export mediapool1
2024-10-23.23:41:37 zpool import -d /dev/disk/by-vdev mediapool1
Later after creating the pool I ran across a post elsewhere discussing using alias’s for the drives. Which I thought would be a great way to ID the drive in a faulted /degraded status for replacement. here is the/etc/zfs/vdev_id.conf file I used
# by-vdev
#------------------------------------------------------------------------------
# setup for mediapool1 raidz2 9 drives wide external bays add 1 spare use internal bay slot
# For an additional pool use internal drive bays within beastie's case Then expand by
#------------------------------------------------------------------------------
# name fully qualified or base name of device link
alias beastdrive1 /dev/disk/by-id/wwn-0x5000c500869ae7bf-part1
#SN ZC116KNN
alias beastdrive2 wwn-0x5000c50093d69a37-part1
#SN ZC11WJ9N
alias beastdrive3 wwn-0x5000c500957d945f-part1
#SN ZC16AC1P
alias beastdrive4 wwn-0x5000c50085716b6b-part1
#SN Z1ZAYE7F
alias beastdrive5 wwn-0x5000c500855fcd97-part1
#SN Z1ZAVMFC
alias beastdrive6 wwn-0x5000c500855fc8c7-part1
#SN Z1ZAVMM6
alias beastdrive7 wwn-0x5000c500631003db-part1
#SN Z1Z77YLN
alias beastdrive8 wwn-0x5000c500579d0a2b-part1
#SN Z1Z2DRWH
alias beastdrive9 wwn-0x5000c500579c809b-part1
#SN Z1Z2DBZC
#alias drive10 /dev/disk/by-id/
#SN XXXXXXXX
#alias drive11 /dev/disk/by-id/
#SN XXXXXXXX
#alias drive12 /dev/disk/by-id/
#SN ZC16AC1P
#alias drive13 /dev/disk/by-id/
#SN XXXXXXXX
#alias drive14 /dev/disk/by-id/
#SN XXXXXXXX
# past this point must go outside Beastie's case
#--------------------------------------------------------
# once fully edited save then issue> sudo udevadm trigger
# note that the alias name is not visable to ls / lsblk or blkid but
# once the zpool is created the zpool will ID by alais name in status and informational reports
In the disk-by -id I chose the wwn# simply because it as well as the SN is usually on the drive label.
mike@Beastie:~$ sudo zpool status
[sudo] password for mike:
pool: mediapool1
state: ONLINE
scan: scrub in progress since Wed Dec 4 16:33:35 2024
7.89T / 7.89T scanned, 6.41T / 7.89T issued at 1.15G/s
0B repaired, 81.23% done, 00:22:02 to go
config:
NAME STATE READ WRITE CKSUM
mediapool1 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
beastdrive1-part1 ONLINE 0 0 0
beastdrive2-part1 ONLINE 0 0 0
beastdrive3-part1 ONLINE 0 0 0
beastdrive4-part1 ONLINE 0 0 0
beastdrive5-part1 ONLINE 0 0 0
beastdrive6-part1 ONLINE 0 0 0
beastdrive7-part1 ONLINE 0 0 0
beastdrive8-part1 ONLINE 0 0 0
beastdrive9-part1 ONLINE 0 0 0
errors: No known data errors
Now as I mentioned in the other post on the support side I’ll practice on drive replacement (of course I’ll wait until the scrub is completed).
But I was researching the ZFS set autoreplace command. Which according to a 4 year old post on Stack Exchange Ubuntu doesn’t really do a good job of doing. But hey that was 4 years ago
without deleving too far and only to provide a bit of background here is a snippet.
**autoreplace=on | off** Controls automatic device replacement. If set to "off", device replacement must be initiated by the administrator by using the "zpool replace" command. If set to "on", any new device, found in the same physical location as a device that previously belonged to the pool, is automatically formatted and replaced. The default behavior is "off". This property can also be referred to by its shortened column name, "replace".
So I will enable the feature and replace one of the drive with HP 4TB SAS SN NHG8JUYN
wwn-5000CCA2430F88B0 labeled drive. IF all goes I only expect to have to update my configuration file export and import again to have the drive labeled the same as the replaced drive
9:41PM CST update
I set the autoreplace =on … think that it would be as simple as removing a drive and installing a drive directly in it’s place within the drive enclosure. I either failed at something or the feature still is lacking.
SO I did this
2024-12-04.21:28:45 zpool offline mediapool1 beastdrive1-part1
2024-12-04.21:31:51 zpool replace -f mediapool1 /dev/disk/by-vdev/beastdrive1-part1 /dev/disk/by-id/wwn-0x5000cca2430f81c8-part1
here is current status for those interested
mike@Beastie:~$ sudo zpool status
pool: mediapool1
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Dec 4 21:31:41 2024
6.43T / 7.89T scanned at 7.44G/s, 648G / 7.89T issued at 749M/s
72.0G resilvered, 8.02% done, 02:49:16 to go
config:
NAME STATE READ WRITE CKSUM
mediapool1 DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
replacing-0 DEGRADED 0 0 0
beastdrive1-part1 OFFLINE 0 0 0
wwn-0x5000cca2430f81c8-part1 ONLINE 0 0 0 block size: 512B configured, 4096B native (resilvering)
beastdrive2-part1 ONLINE 0 0 0
beastdrive3-part1 ONLINE 0 0 0
beastdrive4-part1 ONLINE 0 0 0
beastdrive5-part1 ONLINE 0 0 0
beastdrive6-part1 ONLINE 0 0 0
beastdrive7-part1 ONLINE 0 0 0
beastdrive8-part1 ONLINE 0 0 0
beastdrive9-part1 ONLINE 0 0 0
errors: No known data errors
Now for the over 60 year old Military Retiree to do the adult beverage thing sip on three fingers of Proper number12 while it does it’s thing …
The resilver completed however
mike@Beastie:~$ sudo zpool status
[sudo] password for mike:
pool: mediapool1
state: ONLINE
status: One or more devices are configured to use a non-native block size.
Expect reduced performance.
action: Replace affected devices with devices that support the
configured block size, or migrate data to a properly configured
pool.
scan: resilvered 898G in 03:20:17 with 0 errors on Thu Dec 5 00:51:58 2024
config:
NAME STATE READ WRITE CKSUM
mediapool1 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
wwn-0x5000cca2430f81c8-part1 ONLINE 0 0 0 block size: 512B configured, 4096B native
beastdrive2-part1 ONLINE 0 0 0
beastdrive3-part1 ONLINE 0 0 0
beastdrive4-part1 ONLINE 0 0 0
beastdrive5-part1 ONLINE 0 0 0
beastdrive6-part1 ONLINE 0 0 0
beastdrive7-part1 ONLINE 0 0 0
beastdrive8-part1 ONLINE 0 0 0
beastdrive9-part1 ONLINE 0 0 0
errors: No known data errors
Yeah first thoughts are OHHH Nooo … honestly not a issue it is a mismatch in the drive geometry (it’s a newer drive, coupled with the o-ashift value setting when the pool was setup) … now how to fix it.
- replace enough drives in the existing pool to create a new pool with the older 512b drives this will be to offload the data within the pool . So I can do 4 or 5 (leaning towards 5 for this as it affords enough slope) drives in a raidz1 as it’s just to holding data long enough to destroy the mismatched existing pool (mediapool1), and recreate the pool exactly as before but with the newer drives. Then transfer data back into a properly aligned pool Now will I lose some usable space on the new 4kib pool vs the old 512b yes a little.
or - replace the mismatched one with the drive I pulled.
Now will the vdev within the pool be somewhat affected in performance yes if chose to do nothing. So let me proceed with option 1 as I do have a total of nine drives with native 4Kib and I’ll post the results after all the drives in the vdev is replaced.
If I had a Expander on hand it would save the day, because my HBA will only handle 16 drives. I can’t put the new drives into matching size pool . 9+9 = 18 …
Just a quick Update
Replaced enough drives in the original pool to create a Temp Pool to off load data to. Just waiting on the move to complete.
- I didn’t set the ashift=12 when I created that pool , hence the error as it was set to the default. To set it to support a 4kib requires destroying the pool. < self inflicted
- I did try two differing methods on the replace, which I found out that actually if you don’t offline the drive to be replaced the resilver is actually faster. In my case it was a 30 to 40 min difference per drive that was replaced vs offlining the drive to be replaced.
I’ll reset the pool back up same except for the default ashift valve, so if in the future I use a AF drive I won’t have that happen again.
kind of sharing notes really I guess, as honestly I’m just staring out with the ZFS filesystem. And learning it from the ground up,