Open ZFS Version 2.3 on Plucky Puffin Test

Lounge
1

I know that Version 2.3 is tested / released from the OpenZFS side.
And is posted on the unsupported PPA’s for install. ( uploaded I think yesterday is what I seen).
Just wondering which version of Ubuntu will have this version 2.3 in lieu of the current 2.2 or is /will it incorporated into the test version, before public release.

Just a question that some might be able to answer as honestly not a rush on my end .
I would much rather it be tested etc etc… with Ubuntu and stable before release. (yet in the same breath if it was incorporated into a test version of Ubuntu Server I would happily test it).

2

@anon36188615
As some of ya’ll have been watching I did install new hardware in deepblue the new backup server as I had asked the question above. Getting no response I made the choice to…
install Plucky Puffin 25.04 build aka Plucky Puffin (development branch) (GNU/Linux 6.12.0-12-generic x86_64)

I removed MDADM so no conflicts with ZFS.
Once the base OS was installed I sought out the uploaded ZFS unsupported (yet) ZFS version 2.3.
My reason for doing so was I wanted to look at upcoming features 25.04 as well as ZFS version 2.3 specifacially the adding a drive to a exist zpool. which earlier versions did support doing so with mirrors but not raidz pools.
Some background info
GitHub brief on new features
I went here for the new ZFS v 2.3 repository and instructions

I utilized a by pci path in vdev_id.conf to also test the autoreplace a failing drive.
sample file

#_____________________________Start of File_______________________________________________
#           vdev_id.conf file using disk by-path (pci) alaises
# for drive information issue >  lsblk -o type,name,label,partlabel,size,model,serial,wwn,uuid 
# this will /should provide enough information to edit this file UNLESS configuration is by-path
# To get the pci-path issue > ls -lh /dev/disk/by-path |grep sd[?]
#_________________________________________________________________________________________
# setup for backup pool raidz 4 drives wide x 1-vdev external bays             
# will test the attach command to add a drive to existing pool
# 
# 
#_________________________________________________________________________________________
#     name               fully qualified or base name of device link
#_________________________________________________________________________________________
#------------ backup start -------------------------
alias SAS0               pci-0000:03:00.0-sas-phy6-lun-0
#SN Z1Z77Y
alias SAS1               pci-0000:03:00.0-sas-phy7-lun-0
#SN Z1ZAVM
alias SAS2               pci-0000:03:00.0-sas-phy5-lun-0
#SN ZC116K
alias SAS3               pci-0000:03:00.0-sas-phy4-lun-0
#SN ZC11WJ
alias SAS4               pci-0000:04:00.0-sas-phy3-lun-0
#SN Z1Z2DB
#alias SAS5 not used yet               wwn-0x5000cca2430f90fc
#SN NHG8KD2N
#___________________________________________________________________________________________
# once fully edited save then issue> sudo udevadm trigger
# validate new disk aliases are working by issue> cd /dev/disk/by-vdev && ll
# note that the alias name will not be visable to ls / lsblk or blkid but
# once the zpool is created the zpool will ID by alais name in zpool status and list reports
# if pool is created without alias export the pool then import with
# >  sudo import -d /dev/disk/by-vdev [poolname]
#_____________________________________________End Of File____________________________________

created the pool

sudo zpool create -f -o ashift=9 -o autoexpand=on -o autoreplace=on backup raidz SAS0 SAS1 SAS2 SAS3

then set up features tuned to media that I use on the NFS

sudo zfs set compression=lz4 recordsize=1M xattr=sa atime=off backup

checked the status of the pool …

mike@deepblue-mi6-puffin:~$ sudo zpool status
  pool: backup
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        backup      ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            SAS0    ONLINE       0     0     0
            SAS1    ONLINE       0     0     0
            SAS2    ONLINE       0     0     0
            SAS3    ONLINE       0     0     0

good so far now let’s add SAS4 drive to the pool

sudo zpool add -f backup SAS4

now to check status and see if it was truely added to the raidz1-0 pool

mike@deepblue-mi6-puffin:~$ sudo zpool status
  pool: backup
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        backup      ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            SAS0    ONLINE       0     0     0
            SAS1    ONLINE       0     0     0
            SAS2    ONLINE       0     0     0
            SAS3    ONLINE       0     0     0
          SAS4      ONLINE       0     0     0

errors: No known data errors

Now the downside to adding a drive to a existing raidz pool (note that SAS4 is offset in the raidz1-0)
a. can only add one drive at a time…
b. because the drive is a newcomer the strip width remains the same as the original configuration. Example I used 4 drive for the vdev thus the stripe will remain at 4 drive the newly added drive doesn’t increase the stripe to 5. The result is a penalty on the 5th drive one does lose the storage cap of a stripe in expanding the storage capacity. No at terrible penalty and actually a small one.
so let look at list

mike@deepblue-mi6-puffin:~$ sudo zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
backup  18.2T   298K  18.2T        -         -     0%     0%  1.00x    ONLINE  -

To get ready for the auto replace I need to add some actual data. So I rsync
to the NFS added “some” of the files and directory. which I interrupted as I honestly didn’t want a long resilver time testing the drive replacement… to see if they finally got the method to the documents (man page) which are to just place a drive in the same physical spot / slot and it resilvers without interaction.
I tested the autoreplace by exporting the pool and placing a different drive in SAS0 slot. Then importing the pool results

mike@deepblue-mi6-puffin:~$ sudo zpool status
  pool: backup
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

        NAME                     STATE     READ WRITE CKSUM
        backup                   DEGRADED     0     0     0
          raidz1-0               DEGRADED     0     0     0
            1051945999278060655  UNAVAIL      0     0     0  was /dev/disk/by-vdev/SAS0-part1
            SAS1                 ONLINE       0     0     0
            SAS2                 ONLINE       0     0     0
            SAS3                 ONLINE       0     0     0
          SAS4                   ONLINE       0     0     0

errors: No known data errors

As we can see reality and the Man pages with autoreplace don’t align . Thus it still required a small amount of interact with the replace command

mike@deepblue-mi6-puffin:~$ sudo zpool replace -f backup SAS0
mike@deepblue-mi6-puffin:~$ sudo zpool status
  pool: backup
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Feb  7 21:11:46 2025
        31.1G / 31.1G scanned, 8.68G / 31.1G issued at 808M/s
        1.73G resilvered, 27.91% done, 00:00:28 to go
config:

        NAME                       STATE     READ WRITE CKSUM
        backup                     DEGRADED     0     0     0
          raidz1-0                 DEGRADED     0     0     0
            replacing-0            DEGRADED     0     0     0
              1051945999278060655  UNAVAIL      0     0     0  was /dev/disk/by-vdev/SAS0-part1/old
              SAS0                 ONLINE       0     0     0  (resilvering)
            SAS1                   ONLINE       0     0     0
            SAS2                   ONLINE       0     0     0
            SAS3                   ONLINE       0     0     0
          SAS4                     ONLINE       0     0     0

errors: No known data errors
mike@deepblue-mi6-puffin:~$ sudo zpool status
  pool: backup
 state: ONLINE
  scan: resilvered 6.08G in 00:00:42 with 0 errors on Fri Feb  7 21:12:28 2025
config:

        NAME        STATE     READ WRITE CKSUM
        backup      ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            SAS0    ONLINE       0     0     0
            SAS1    ONLINE       0     0     0
            SAS2    ONLINE       0     0     0
            SAS3    ONLINE       0     0     0
          SAS4      ONLINE       0     0     0

errors: No known data errors

now back to healthy again

Thoughts on Plucky Puffin (development branch) (GNU/Linux 6.12.0-12-generic x86_64)
I really like the install status on the screen vs earlier versions. On this Supermicro X9 board seems to run smooth with little errors or problems this far. which I do plan on being rather abusive to it to check the recoverability.
One interesting note is this board has a onboard SAS controller flashed to a LSI 2308 - 8i IT which uses MPT2 driver.
And I have installed a LSI 9300-4i4e board in addition to the onboard SAS, using a newer driver MPT3
as well as a SATA controller all working well without conflicts with Plucky Puffin

3

Thank you for posting your experiments. I’m looking forward to RAIDZ expansion in 2.3 as well.

But I’m not sure that’s what you’ve done above. According to the documentation it looks like the syntax would be something like:

sudo zpool attach backup raidz1-0 SAS4
2 Likes
4

I had spoke of a penalty with adding a drive to a existing raidz vdev, but what was the actual penalty? Does the added stripe actually help??
so I decided to see
here is the benchmark for the 4 wide raidz1 and adding a drive via the new zfs add command

root@deepblue-mi6-puffin:/backup# fio --name TEST --eta-newline=5s --filename=temp.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.38
Starting 1 process
TEST: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [W(1)][11.7%][w=62.0MiB/s][w=62 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [W(1)][21.7%][w=41.0MiB/s][w=41 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [W(1)][31.7%][w=62.0MiB/s][w=62 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [W(1)][41.7%][w=61.0MiB/s][w=61 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [W(1)][51.7%][w=62.1MiB/s][w=62 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [W(1)][61.7%][w=62.1MiB/s][w=62 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [W(1)][71.7%][w=58.1MiB/s][w=58 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [W(1)][81.7%][w=48.0MiB/s][w=48 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [W(1)][91.7%][w=56.1MiB/s][w=56 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [W(1)][100.0%][w=62.1MiB/s][w=62 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=73625: Fri Feb  7 22:03:01 2025
 write: IOPS=58, BW=58.3MiB/s (61.1MB/s)(3500MiB/60017msec); 0 zone resets
   slat (msec): min=9, max=124, avg=17.12, stdev= 6.13
   clat (usec): min=14, max=948748, avg=525743.99, stdev=89245.60
    lat (msec): min=17, max=966, avg=542.87, stdev=90.72
   clat percentiles (msec):
    |  1.00th=[  271],  5.00th=[  447], 10.00th=[  481], 20.00th=[  493],
    | 30.00th=[  502], 40.00th=[  506], 50.00th=[  506], 60.00th=[  510],
    | 70.00th=[  518], 80.00th=[  527], 90.00th=[  659], 95.00th=[  726],
    | 99.00th=[  827], 99.50th=[  877], 99.90th=[  944], 99.95th=[  944],
    | 99.99th=[  953]
  bw (  KiB/s): min= 8192, max=122880, per=100.00%, avg=60235.29, stdev=11999.43, samples=119
  iops        : min=    8, max=  120, avg=58.82, stdev=11.72, samples=119
 lat (usec)   : 20=0.06%
 lat (msec)   : 20=0.06%, 50=0.09%, 100=0.20%, 250=0.51%, 500=31.06%
 lat (msec)   : 750=65.34%, 1000=2.69%
 cpu          : usr=0.43%, sys=40.66%, ctx=3606, majf=0, minf=10
 IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.5%, 16=0.9%, 32=98.2%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
    issued rwts: total=0,3500,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
 WRITE: bw=58.3MiB/s (61.1MB/s), 58.3MiB/s-58.3MiB/s (61.1MB/s-61.1MB/s), io=3500MiB (3670MB), run=60017-60017msec
-------------------
now a read
-------------------
root@deepblue-mi6-puffin:/backup# fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.38
Starting 1 process
Jobs: 1 (f=1): [R(1)][30.4%][r=480MiB/s][r=480 IOPS][eta 00m:16s]
Jobs: 1 (f=1): [R(1)][56.5%][r=430MiB/s][r=430 IOPS][eta 00m:10s]
Jobs: 1 (f=1): [R(1)][86.4%][r=479MiB/s][r=479 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [R(1)][100.0%][r=371MiB/s][r=371 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=75798: Fri Feb  7 22:04:05 2025
 read: IOPS=453, BW=454MiB/s (476MB/s)(10.0GiB/22575msec)
   slat (usec): min=1084, max=175367, avg=2186.75, stdev=2221.82
   clat (usec): min=12, max=272976, avg=67070.10, stdev=11356.27
    lat (usec): min=1472, max=293874, avg=69256.85, stdev=11792.90
   clat percentiles (msec):
    |  1.00th=[   45],  5.00th=[   59], 10.00th=[   60], 20.00th=[   62],
    | 30.00th=[   63], 40.00th=[   64], 50.00th=[   65], 60.00th=[   67],
    | 70.00th=[   69], 80.00th=[   73], 90.00th=[   79], 95.00th=[   86],
    | 99.00th=[  106], 99.50th=[  132], 99.90th=[  148], 99.95th=[  155],
    | 99.99th=[  155]
  bw (  KiB/s): min=198656, max=544768, per=99.77%, avg=463439.64, stdev=60117.25, samples=45
  iops        : min=  194, max=  532, avg=452.58, stdev=58.71, samples=45
 lat (usec)   : 20=0.05%
 lat (msec)   : 2=0.05%, 4=0.05%, 10=0.15%, 20=0.20%, 50=0.57%
 lat (msec)   : 100=97.59%, 250=1.35%, 500=0.01%
 cpu          : usr=0.80%, sys=17.17%, ctx=10247, majf=0, minf=8202
 IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
    issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  READ: bw=454MiB/s (476MB/s), 454MiB/s-454MiB/s (476MB/s-476MB/s), io=10.0GiB (10.7GB), run=22575-22575msec

Next I destroyed the pool and then reassembled them this time 5 wide with the exact setting except the add command…

root@deepblue-mi6-puffin:/backup# fio --name TEST --eta-newline=5s --filename=temp.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.38
Starting 1 process
TEST: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [W(1)][11.7%][w=64.1MiB/s][w=64 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [W(1)][21.7%][w=63.1MiB/s][w=63 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [W(1)][31.7%][w=63.1MiB/s][w=63 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [W(1)][41.7%][w=20.0MiB/s][w=20 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [W(1)][51.7%][w=53.1MiB/s][w=53 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [W(1)][61.7%][w=61.0MiB/s][w=61 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [W(1)][71.7%][w=63.0MiB/s][w=63 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [W(1)][81.7%][w=65.1MiB/s][w=65 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [W(1)][91.7%][w=63.1MiB/s][w=63 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [W(1)][100.0%][w=63.0MiB/s][w=63 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=10532: Fri Feb  7 22:19:35 2025
  write: IOPS=56, BW=56.1MiB/s (58.9MB/s)(3369MiB/60008msec); 0 zone resets
    slat (msec): min=9, max=634, avg=17.78, stdev=22.23
    clat (usec): min=15, max=2779.3k, avg=546122.72, stdev=251609.58
     lat (msec): min=15, max=2805, avg=563.91, stdev=257.89
    clat percentiles (msec):
     |  1.00th=[  266],  5.00th=[  435], 10.00th=[  464], 20.00th=[  485],
     | 30.00th=[  489], 40.00th=[  493], 50.00th=[  498], 60.00th=[  498],
     | 70.00th=[  506], 80.00th=[  510], 90.00th=[  651], 95.00th=[  693],
     | 99.00th=[ 2123], 99.50th=[ 2198], 99.90th=[ 2769], 99.95th=[ 2769],
     | 99.99th=[ 2769]
   bw (  KiB/s): min= 2048, max=126976, per=100.00%, avg=57980.77, stdev=19150.32, samples=119
   iops        : min=    2, max=  124, avg=56.62, stdev=18.70, samples=119
  lat (usec)   : 20=0.06%
  lat (msec)   : 20=0.03%, 50=0.12%, 100=0.18%, 250=0.56%, 500=61.44%
  lat (msec)   : 750=33.93%, 1000=0.06%, 2000=2.32%, >=2000=1.31%
  cpu          : usr=0.43%, sys=39.05%, ctx=3510, majf=1, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.5%, 16=0.9%, 32=98.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3369,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=56.1MiB/s (58.9MB/s), 56.1MiB/s-56.1MiB/s (58.9MB/s-58.9MB/s), io=3369MiB (3533MB), run=60008-60008msec

--------------------------
now a read
-------------------------
root@deepblue-mi6-puffin:/backup# fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.38
Starting 1 process
Jobs: 1 (f=1): [R(1)][43.8%][r=669MiB/s][r=668 IOPS][eta 00m:09s]
Jobs: 1 (f=1): [R(1)][81.2%][r=676MiB/s][r=676 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [R(1)][93.8%][r=629MiB/s][r=629 IOPS][eta 00m:01s]
TEST: (groupid=0, jobs=1): err= 0: pid=12748: Fri Feb  7 22:20:56 2025
  read: IOPS=647, BW=647MiB/s (679MB/s)(10.0GiB/15818msec)
    slat (usec): min=891, max=157161, avg=1527.68, stdev=1685.54
    clat (usec): min=14, max=239810, avg=46904.92, stdev=5508.57
     lat (usec): min=1222, max=241820, avg=48432.60, stdev=5881.90
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[42206], 10.00th=[43779], 20.00th=[44827],
     | 30.00th=[45351], 40.00th=[46400], 50.00th=[46924], 60.00th=[47449],
     | 70.00th=[47973], 80.00th=[49021], 90.00th=[50070], 95.00th=[51643],
     | 99.00th=[67634], 99.50th=[70779], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[84411]
   bw (  KiB/s): min=327680, max=727040, per=99.65%, avg=660579.10, stdev=65219.16, samples=31
   iops        : min=  320, max=  710, avg=645.10, stdev=63.69, samples=31
  lat (usec)   : 20=0.05%
  lat (msec)   : 2=0.05%, 4=0.04%, 10=0.23%, 20=0.32%, 50=88.01%
  lat (msec)   : 100=11.29%, 250=0.01%
  cpu          : usr=1.11%, sys=22.51%, ctx=10252, majf=0, minf=8202
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=647MiB/s (679MB/s), 647MiB/s-647MiB/s (679MB/s-679MB/s), io=10.0GiB (10.7GB), run=15818-15818msec

from my view point it does make a difference. And if one is just seeking performance it would make a difference Yet it does not undercut the ability for those whom just need to add a drive or two to a existing raidz vdev for increased storage on the fly …

@demyers
Standby I’ll retry (test) that I did attempt it (attach , but may have left off -f) earlier and may have messed up in syntax… although from what I hit on the guidance was add not attach but that was bad advice on a differing site …
bad site
bad site
(and naturally If it does assemble I’ll benchmark it as well)

5

@demyers
Yep I pulled a bone head and actually forgot the correct attach command which is

zpool attach -f backup raidz1-0 SAS4

which appears to be processing … Thank you for pointing that out so here is the correct method utilized. Which I had used the same exact command you pointed out at first, I overlooked the actual error, and forgot that the vdev must be identified prior to the drive to be added

I didn’t capture the status output prior to adding the drive (in a rush I guess to correct my error above)

root@deepblue-mi6-puffin:/home/mike# zpool status
  pool: backup
 state: ONLINE
  scan: scrub repaired 0B in 00:00:02 with 0 errors on Fri Feb  7 22:45:57 2025
expand: expanded raidz1-0 copied 900K in 00:00:25, on Fri Feb  7 22:45:55 2025
config:

        NAME        STATE     READ WRITE CKSUM
        backup      ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            SAS0    ONLINE       0     0     0
            SAS1    ONLINE       0     0     0
            SAS2    ONLINE       0     0     0
            SAS3    ONLINE       0     0     0
            SAS4    ONLINE       0     0     0

errors: No known data errors

yep no offset
now list

root@deepblue-mi6-puffin:/home/mike# zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
backup  18.2T   479K  18.2T        -         -     0%     0%  1.00x    ONLINE  -

now the correct benchmark

root@deepblue-mi6-puffin:/backup# fio --name TEST --eta-newline=5s --filename=temp.file --rw=write --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.38
Starting 1 process
TEST: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [W(1)][11.7%][w=61.1MiB/s][w=61 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [W(1)][21.7%][w=22.0MiB/s][w=22 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [W(1)][31.7%][w=63.1MiB/s][w=63 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [W(1)][41.7%][w=63.1MiB/s][w=63 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [W(1)][51.7%][w=63.1MiB/s][w=63 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [W(1)][61.7%][w=45.0MiB/s][w=45 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [W(1)][71.7%][w=53.1MiB/s][w=53 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [W(1)][81.7%][w=61.1MiB/s][w=61 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [W(1)][91.7%][w=62.1MiB/s][w=62 IOPS][eta 00m:05s]
Jobs: 1 (f=1): [W(1)][100.0%][w=62.0MiB/s][w=62 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=12339: Fri Feb  7 22:52:50 2025
  write: IOPS=53, BW=53.7MiB/s (56.3MB/s)(3222MiB/60005msec); 0 zone resets
    slat (msec): min=9, max=617, avg=18.60, stdev=23.49
    clat (usec): min=7, max=2727.8k, avg=570892.86, stdev=283520.85
     lat (msec): min=12, max=2745, avg=589.49, stdev=290.39
    clat percentiles (msec):
     |  1.00th=[  249],  5.00th=[  460], 10.00th=[  477], 20.00th=[  489],
     | 30.00th=[  493], 40.00th=[  498], 50.00th=[  498], 60.00th=[  502],
     | 70.00th=[  506], 80.00th=[  518], 90.00th=[  751], 95.00th=[  844],
     | 99.00th=[ 2140], 99.50th=[ 2567], 99.90th=[ 2735], 99.95th=[ 2735],
     | 99.99th=[ 2735]
   bw (  KiB/s): min= 2048, max=112640, per=99.81%, avg=54879.46, stdev=18736.89, samples=118
   iops        : min=    2, max=  110, avg=53.59, stdev=18.30, samples=118
  lat (usec)   : 10=0.03%, 20=0.03%
  lat (msec)   : 20=0.06%, 50=0.09%, 100=0.22%, 250=0.59%, 500=56.11%
  lat (msec)   : 750=32.81%, 1000=5.80%, 2000=2.73%, >=2000=1.52%
  cpu          : usr=0.44%, sys=37.85%, ctx=3375, majf=1, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.5%, 16=1.0%, 32=98.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3222,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=53.7MiB/s (56.3MB/s), 53.7MiB/s-53.7MiB/s (56.3MB/s-56.3MB/s), io=3222MiB (3379MB), run=60005-60005msec

------------------------------
now a read
-------------------------------
root@deepblue-mi6-puffin:/backup# fio --name TEST --eta-newline=5s --filename=temp.file --rw=read --size=2g --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
TEST: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.38
Starting 1 process
Jobs: 1 (f=1): [R(1)][41.2%][r=633MiB/s][r=633 IOPS][eta 00m:10s]
Jobs: 1 (f=1): [R(1)][81.2%][r=599MiB/s][r=598 IOPS][eta 00m:03s]
Jobs: 1 (f=1): [R(1)][100.0%][r=599MiB/s][r=598 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=14332: Fri Feb  7 22:55:42 2025
  read: IOPS=614, BW=615MiB/s (645MB/s)(10.0GiB/16654msec)
    slat (usec): min=914, max=160287, avg=1607.56, stdev=1698.16
    clat (usec): min=12, max=82064, avg=49401.95, stdev=5511.82
     lat (usec): min=1341, max=237280, avg=51009.51, stdev=5868.19
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[45351], 10.00th=[46400], 20.00th=[47449],
     | 30.00th=[47973], 40.00th=[48497], 50.00th=[49021], 60.00th=[50070],
     | 70.00th=[50594], 80.00th=[51119], 90.00th=[52691], 95.00th=[54264],
     | 99.00th=[72877], 99.50th=[74974], 99.90th=[80217], 99.95th=[81265],
     | 99.99th=[82314]
   bw (  KiB/s): min=303104, max=663552, per=99.70%, avg=627766.55, stdev=62204.34, samples=33
   iops        : min=  296, max=  648, avg=613.03, stdev=60.74, samples=33
  lat (usec)   : 20=0.05%
  lat (msec)   : 2=0.04%, 4=0.06%, 10=0.21%, 20=0.34%, 50=59.08%
  lat (msec)   : 100=40.22%
  cpu          : usr=1.17%, sys=25.78%, ctx=10263, majf=0, minf=8203
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=615MiB/s (645MB/s), 615MiB/s-615MiB/s (645MB/s-645MB/s), io=10.0GiB (10.7GB), run=16654-16654msec

again thanks for pointing that out I totally overlooked it

6

Yeppers to the above. :slight_smile:
@sgt-mike this has been very thought provoking for me.
I try to keep one ZFS system on the newest ZFS version to see any changes that might be.

Plus it keeps me on a contant learning curve, I have the desire to master my ZFS systems…LOL

7

So far I have observed no ill effect of the unsupported version, nor any bad manners. Unless I make it fault or crash. which yes I have done the most bone-headed things. Like pull a drive in the middle of a scrub… etc . And it has been able to recover to a healthy state. So far impressed… which 2.2 has no flies on it the big improvement to me is the attach command fixed to support Raidz pools.

Now about Plucky Puffin which we all know is development stage, is running better that I expected. I only had one issue in the installation, which was a bone-headed move on my part. As this board has three NIC’s 1 dedicated to the IPMI, 2 for network, I decided to bond the two nic’s … normal right?

Yeah not when one has not really done their homework on bonding, that venture did not go well. I screwed up so badly that I had to actually clear the cmos / bios in order to address the 2 nic’s. When I finally did that I reinstalled the base OS, with the thought I’ll read up on that later. Definitely not the OS’s fault there, rather the loose nut on the keyboard .

I will definitely agree there and feel the same way. Which is why I guess I do these little ventures.

So what is cooking in the lab today not much threw the backup server into it’s role last night and sending files in a rsnyc command to observe throughput and resource on the ram and processor.
At 64gb the ram is hitting 90 - 95% loads, the processor is spiking from 20% to 50% load. Not bad stats. (yes at some point soon I will do my normal Max the Ram out - 512gb)
So much to learn and so little time to do it in.

Now I am having a issue but it is not related to ZFS / Plucky Puffin, rather IPMI itself with the redirect console. That is not a here question merely a irritant, which I’ll stumble on the fix at some point. But everything else in IPMI is working. Once I figured out the login / password.
2025-02-10T06:00:00Z
I did manage to fix it is was really simple…

8

Not bad at all, and agreed so little time to learn all that ZFS is and can be.
It humbles me to feel like a 6th grader on ZFS…I just push forward and pick up something new dailey.

1 Like