root@media:/home/haaken# smartctl -a /dev/sdd
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.11.0-14-generic] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS726060AL4210
Revision: AAG0
Compliance: SPC-4
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Logical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca271b5d120
Serial number: K8K6ZT5N
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Sun Feb 2 17:32:14 2025 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 26 C
Drive Trip Temperature: 85 C
Accumulated power on time, hours:minutes 24651:37
Manufactured in week 14 of year 2019
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 24
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 6807
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 23449946933952512
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 284 0 284 1300710 374552.907 0
write: 0 11 0 11 1084697 40827.516 0
verify: 0 0 0 0 329704 0.000 0
Non-medium error count: 0
Pending defect count:0 Pending Defects
No Self-tests have been logged
root@media:/home/haaken# smartctl -a /dev/sdf
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.11.0-14-generic] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS726060AL4210
Revision: AAG0
Compliance: SPC-4
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Logical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca271b73f40
Serial number: K8K7T51N
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Sun Feb 2 17:32:39 2025 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 26 C
Drive Trip Temperature: 85 C
Accumulated power on time, hours:minutes 24437:44
Manufactured in week 14 of year 2019
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 25
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 1228
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 1544130333245440
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 76016 25529.906 0
write: 0 0 0 0 28745 12518.477 0
verify: 0 0 0 0 33261 0.000 0
Non-medium error count: 0
Pending defect count:0 Pending Defects
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 491 - [- - -]
# 2 Background short Completed - 0 - [- - -]
Long (extended) Self-test duration: 58876 seconds [16.4 hours]
root@media:/home/haaken# smartctl -a /dev/sdh
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.11.0-14-generic] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS726060AL4210
Revision: AAG0
Compliance: SPC-4
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Logical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca271b715b8
Serial number: K8K7PD9N
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Sun Feb 2 17:33:09 2025 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 26 C
Drive Trip Temperature: 85 C
Accumulated power on time, hours:minutes 24634:32
Manufactured in week 14 of year 2019
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 22
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 6777
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 23367657541599232
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 611 0 611 1417014 370680.054 0
write: 0 8 0 8 988158 40847.248 0
verify: 0 0 0 0 2142735 0.000 0
Non-medium error count: 0
Pending defect count:0 Pending Defects
No Self-tests have been logged
root@media:/home/haaken# smartctl -a /dev/sdf
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.11.0-14-generic] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS726060AL4210
Revision: AAG0
Compliance: SPC-4
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Logical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca271b73f40
Serial number: K8K7T51N
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Sun Feb 2 17:33:26 2025 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 26 C
Drive Trip Temperature: 85 C
Accumulated power on time, hours:minutes 24437:44
Manufactured in week 14 of year 2019
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 25
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 1228
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 1544130333245440
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 76016 25529.906 0
write: 0 0 0 0 28745 12518.477 0
verify: 0 0 0 0 33261 0.000 0
Non-medium error count: 0
Pending defect count:0 Pending Defects
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 491 - [- - -]
# 2 Background short Completed - 0 - [- - -]
Long (extended) Self-test duration: 58876 seconds [16.4 hours]
root@media:/home/haaken#l
If it were me I would first try to remove the Hotspares and show us “zpool status” again.
How or what method did you use to add the spare drives? In theory, To replace a failed disk with a hot spare, you do not need to zpool replace at all (and in fact this might cause you all sorts of grief later; I’ve never done this). Instead you are supposed to simply zpool detach the failed disk and the hot spare automatically replaces it.
But I’m still reading all in this thread.
I’ve tried to reproduce your error eg:
zpool status
pool: dozer
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:
NAME STATE READ WRITE CKSUM
dozer ONLINE 0 0 0
/dev/disk/by-id/usb-SanDisk_Ultra_4C530001050423123075-0:0-part1 ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
/dev/sdh1 ONLINE 0 0 0
errors: No known data errors
pool: zpcachyos
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zpcachyos ONLINE 0 0 0
/dev/disk/by-partuuid/520c8685-5eda-4b86-8360-a54c170338bf ONLINE 0 0 0
errors: No known data errors
As you can see “tank” is not in the best state currently, but this should sort it out.
sudo zpool upgrade tank
This system supports ZFS pool feature flags.
Enabled the following features on 'tank':
redaction_list_spill
raidz_expansion
fast_dedup
longname
large_microzap
┌───────────────────>
│~
└─> zpool status
pool: dozer
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:
NAME STATE READ WRITE CKSUM
dozer ONLINE 0 0 0
/dev/disk/by-id/usb-SanDisk_Ultra_4C530001050423123075-0:0-part1 ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
/dev/sdh1 ONLINE 0 0 0
errors: No known data errors
pool: zpcachyos
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zpcachyos ONLINE 0 0 0
/dev/disk/by-partuuid/520c8685-5eda-4b86-8360-a54c170338bf ONLINE 0 0 0
errors: No known data errors
I have to agree with @sgt-mike those spare drives don’t look good ATM.
I first wanted to suggest to do that @1fallen
which is why I wanted to see the History I was looking for how the exact manner the spares was attached.
Then had the thought to do the clear and scrub which may work and may not work — which depending on the amount of data could take a while to do for some of my pools we are talking 4 + hours. which is the down side and the drive could still be in fault
I would like @ 1 fallen says remove the spares from the pool even if the everything cleans up.
Then use them as Cold Spares.
the faster solution would be to remove the spares I think
Looking over the OP’s first post
These look healthy
No.
scan: scrub in progress since Sun Feb 2 17:54:23 2025
8.60T / 17.5T scanned at 9.87G/s, 470G / 17.5T issued at 539M/s
0B repaired, 2.63% done, 09:11:56 to go.
Wait until completely done don’t do anything don’t write to the drive nothing
run status again once completed check it…
If good to go you
could remove the spare
or
leave it your choice
Of course zeepool for you will be Mypool
The above mentioned c2t1d0 is using a drive alais (which I personally am fond of)
so your should be /dev/by-uuid/12447916164639100982
IF I’m reading everything correct
at which point if removing I’d do another status check , after issuing the detach (make sure it’s not part of the pool) since we have detached the spare from the pool you could shutdown and remove the drive or simply unplug it makes no differance as it is not part of the pool anymore