0

I am facing a similar issue with free space accounting as described in zpool list vs zfs list - why free space is 10x different?. My situation is however different, even after accounting for the reserved (slop) space, the numbers do not seem to add up.

Zpool list reports 8.1TB free but zfs really only has 339GB free. Slop space was already lowered to 2^-7, since the pool is quite big.

The free space is sorely missed, any help will be highly appreciated.

Can the free space be recovered somehow?

#zpool list z
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
z     65.4T  57.3T  8.10T        -         -    27%    87%  1.21x    ONLINE  -
#zfs list z
NAME   USED  AVAIL     REFER  MOUNTPOINT
z     69.4T   339G       96K  none
#zfs get all z
NAME  PROPERTY              VALUE                  SOURCE
z     type                  filesystem             -
z     creation              Mon Jun  8  1:29 2020  -
z     used                  69.4T                  -
z     available             338G                   -
z     referenced            96K                    -
z     compressratio         1.22x                  -
z     mounted               no                     -
z     quota                 none                   default
z     reservation           none                   default
z     recordsize            128K                   local
z     mountpoint            none                   local
z     sharenfs              off                    default
z     checksum              on                     default
z     compression           lz4                    local
z     atime                 off                    local
z     devices               on                     default
z     exec                  on                     default
z     setuid                on                     default
z     readonly              off                    default
z     zoned                 off                    default
z     snapdir               hidden                 default
z     aclinherit            restricted             default
z     createtxg             1                      -
z     canmount              on                     default
z     xattr                 sa                     local
z     copies                1                      default
z     version               5                      -
z     utf8only              off                    -
z     normalization         none                   -
z     casesensitivity       sensitive              -
z     vscan                 off                    default
z     nbmand                off                    default
z     sharesmb              off                    default
z     refquota              none                   default
z     refreservation        none                   default
z     guid                  4880913883724291733    -
z     primarycache          all                    default
z     secondarycache        all                    default
z     usedbysnapshots       0B                     -
z     usedbydataset         96K                    -
z     usedbychildren        69.4T                  -
z     usedbyrefreservation  0B                     -
z     logbias               latency                default
z     objsetid              54                     -
z     dedup                 on                     local
z     mlslabel              none                   default
z     sync                  disabled               local
z     dnodesize             auto                   local
z     refcompressratio      1.00x                  -
z     written               96K                    -
z     logicalused           83.6T                  -
z     logicalreferenced     42K                    -
z     volmode               default                default
z     filesystem_limit      none                   default
z     snapshot_limit        none                   default
z     filesystem_count      none                   default
z     snapshot_count        none                   default
z     snapdev               hidden                 default
z     acltype               off                    default
z     context               none                   default
z     fscontext             none                   default
z     defcontext            none                   default
z     rootcontext           none                   default
z     relatime              off                    default
z     redundant_metadata    all                    default
z     overlay               off                    default
z     encryption            off                    default
z     keylocation           none                   default
z     keyformat             none                   default
z     pbkdf2iters           0                      default
z     special_small_blocks  64K                    local
#cat /sys/module/zfs/parameters/spa_slop_shift
7
#zpool iostat -Lv
                 capacity     operations     bandwidth
pool           alloc   free   read  write   read  write
-------------  -----  -----  -----  -----  -----  -----
z              57.3T  8.10T    317  4.47K  47.0M  29.1M
  mirror       13.7T   862G     18     10  11.1M  3.04M
    sda            -      -      9      5  5.53M  1.52M
    sdg            -      -      9      5  5.53M  1.52M
  mirror       13.7T   899G     18     10  11.0M  3.03M
    sdb            -      -      9      5  5.51M  1.52M
    sdh            -      -      9      5  5.51M  1.52M
  mirror       13.7T   838G     18     10  11.1M  3.04M
    sdc            -      -      9      5  5.54M  1.52M
    sdi            -      -      9      5  5.54M  1.52M
  mirror       12.5T  2.04T     17     10  9.87M  3.61M
    sdd            -      -      8      5  4.93M  1.81M
    sdj            -      -      8      5  4.93M  1.81M
dedup              -      -      -      -      -      -
  mirror        238G  16.6G    170  4.03K   822K  9.72M
    nvme0n1p3      -      -     85  2.01K   411K  4.86M
    nvme1n1p3      -      -     85  2.01K   411K  4.86M
special            -      -      -      -      -      -
  mirror       1.76T  1.72T     36    195  1.62M  3.15M
    sdf            -      -     18     97   831K  1.57M
    sdl            -      -     18     97   831K  1.57M
  mirror       1.70T  1.78T     37    213  1.58M  3.55M
    sde            -      -     18    106   807K  1.77M
    sdk            -      -     18    106   807K  1.77M
logs               -      -      -      -      -      -
  mirror           0  15.5G      0      0      2      0
    nvme0n1p2      -      -      0      0      1      0
    nvme1n1p2      -      -      0      0      1      0
cache              -      -      -      -      -      -
  nvme0n1p4     569G  37.2G    178     40  2.96M  3.81M
  nvme1n1p4     568G  37.8G    178     46  2.96M  3.82M
-------------  -----  -----  -----  -----  -----  -----
#
justsomeguy
  • 368
trashbarg
  • 9

1 Answers1

0

Good News, you have not lost any space. At least in 15 years of using ZFS, I've never seen lost free space.

There are a couple of things going on here, but mostly it is that the two free space values are not the same.

The zpool free space is all un-allocated blocks. Where the zfs free space value removes the slop area and more.

You are also exceeding the 80% recommendation. This should only impact performance, but it can also cost additional space usage. In short, if the disk is so fragmented that it can not allocate a whole chunk to a file it uses additional space to manage smaller chunks.

James Risner
  • 1,282
  • Thanks for your reply.

    The slop space is only roughly 0.5TB, the /sys/module/zfs/parameters/spa_slop_shift was set to 7.

    How does that work then with the un-allocated blocks? With 8.1TB of free space in un-allocated blocks, how come it only show as 339GB remaining free space? And let's set the 3.5TB of free space on the special and dedup vdevs aside and only drill down how the other 4.5TB of un-allocated block do not show as free space in zfs list.

     – trashbarg Dec 24 '22 at 23:12
  • Yes, I noticed that. I don't have a precise answer for where it is all gone. But it hasn't vanished or otherwise missing. it's being used, allocated, reserved for things ZFS needs to keep available. I have a zpool list of SIZE 52.6T, ALLOC 39.6T and FREE 13.1T. zfs list shows 12.8T free. So my theory is a lot of replacement/indirection tables using up a tremendous amount of space. But without being on the system, I couldn't know. – James Risner Dec 24 '22 at 23:17
  • There have never been any vdev removals on this pool, I do not see a need for the indirection table entries. Anyway, even if, blocks occupied by indirection table would not be reported as free by zpool list, would they? In the case of your pool did you increase spa_slop_shift to 7 as well? It looks like it was tuned. I can post more data about the pool as required if you are willing to have a look. Let me know what to share. – trashbarg Dec 24 '22 at 23:33
  • The default allocation for new data is 128k. If the pool has insufficient number of 128K blocks to write a new bit of data, it allocates a block to hold the points to the various smaller blocks. Then allocates data blocks to reach 128K. Then allocates the 128k it needed to allocate. That's what I'm talking about. Notice the zpool list free value is any blocks not used by data, so it's likely this is another way you "lose" free space to get to the zfs list. – James Risner Dec 24 '22 at 23:40
  • You pool is way too full. The only fix for your pool now is to back up and restore to remove those blocks of indirect points and replace with full 128k blocks. But for that to work you'd need another 12T in disk space added to the pool. – James Risner Dec 24 '22 at 23:41
  • I believe what you say means that if I were to create the same pool and do zfs send | zfs recv of all the data including snapshots to the new one, the new pool would have more free space than the old one.

    All the vdevs are two-way mirrors. I can detach one device form each of them, construct a new poll from the detached devices and then compare. That's a bit too much effort and will take days to transfer the data.

    What is the scenario to reproduce this issue on a much smaller pool so that I can test it? I don't think I quite follow your thoughts, but can spend time troubleshooting this.

     – trashbarg Dec 25 '22 at 20:43
  • I believe you could create a 1g pools, fill it to 98% (or similar to now), then remove and recreate files repetitively to simulate operating as you have been. This will be tricky if not impossible to get a test pool in the current state. – James Risner Dec 25 '22 at 21:06