Is there an easy way to track total bytes written to certain files?

Support and Help
,
1

Is there an easy and simple way to track which files in a given folder (and subfolders) are being written to over a given period, and more importantly how many bytes have been written to those files? This includes disk writes that overwrite parts of existing files without modifying the file size, so I cannot simply check the file size before and after to get the result. Perhaps something like iotop that lists bytes read/written per file for a given folder, and possibly what PID’s did the IO?

ChatGPT mentioned tools like inotifywait, strace and bpftrace, but those will output partial and highly fragmented information.

2

I think we are lacking some basic information here that would help the volunteer users offer suggestions.

  1. which Ubuntu version?
  2. whether you need to log this data to a database/file or just watch it live?
  3. If you need a script to aggregate the total bytes automatically?
  4. do you want to watch files/folder under your username or system folders?
1 Like
3

Hello.

  1. Ubuntu 26.04 LTS.

  2. I just need temporarily to identify what’s going on in a certain folder (and sub-folders). No permanent logging needed.

  3. A script would work too.

  4. For this purpose, some folders in my home directory. Trying to identify which processes are writing to ~/.cache/ folder and what files specifically, so I can attempt to reduce my SSD wear.

4

Sounds like a job for inotify-tools, but, as (not very) far as I can tell, they only enable you to monitor events, such as opening, modifying files etc.

watch_me="${TMPDIR:-/tmp}/watched_dir"
mkdir -p "$watch_me"

# recursively monitor dir in a background job (note the trailing '&')
fsnotifywait -mr "$watch_me" >> ~/fswatch.log &

# give it some time to get up and running
sleep 1

# (over)write a file 3 times with 1 MiB of random garbage
for _ in {1..3}; do head -c 1M /dev/urandom >| "$watch_me/overwrite_me"; done

# kill the (last) background job
kill %%

(this is a runnable shell “scriplet”, so lines starting with # are comments)

The log will be in ~/fswatch.log and look something like this:

[fid=8749d727.4b327352.862c956be811;name='.'] 
[fid=8749d727.4b327352.862c956be811;name='overwrite_me'] /home/peter/tmp/watched_dir/overwrite_me
/home/peter/tmp/watched_dir/ OPEN overwrite_me
/home/peter/tmp/watched_dir/ MODIFY overwrite_me
/home/peter/tmp/watched_dir/ MODIFY overwrite_me
/home/peter/tmp/watched_dir/ MODIFY overwrite_me
[several repetitions of previous event squashed]
/home/peter/tmp/watched_dir/ CLOSE_WRITE,CLOSE overwrite_me
/home/peter/tmp/watched_dir/ OPEN overwrite_me
/home/peter/tmp/watched_dir/ MODIFY overwrite_me
[several repetitions of previous event squashed]
/home/peter/tmp/watched_dir/ CLOSE_WRITE,CLOSE overwrite_me
/home/peter/tmp/watched_dir/ OPEN overwrite_me
/home/peter/tmp/watched_dir/ MODIFY overwrite_me
[several repetitions of previous event squashed]
/home/peter/tmp/watched_dir/ CLOSE_WRITE,CLOSE overwrite_me

As you can see, there is no info on the amount of data written, and I don’t know an easy way to get it. The only thing I can come up with off-hand involves exploiting the copy-on-write (CoW) design of filesystems like ZFS and BTRFS; since every MODIFY event actually writes a new copy of the filesystem block (extent), one should be able to to keep track of the old and new ones. So my idea would involve a combo of fsnotifywait and some other tool to do the before and after. IIRC, BTRFS keeps some kind of shadow snapshot for the most recent changes to the filesystem, so that should enable an exact count of bytes written by comparing the extent references of the current state and the last one before the modification.

But, since this approach would require assistance from CoW filesystems anyway, there might be simpler ways to do it with the respective native tool set. The most trivial way of doing it would just be to take a snapshot on every OPEN (for writing) event. But I don’t know if reacting to those (outside the loop) would be sufficient; looks like a recipe for race conditions.

OTOH, if you can adapt your workflow to take a snapshot before doing any filesystem changes, the data you want will just be part of the metadata of said CoW filesystems, for free.

Beware though that CoW filesystems don’t come for free. If this is about monitoring writes to estimate the lifetime expectancy of an SSD, you are never going to see the whole picture, because of write amplification effects, and using CoW could actually hurt because they are prone to write amplification in some circumstances; talk about Beelzebub doing an exorcism. :wink:

5

Nah, I’m on ext4, not btrfs. ChatGPT mentioned the use of strace and inotifywait like so:

strace -f -e trace=openat,write,pwrite64,fsync,fdatasync,rename -p <PID>

inotifywait -m -r -e modify -e close_write /home/user/.cache/

These will output file writes per process and per directory, but as I said, the information they output is very fragmented. Most of the output from the first command will show IO writes to file descriptors rather than paths, so they need to be somehow matched to the file opening API calls, which translate paths to file descriptors. The second command will only show what files are being written to, but not the pid or buffer size.

6

Sorry, that part somehow totally failed to register with me. :innocent: While there shouldn’t be too much to worry about with regards to SSD wear in the designated cache hierarchy[1], you can totally take it out of the picture by mounting it as a tmpfs with the noswap option[2]; put this in your /etc/fstab:

volatile_cache /home/<user name>/.cache tmpfs noswap,uid=<user name>,gid=<user name>,mode=0750

(replace <user name> with your user name; can set (almost) anything you like instead of volatile_cache)

Then run systemctl daemon-reload to (re)generate the systemd mount units and follow up with sudo mount volatile_cache. Any new cache data will now be written to RAM and never touch the SSD. The downside is that after a reboot the cache will always be cold, but, if your usage pattern is like mine, i.e. long uptimes due to suspend/resume between (absolutely necessary) reboots (for things like kernel upgrades), that shouldn’t be too much of a problem. The upside, besides :high_voltage: speed, is that there will never be stale cached data taking up space long after it has outlived its usefulness[3].

To get an estimate of the RAM required for that kind of cache on tmpfs, run du -sh ~/.cache. But I guess that you needn’t worry too much about the space requirements, unless you have tools like ccache and do a lot of compiling.

While I don’t have such volatile cache directory — mainly because I do use ccache and couldn’t yet be bothered to handle it specially —, I do employ a version of it, specifically for Firefox’s cache; I have set this in about:config:

browser.cache.disk.parent_directory = /home/peter/tmp/firefox

/tmp should be a tmpfs by default nowadays. If you don’t want to risk an no-space-left situation on /tmp, one can just mount a user’s $TMPDIR as tmpfs, which is what I am actually running:

  • mount tmpfs on user’s $TMPDIR by an entry in /etc/fstab:
    peter_tmp /home/peter/tmp tmpfs noswap,uid=peter,gid=peter,mode=0750
    (don’t forget to create the directory as the user: mkdir ~/tmp, otherwise the mount will fail)
  • put export TMPDIR="$HOME"/tmp && mkdir -p "$TMPDIR" in ~/.profile, or, better yet, put it in an environment.d drop-in, to make it available to the whole user login session, not just shells:
    echo 'TMPDIR="$HOME"/tmp' > ~/.config/environment.d/99-tmpdir.conf

Since Firefox seems to be the heaviest user of the cache, at least in my case (ignoring ccache), that should save the most write churn on the SSD. But I am not too worried about the latter, TBH, because modern SSD’s should be able to handle lots of drive writes; it’s more about speed for me — nothing beats RAM at access latency. :smiling_face_with_sunglasses:
(takes me about a week to fill Firefox’s cache to the default max of 1 GiB; current uptime: 14 days; plus, Firefox’s cache is smart and monitors available space, so the maximum is adjusted accordingly — default size cap of tmpfs is 50% of RAM total)

The above setup also allows to set XDG_CACHE_HOME="$TMPDIR"/cache; one can do so selectively by only setting it in the environment of specific applications.

  1. intended use is WORM (write once, read many) anyway ↩︎

  2. otherwise some contents might end up being swapped to the SSD-backed swap space ↩︎

  3. there seem to be no provisions to purge cold cache data ↩︎

7

You can use AIDE. It’ll show the files are editted but not what edits happened. I believe it shows the file size difference. Is this what you need?