This may be due to the Hardening settings in the systemd service file. However, tracking down which of the settings are responsible can be very time-consuming, unless you are experienced with them (which I’m not).
You seem to have unusual ownership of the /home folder with it being in group video (normally this is root:root). Given the error message of “Unable to create directory for new file: Permission denied”, it may be linked to this.
One obvious step to try is to change the ownership of /home to root:root. If you cannot do this or it does not work, as a first step at changing the hardening settings, I would suggest the following:
sudo systemctl edit transmission-daemon.service
This will create an override file in /etc/systemd/system/transmission-daemon.service.d/override.conf.
Add the following:
[Service]
CapabilityBoundingSet=CAP_CHOWN
This will give transmission-daemon the permission to change file UIDs and GIDs.
Then restart transmission:
sudo systemctl daemon-reload
sudo systemctl restart transmission-daemon.service
If this does not work, you would have try more changes.
To get back to the original settings, just delete the file in /etc.
You can research further with your favorite search engine / AI, but a couple of links to the official docs to get you started:
https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html
https://man7.org/linux/man-pages/man7/capabilities.7.html