Thanks @stgraber - that worked like a charm
For posterity, it was as simple as:
# remove my user from the lxd group
sudo gpasswd -d $USER lxd
# reboot to cleanly logout and back in again without being in
# the lxd group anymore
# set the lxd user group to just my user's primary group since
# there is no other users on my local machine
sudo snap set lxd daemon.user.group=$USER
# lxc will have created a new project called user-$UID for your user
lxc project list
# set this as the default project
lxc project switch user-$UID
# then migrate instances to this user's project by stopping them all
# and then moving them across
sudo lxc stop --all
# moving may take a while depending on how many containers/vms you have
for instance in $(sudo lxc list --format json | jq ".[].name" -r); do
sudo lxc move $instance --target-project user-$UID
done
# then I can use those instances from my local user as before