Good day everyone,
I am trying to make an automated rclone backup of my server using Murena cloud. I however cannot get the script to work. I do know that rclone does work since the Devices folder from murena cloud is nicely backed up to my raid1 drive. I however get errors when I try to mirror my local raid 1 murena folder back to murena cloud. Please see the description below.
My situation is as follows:
-
(THIS WORKS) The base of my data is on a 4 TB raid 1 drive. Here I have 2 folders, one being my base of murena drive and the other my immich photo library.
1.1) On my phone Iāve set up the immich app to upload the photos to my immich instance on my raid 1 drive and Iāve set up the nextcloud app to upload the photos to my Photos/Camera folder in the murena folder on my raid 1 drive.
1.2) My murena drive mostly contains my documents, music and photos.
1.3) My immich folder also contains all my photos from the murena drive but this time with the manipulation of immich with facial recognition. Iād like to have a duplicate photo library since i dont want my photos to be able to get corrupted from immich since itās still in beta. -
(THIS DOES NOT WORK) Every night I want to sync these folders to a second raid 0 drive and to murena cloud. Iāve set up rclone using this guide.
2.1) First I want to sync to my raid 0 drive. I prefer this before syncing to murena since this builds in a safeguard that if murena causes any corruption i still have a valid backup. I have written the following script to backup my murena folder, the script also backs up any changes for 7 days. Please feel free to also point out any problems in my code if you like, Iām an mechanical engineer by profession and not a software engineer. Iām planning to trigger this script every morning at 4 am using cron.
~/Headless$ cat backup_murena_external.sh
#!/bin/bash
# Source and Destination folders
SOURCE="/home/USERNAME/externalRAID/Murena"
DESTINATION="/home/USERNAME/external2/Backup/Murena"
BACKUP_OLD="/home/USERNAME/external2/Backup/Murena_recently_deleted"
# Number of daily backups to keep
DAYS_TO_KEEP=7
# Get the current date in YYYY-MM-DD format
DATE=$(date +%Y-%m-%d)
# Function to create the daily backup directory if it doesn't exist
create_backup_dir() {
if [ ! -d "$BACKUP_OLD/$1" ]; then
mkdir -p "$BACKUP_OLD/$1"
echo "Created backup directory: $BACKUP_OLD/$1"
fi
}
# Check if the destination folder exists
if [ ! -d "$DESTINATION" ]; then
echo "Error: Destination folder '$DESTINATION' does not exist."
exit 1
fi
# Check if the backup old folder exists
if [ ! -d "$BACKUP_OLD" ]; then
mkdir -p "$BACKUP_OLD"
echo "Created backup directory for old files: $BACKUP_OLD"
fi
# Iterate through files in the source folder
find "$SOURCE" -type f -print0 | while IFS= read -r -d $'\0' source_file; do
# Get the relative path of the source file from the source folder
relative_path=$(echo "$source_file" | sed "s|^$SOURCE/||")
# Construct the full path of the corresponding file in the destination folder
destination_file="$DESTINATION/$relative_path"
# Check if the destination file exists
if [ -e "$destination_file" ]; then
# Create the daily backup directory if it doesn't exist
create_backup_dir "$DATE"
# Construct the backup path for the old file
backup_file="$BACKUP_OLD/$DATE/$relative_path"
backup_dir=$(dirname "$backup_file")
# Create the necessary subdirectories in the backup folder
mkdir -p "$backup_dir"
# Move the existing file in the destination to the backup folder
mv "$destination_file" "$backup_file"
echo "Moved existing file to backup: $destination_file -> $backup_file"
fi
done
# Perform the rsync operation
rsync -av --delete "$SOURCE/" "$DESTINATION/"
echo "Rsync backup completed."
# --- Optional: Cleanup old daily backups ---
find "$BACKUP_OLD" -maxdepth 1 -type d -ctime +"$DAYS_TO_KEEP" -exec rm -rf {} \;
echo "Cleaned up backups older than $DAYS_TO_KEEP days (if any)."
exit 0
2.2) I also want to perform a backup to murena cloud after the above script is completed. The idea is that i first mirror the Devices folder from murena cloud to my raid 1 drive and then mirror the murena folder on my raid 1 drive back to murena cloud, excluding the Devices folder.
I however cannot get the script to work. I do know that rclone does work since the Devices folder from murena cloud is nicely backed up to my raid1 drive. I however get errors when I try to mirror my local raid 1 murena folder back to murena cloud.
This is my current script:
~/Headless$ cat backup_murena_cloud.sh
#!/bin/bash
echo "START CLOUD BACKUP MURENA"
echo "$(date +'%d-%m-%Y %H:%M')"
echo "Running as user: $(whoami)"
echo "Primary group: $(id -gn)"
echo "All groups: $(id -Gn)"
# ====== CONFIGURATION ======
# Toggle for dry run mode (true or false)
DRY_RUN=true
# Rclone remote name
remote_name="murena"
# Local base Murena folder
local_base="/externalRAID/Murena"
local_devices="$local_base/Devices"
# Log file
logfile=/home/USERNAME/Desktop/sync_murena.log
# ====== FUNCTIONAL LOGIC ======
# Set dry-run flag
dry_flag=""
if [ "$DRY_RUN" = true ]; then
dry_flag="--dry-run"
echo "Dry-run mode enabled." >> "$logfile"
fi
echo "========== SYNC STARTED: $(date) ==========" >> "$logfile"
# Step 1: Sync Devices from remote to local
echo "Step 1: Syncing Devices from remote to local..." >> "$logfile"
rclone sync "$remote_name:Devices" "$local_devices" \
$dry_flag --delete-excluded --progress -vv >> "$logfile" 2>&1
# Step 2: Sync all other folders from local to remote, excluding Devices
echo "Step 2: Syncing all other folders (excluding Devices) to remote..." >> "$logfile"
rclone sync "$local_base" "$remote_name:" \
--exclude "Devices/**" \
$dry_flag --progress -vv >> "$logfile" 2>&1
echo "========== SYNC COMPLETED: $(date) ==========" >> "$logfile"
My log has the following output:
~/Headless$ cat ~/Desktop/sync*
Dry-run mode enabled.
========== SYNC STARTED: wo 16 apr 2025 0:16:57 CEST ==========
Step 1: Syncing Devices from remote to local...
2025/04/16 00:16:59 DEBUG : rclone: Version "v1.69.1" starting with parameters ["/snap/rclone/528/bin/rclone" "sync" "murena:Devices" "/externalRAID/Murena/Devices" "--dry-run" "--delete-excluded" "--progress" "-vv"]
2025/04/16 00:16:59 DEBUG : Creating backend with remote "murena:Devices"
2025/04/16 00:16:59 DEBUG : Using config file from "/home/USERNAME/snap/rclone/528/.config/rclone/rclone.conf"
2025/04/16 00:16:59 DEBUG : found headers:
2025/04/16 00:16:59 DEBUG : Chunks temporary upload directory: https://murena.io/remote.php/dav/uploads/USERNAME2/
2025/04/16 00:16:59 DEBUG : Creating backend with remote "/externalRAID/Murena/Devices"
2025/04/16 00:16:59 DEBUG : Added delayed dir = "Fairphone_FP5_42708826", newDst=<nil>
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.1s
2025/04/16 00:16:59 DEBUG : Added delayed dir = "Fairphone_FP5_42708826/rom_settings", newDst=<nil>
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.2s
2025/04/16 00:17:00 DEBUG : Added delayed dir = "Fairphone_FP5_42708826/rom_settings/app_list", newDst=<nil>
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.3sTransferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.4s
2025/04/16 00:17:00 DEBUG : Fairphone_FP5_42708826/rom_settings/app_list/packages_list.csv: Need to transfer - File not found at Destination
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.5s
2025/04/16 00:17:00 NOTICE: Fairphone_FP5_42708826/rom_settings/app_list/packages_list.csv: Skipped copy as --dry-run is set (size 9.298Ki)
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Transferred: 0 / 1, 0%
Elapsed time: 0.5s
Transferring:
* Fairphone_FP5_42708826ā¦list/packages_list.csv: transferring
2025/04/16 00:17:00 DEBUG : Local file system at /externalRAID/Murena/Devices: Waiting for checks to finish
Transferred: 9.298 KiB / 9.298 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5s
2025/04/16 00:17:00 DEBUG : Local file system at /externalRAID/Murena/Devices: Waiting for transfers to finish
Transferred: 9.298 KiB / 9.298 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5s
2025/04/16 00:17:00 DEBUG : Waiting for deletions to finish
Transferred: 9.298 KiB / 9.298 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5s
2025/04/16 00:17:00 NOTICE: Fairphone_FP5_42708826/rom_settings/app_list: Skipped set directory modification time as --dry-run is set
Transferred: 9.298 KiB / 9.298 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5s
2025/04/16 00:17:00 NOTICE: Fairphone_FP5_42708826/rom_settings: Skipped set directory modification time as --dry-run is set
Transferred: 9.298 KiB / 9.298 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5s
2025/04/16 00:17:00 NOTICE: Fairphone_FP5_42708826: Skipped set directory modification time as --dry-run is set
Transferred: 9.298 KiB / 9.298 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5sTransferred: 9.298 KiB / 9.298 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5s
2025/04/16 00:17:00 NOTICE:
Transferred: 9.298 KiB / 9.298 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.5s
2025/04/16 00:17:00 DEBUG : 6 go routines active
Step 2: Syncing all other folders (excluding Devices) to remote...
2025/04/16 00:17:00 DEBUG : rclone: Version "v1.69.1" starting with parameters ["/snap/rclone/528/bin/rclone" "sync" "/externalRAID/Murena" "murena:" "--exclude" "Devices/**" "--dry-run" "--progress" "-vv"]
2025/04/16 00:17:00 DEBUG : Creating backend with remote "/externalRAID/Murena"
2025/04/16 00:17:00 DEBUG : Using config file from "/home/USER/snap/rclone/528/.config/rclone/rclone.conf"
2025/04/16 00:17:00 DEBUG : Creating backend with remote "murena:"
2025/04/16 00:17:00 DEBUG : found headers:
2025/04/16 00:17:00 DEBUG : Chunks temporary upload directory: https://murena.io/remote.php/dav/uploads/USERNAME2/
2025/04/16 00:17:00 DEBUG : Devices: Excluded
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.2s
2025/04/16 00:17:00 ERROR : Local file system at /externalRAID/Murena: error reading source root directory: directory not found
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.2s
2025/04/16 00:17:00 DEBUG : webdav root '': Waiting for checks to finish
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.2s
2025/04/16 00:17:00 DEBUG : webdav root '': Waiting for transfers to finish
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.2s
2025/04/16 00:17:00 ERROR : webdav root '': not deleting files as there were IO errors
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.2s
2025/04/16 00:17:00 ERROR : webdav root '': not deleting directories as there were IO errors
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.2s
2025/04/16 00:17:00 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.2s
2025/04/16 00:17:00 DEBUG : Devices: Excluded
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.3s
2025/04/16 00:17:00 ERROR : Local file system at /externalRAID/Murena: error reading source root directory: directory not found
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.3s
2025/04/16 00:17:00 DEBUG : webdav root '': Waiting for checks to finish
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.3s
2025/04/16 00:17:00 DEBUG : webdav root '': Waiting for transfers to finish
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.3s
2025/04/16 00:17:00 ERROR : webdav root '': not deleting files as there were IO errors
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.3s
2025/04/16 00:17:00 ERROR : webdav root '': not deleting directories as there were IO errors
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.3s
2025/04/16 00:17:00 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.3sTransferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.5s
2025/04/16 00:17:00 DEBUG : Devices: Excluded
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.5s
2025/04/16 00:17:00 ERROR : Local file system at /externalRAID/Murena: error reading source root directory: directory not found
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.5s
2025/04/16 00:17:00 DEBUG : webdav root '': Waiting for checks to finish
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.5s
2025/04/16 00:17:00 DEBUG : webdav root '': Waiting for transfers to finish
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.5s
2025/04/16 00:17:00 ERROR : webdav root '': not deleting files as there were IO errors
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.5s
2025/04/16 00:17:00 ERROR : webdav root '': not deleting directories as there were IO errors
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.5s
2025/04/16 00:17:00 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.5sTransferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.5s
2025/04/16 00:17:00 NOTICE:
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.5s
2025/04/16 00:17:00 DEBUG : 4 go routines active
2025/04/16 00:17:00 NOTICE: Failed to sync: directory not found
========== SYNC COMPLETED: wo 16 apr 2025 0:17:00 CEST ==========
Can someone please help me set this up? And possibly point out any caviats Iām missing in my code because Iām not an experienced coder?
This is my system info:
cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Thank you in advance.
Johan