LXD instance backups questions

LXD Support
1

Hi LXD gurus,

I recently configured my homelab to use LXD for both containers and VMs and so far I am having a great time. Awesome project and awesome documentation!

As I am moving from docker, my server still has a basic ext4 setup on a RAID10 array. I am soon to move my 4 HDDs to a ZFS pool dedicated to LXD.
I am configuring automated instance exports and I found some issues and quirks I want to understand better, so here I am:

  • Backing up some of my bigger instances (one of my instances is 6TB). Backup fails after a short time. I found out that LXD uses a temporary backup storage which is in the drive where the LXD snap is installed. In my case, that is on a small partition, so I created a dedicated backup volume on the large partition. This unblocked the issue. Having said that, it looks like before being dumped to my external HDD, my entire backup holds 6TB extra on the internal RAID array. Which means that I need huge, free storage space only for the backup procedure. Am I doing something wrong and do you have any tips?

  • I noticed that when I was using slower compression, the backup would take more than 24 hours and fail. I believe it is related to how long temporary backup files are held in the temp backup volume. Could you please tell me how to change the default expiration timeout for backup temporary file retention so it does not expire?

[
{
“container_only”: false,
“created_at”: “2024-11-29T20:04:27Z”,
“expires_at”: “2024-11-30T20:04:27Z”,
“instance_only”: false,
“name”: “backup0”,
“optimized_storage”: false
}
]

Thank you!

1 Like
3

Hi,

Could you elaborate where is the desired destination of the instance backups (just for my understanding)?

If I understand correctly, you have already configured storage.backups_volume which allows configuring a custom volume for backups:

lxc config set storage.backups_volume <pool_name>/<volume_name>

Are you exporting the instance using lxc export?
If you are using ZFS and you know the same storage driver will be used for importing the instance, you can use --optimized-storage flag. This should speed up the export, especially if you are exporting the snapshots along the instance (not using --instance-only flag), as the snapshots will be just the difference from the main volume.

Could you please tell me how to change the default expiration timeout for backup temporary file retention so it does not expire?

I am not sure if we currently allow setting the expiry date for backups. Could you maybe share the error from the logs to help identify the exact cause for failing backup?

4

Hello @dinmusic and thanks for your reply.

Quick overview of the system

Partitions

  • / is on a small partition 256GB, ext4
  • /mnt/lxd is on a large partition ~22TB, ext4 (later will be repurposed to ZFS pool)

Backups

  • I run my backups with this command lxc export instance --compression=zstd
  • I backup to a single SATA disk mounted with USB3 caddy. Multiple disks exist but only one is connected during backups. All other copies are offline and safely stored.

Storage backup volume

Yes, you are correct, I already moved storage.backups_volume to /mnt/lxd where I have enough capacity to store all the temporary files even for my larger containers.

Backup expiration

I noticed that depending on container size, compression used, media speed, 24 hours could not be enough time. I don’t have the error message handy, but this is what I believe happened:

  • lxc export runs for more than 24h and completes to the temporary backup volume.
  • export to the final location then tries to start and cannot find the backup file because it was deleted.
5

Yes this should be a problem. We could probably expose backups expiry as a setting, but would need to consult with the team if there are any unforeseen reasons against doing so.

Just out of curiosity, are tarballs specifically required/desired for backups?
Otherwise, you could copy instances with snapshots to another LXD server, which does not have to be powerful, just need adequate storage space. This way, subsequent refreshes would be just deltas, drastically speeding up the process. Link to the docs: https://documentation.ubuntu.com/lxd/en/latest/backup/#secondary-backup-lxd-server

1 Like
6

Hey, thanks again for the help.

Regarding backup file retention setting, yeah it would be great if it could be changed, even if it’s a command line setting.

Regarding backups, I believe that in the target market that lxd and microcloud there could be quite a few small/medium business. In my experience a VMware like setup has fairly common scenario where backups are taken to targets like on premises NAS, cloud storage, or media that can then be taken offline.

I think that for a start, it would help to have these design choices added to the official documentation, so if errors happen, users know how to fix or work around those. It should help with the overall user experience of this already awesome project.

I am happy to run some tests or collaborate to the documentation if needed. Keep up the great work!

1 Like