Improve VM import from external sources

LXD Specifications
1
Project LXD
Status Implementing
Author(s) @dinmusic
Approver(s)
Release
Internal ID LX065

Abstract:

We propose an extension to the LXD server that will allow the lxd-migrate tool and LXD UI to support additional VM image formats when importing an instance into LXD. Currently, only raw disk images can be imported. Our goal is to include support for VMDK (Virtual Machine Disk) and QCow2 (QEMU copy-on-write) formats. This involves integrating image conversion on the server side, to enable various clients (particularly the LXD UI) to seamlessly support instance imports. Additionally, we aim to streamline the import process for Windows virtual machines by automating the installation of required drivers during import.

Rationale:

Expanding the lxd-migrate tool to support multiple virtual machine import formats, including VMDK and QCow2, will significantly increase its utility.
Automating the installation of necessary drivers during the migration process will further streamline the workflow and make the tool more user-friendly and efficient.

Specification

Background

The current iteration of the lxd-migrate tool is limited to importing raw disk image formats. This limitation requires users to convert their disk images to the raw format before importing them into LXD.

For Windows disk images that originate from different hypervisors (other than QEMU/KVM), it is essential to install the virtio-win driver beforehand. This process currently requires users to manually download the necessary drivers and use the virt-v2v tool, which installs those drivers during disk image conversion.

We have also observed that some older Linux distributions have virtio drivers present but not in the initramfs, which leads to boot issues after instance is imported. Utilizing the virt-v2v tool for conversion to raw format can effectively resolve this problem by rebuilding drivers into the initramfs.

To address the above issues and improve the user experience when importing instances to LXD, we plan to convert any imported disk image (except for raw format) using a virt-v2v tool once the image is uploaded to the server.

Import Workflow

The import process in LXD involves transferring the disk image to the LXD server’s backups directory. The storage.backups_volume setting allows specifying a storage pool for this transfer.

Upon successful transfer the image is analyzed using qemu-img info to extract the image format and its final size (after potential decompression). If instance’s volume is large enough to fit the converted image, the image is converted to a raw format directly into instance’s root volume. Otherwise the process is aborted.

If driver injection is enabled, the instance’s root volume is further analyzed to determine whether any partition is formatted as NTFS . In such case, we assume that the image contains Windows operating system, and required Windows drivers are downloaded into LXD’s cache directory. Additional files required for Windows conversion are as follows:

Note that in the initial implementation, the virt-v2v-in-place will not be shipped with LXD Snap package, as it was initially planned. Therefore, the virt-v2v-in-place needs to be manually installed on the host where LXD server is running.

Packaging Changes:

LXD Snap

Snap package will be potentially extended with virt-v2v-in-place tool, and rhsrvany.exe and pvvxsvc.exe files that are required when importing Windows images. The initial implementation will require manual installation of virt-v2v-in-place on the host where LXD server is running.

API Changes:

The LXD migration API will be extended to support new instance source type conversion and field ConversionOptions. New type is introduced to retain backwards compatibility where instance is created from a raw image format.

Conversion options allow to choose how the image should be converted during the import phase.
The following options will be supported:

  • format (default) - Convert image into a raw format raw format, but virtio drivers are not injected. This works for images that already have virtio drivers present.

  • virtio - Inject virtio drivers using virt-v2v-in-place. When enabled, server expects virt-v2v-in-place to be installed on the host.

  • none - Indicates that conversion should be skipped and the disk imported as is. This option is mutually exclusive with all other options.

type InstanceSource struct {
	// Add new valid value "conversion".
	Type string `json:"type" yaml:"type"`

	// Accepts a list of converison options described above.
    // Example: ["format", "virtio"]
	ConversionOptions []string `json:"conversion_options" yaml:"conversion_options"`
}

CLI Changes:

Introduce new flag --conversion within lxd-migrate tool.

Database Changes:

  • None

Upgrade Handling:

  • None

Spec Changes:

  • Initially, virt-v2v was planned both for image conversion into RAW image format and injecting virtio drives into the raw image. This approach was replaced with qemu-img for converting the format and virt-v2v-in-place for injecting the virtio drivers. This is because virt-v2v supports only conversion to the filesystem which would in terms of LXD support only storage driver dir. For other storage drivers, the image would first require conversion on the filesystem (in backups directory) and then to be moved into into instance’s volume.

  • The virt-inspector will not be used for detecting Windows operating system, as this would drastically prolong the import/conversion process. Instead, we will check the partition format to determine if any of them is NTFS and download Windows drivers into the cache.

2 Likes
2

We can clarify this section that virt-v2v will take the virtio drivers from the guest and rebuild them into the initramfs to allow booting.

3

How about this:

We propose an extension to the LXD server that will allow the lxd-migrate tool and LXD UI to support additional VM image formats when importing an instance into LXD.

1 Like
4

This is true, but virt-v2v wont help with that. The scenario where virt-v2v will help with is where the guest has virtio drivers present but not in its initramfs.

5

We should still use virt-v2v tool even if its raw in case the guest needs any modification for its virtio drivers.

1 Like
6

Can you mention here that this allows using the storage.backups_volume setting to store temporary uploads in a volume on a storage pool too.

7

We need to check how this is done to ensure that we dont try and mount the guest on the host.

8

Great job, Din!

Tom’s point about checking if we mount guest’s filesystems on the host is really interesting. If I don’t miss anything all the guest’s filesystems should be mounted inside the temporary QEMU VM that virt-v2v creates. But we need to check that. The easiest way to do that is to take a look into the virt-v2v logs (with the verbose mode turned on).

2 Likes
9

@tomp Thanks for the comments, I’ve improved the spec accordingly.

@amikhalitsyn As you have pointed out, all the guest’s filesystems are mounted within a temporary QEMU VM, even during the inspection of the image.

1 Like
10

@tomp @amikhalitsyn Could you please take another look at the spec? It has been updated with the recent changes.

1 Like
11

How do you indicate you want to do both?

12

Whats the plan with ensuring the additional agent binary isn’t also injected?

13

We will also need to consider how this cached download should be refreshed periodically, otherwise it wont get new versions.

14

Did you confirm whether qemu-img can accept an input stream, and whether it can auto detect the input format?

15

While qemu-img info can auto detect which input format it got, this relies on content sniffing which has been a source of CVEs (https://seclists.org/oss-sec/2014/q1/662, https://seclists.org/oss-sec/2015/q2/704) and IIRC, the reason why we always provide it -f <FORMAT>.

2 Likes
16

We could likely run qemu-img in a locked down way (non-root, reduced caps, apparmor etc) to allow the user experience benefits of auto detection without the risk.

17

. format (default) - Convert image into a raw format raw format, but virtio drivers are not injected. This works for images that already have virtio drivers present.
. virtio - Inject virtio drivers using virt-v2v-in-place. When enabled, server expects virt-v2v-in-place to be installed on the host.

How do you indicate you want to do both?

My though was that if virtio option is enabled that format is enabled as well. So that virtio is just addon on top of format.

We could change that so we accept a list of flags, to avoid needing all options? For example:

--conversion=virtio,format

Whats the plan with ensuring the additional agent binary (qemu guest agent) isn’t also injected?

Virt-v2v does not support disabling it. We could either modify virt-v2v (which I would personally avoid if not really necessary, as we would need to keep that version in sync with upstream) or disable/remove it when lxd-agent is mounted?
Does anyone have any better suggestion?

Did you confirm whether qemu-img can accept an input stream, and whether it can auto detect the input format?

The manpage of qemu-img states that the source should be a filename.

I’ve tried piping the file directly into qemu-img convert and the error states that it qemu-img expects regular file:

$ cat img-u22.vmdk | qemu-img convert -f vmdk -O raw /dev/stdin img-u22.raw
qemu-img: Could not open '/dev/stdin': 'file' driver requires '/dev/stdin' to be a regular file
18

OK, have you thought about how the lxd-agent unit files are going to get installed?

19

OK at least we know we do need to do the temporary upload to a file.
I’m not too surprised TBH because I expected that converting formats may require some back and forth reading the input file.

1 Like
20

I like that approach. Although ordering shouldn’t matter.

Would the default value for that flag be set to virtio,format?
But would we also support a value of --conversion=none?
So that if the target server doesn’t have the new API extension, we would require them to specify --conversion=none?