I’ve come across this project called Lima (https://github.com/AkihiroSuda/lima), which aims to be a WSL-type solution for macOS, based on QEMU.
Right now it’s terminal only, but I personally would be interested in seeing GUI app support come to that (and possibly even Multipass), which got me thinking: Maybe one can solve that using Mir.
This project doesn’t have a name yet so I’m basically at the stage of making concepts right now.
The idea is to provide a Wayland compositor for guest Linux distributions, like Ubuntu and Fedora,
having the Wayland client protocol handled by Mir, and providing input forwarding + display of the surfaces using a client on the macOS side, connected over a vsock across host/guest boundaries.
It should provide seamless integration into the host OS, basically displaying Wayland windows as separate windows on macOS instead of typical per-screen QEMU windows.
The foundation builds on top of:
QEMU for virtualization
Virgl for virtual GPU + OpenGL support
Mir + this project to fulfill the needs of Wayland clients
So I’m preparing and working out how to solve it. Basic questions around it are:
Is there anything needed in Mir? (I guess not, but I could be wrong)
Is it enough to scan out surface data on the guest compositor and sending surfaces over the wire?
Should this make use of RDP (+ RAIL & VAIL) or is a custom protocol sufficient?
This sounds interesting, and agreed most of the bits should be in place already.
I believe WSL2 implements a Wayland compositor on the Windows side, and I would suggest looking into a similar approach with Mir being the compositor on the macOS side. So you’d actually export the Wayland socket over vsock into the VM. This way you don’t need another abstraction layer like RDP.
Firstly, virgil does meet Mir’s graphics needs, but Mir is intended to do compositing and window management and expects to get a display buffer and composite into that.
I know it isn’t the end goal, but a Mir “platform” that implements a window on the macOS host and lets Mir render into that as an “output” would be a good and usable first step. (Like the existing “x11” platform - which also provides input forwarding.)
To get better integration into the host environment by presenting application windows separately needs a lot of the “Window Management” and compositing logic in Mir needs to be passed through to the host. I think that can be done, but I’d aim for the less ambitious goal first.
Once upon a time (before my volunteer work in other projects in the Ubuntu space) I was eager to write a remote desktop platform for Mir. I guess it’s time to pick that up again.
Though I’m wondering, if the platform is supposed to allow rendering on “fake” screens, is it even possible using current means to access and read out Wayland buffers directly from within the platform? I’m just thinking ahead with RAIL/VAIL.
Anyhow, copying pixels seems inevitable as long as virtual GPU resources cannot be shared with the host system directly, so I think something with compression and dirty regions like RDP sounds like a reasonable approach (even though I don’t have first-hand experience with the protocol)
This part of Mir is something that @raof is reworking for the next release, that rework should simplify some of what you will need. There will be additional work involved but it should be possible to rework the “compositing and rendering” logic so that a platform can ship surfaces out of the VM individually instead of creating a display buffer.
Ok cool, I’m probably not going to wait for it to be finished though, in the meantime I’ll focus on a PoC and on the protocol between the host and the guest. Will have to look up whether the FreeRDP changes have actually been upstreamed by Microsoft. Also, it will probably require some changes to QEMU to not open an on-screen window.
Anyway, as a quick and dirty (and yes, unnecessarily slower) approach I’ll write a little shell that just sends over pre-rendered glReadPixel’ed buffers over the wire first. Doing input injection at the platform level seems a little bit more work to get it right, since it should be fair to use it for proper remote desktop purposes too.
I just started implementing a little on-screen shell that will send window contents over RDP using VAIL + Snap packaging for easy distribution.
It’s a stripped down Kiosk shell for now. I’m currently at the stage where I’m learning about the FreeRDP API and Microsoft’s changes to make this work, so it might take a month or two until I can show off something.
I will make sure to have the license compatible with Mir itself, so that code can easily be pushed into a platform plugin later on.
The repo for the guest-side is here: https://github.com/fredldotme/stereopticon
If you desire to help out and/or give suggestions, then this is the place to go.
I will definitely check the forums to stay on top of news, but if you want me to notice you, then check the repo.
I’ve read about Microsoft’s approach again and it seems like they’re using the same technique as the PoC will. Specifically:
Please note that for the first release of WSLg, vGPU interops with the Weston compositor through system memory. If running on a discrete GPU, this effectively means that the rendered data is copied from VRAM to system memory before being presented to the compositor within WSLg, and uploaded onto the GPU again on the Windows side. As a result, there is a performance penalty proportionate to the presentation rate. At very high frame rates such as 600fps on a discrete GPU, that overhead can be as high as 50%. At lower frame rate or on integrated GPU, performance much closer to native can be achieved depending on the workload. Using a vGPU still provides a very significant performance and experience improvement over using a software renderer despite this v1 limitation.
In the new world there will be a kernel module providing the ability to share surfaces from the virtual GPU’s memory via DMA-BUFs to an IVSHMEM device.
The compositor then provides those DMA-BUFs to the kernel device whenever new surface content is ready to be presented.
A command issued via a vsock causes the host display to refresh the window contents and upload them to the hosts VRAM.
The host painter responds via the same vsock, causing the guest compositor to wait for the next surface to be redrawn.
Input is passed from the host to the guest through the same vsock.
This allows for much better performance by reducing the number of copies the guest and the host have to do, also removing the need for (de-)compression on both ends.
Does this sound like a feasible approach or am I missing something? Input is highly welcomed.
(Of course I would like to build the reference guest OS on Ubuntu, if that’s okay. )
This is a quick diagram of what I envision the architecture to look like.
