Mir's next steps - we need your input!

Hi folks
Mir’s work to implement Wayland support is going well! The basics are in place: clients can connect, create windows and draw into them, and mouse, keyboard and touch inputs work. We’re building our Wayland Conformance Suite wlcs to ensure Mir is behaviour equivalent to other Wayland implementations.

Of course there’s many more features to implement before we have full Wayland support: copy/paste, drag & drop, more window management functionality, sub-compositor and sub-surface support. There are all the protocol extensions in the Wayland-protocols repository to deal with too. We’ve got plenty to do!

However we’re also at the place where we need to reach out to the community and ask what are the aspects of your desktop that you value most, to help us figure out a direction for Mir. Below there’s a couple of topics, please have a read, and if you’ve got any thoughts on those topics, do reply. All feedback welcome!

1. Desktop architecture

Super simply speaking, a desktop environment consists of a server/compositor/window manager to show apps windows, together with panels, docks and desktop to allow users launch those apps and switch between app windows.

There is a spectrum of ways to architect this, but here are the extrema:

Monolithic desktop

This is where server/compositor/window manager and panels/docks/desktop are all in a single process.

Pros:

  • ease of implementation: no IPC needed to sync state
  • security - impossible for apps to learn about or manipulate shell in unexpected ways

Cons:

  • non-modular, cannot mix & match third-party panels/docks.
  • less robust: a line of code in the dock can cause the entire shell to crash, losing the user’s session.
  • less efficient: rendering pipeline more complex, and more susceptible to framedrops.

This is largely how Gnome Shell’s Wayland support is implemented.

Highly-modular desktop

Here the server, compositor, window manager and each panel/dock/desktop are all individual processes

Pros:

  • can easily mix & match third-party panels/docks to build a shell that works for you.
  • more robust: separate processes means a crash in any component (except the server) is recoverable, meaning User less likely to lose session.
  • more efficient: rendering is simpler, so compositing is more efficient and less likely to drop frames.

Cons:

  • complex to implement, rely on IPC to sync state between separate processes (risking race conditions - what if server destroys a window just as compositor starts to draw it - needs extreme care to avoid)
  • security is compromised: e.g. the APIs screenshot utils or hotkey listeners use can be used maliciously by a bad app.

This is how X11-based desktops work today.

I think we all can agree we would like all of the Pros, and none of the Cons, but is that even possible? Can you suggest a design that gives the best user experience, yet doesn’t leave the developer with a complex job?

To throw my idea in the mix, I like this approach:

Server/Compositor/Window Manager is a single process. Panels/Docks/Desktop are all separate processes, but have privileged access to Wayland extension(s) to give them special APIs so they can control the window stack, position themselves on screen, etc.

I think this design would solve most issues. There are ways to implement privileged clients with Wayland already (Weston does it), but this approach carries threat of adding yet more extension(s) to the Wayland ecosystem. I don’t want Docks/Panels that only support Mir’s Wayland, they should work on all, but how to do that…

2. Desktop customisation/theming

How much UI flexibility would you like for your desktop? Do you yearn for wobbly windows and fire effects, and mourn the desktop cube? Or are you more contented with a more 2D desktop, with custom animations for window manipulations like open/close/minimize/maximise? Or is a tiling window manager more your thing?

Mir server was designed to be modular and easily integrated with other toolkits. The current built-in Mir compositor is super-basic, it draws 2d windows with no fancy effects, but it could use hardware compositing APIs on Android for low-latency and power efficient graphics. Also Mir’s window manager exposes multiple hooks to let you impose whatever WM strategy you’d like.

We want to know what kind of graphics capabilities you’d like your desktop to have. Is there an existing graphics toolkit that you think would provide all the pizazz you need? Should tiling be a built-in option?

For “prior art” consider Unity8. This was built on QtMir (similar to QtWayland server), a library on top of the Mir server library that replaced the default Mir compositor with the Qt scenegraph renderer, so we could build the whole desktop (including panels/docks) in QML. We lost access to some acceleration techniques though that raised latency (but I had ideas how to restore that!).

3. Most Requested Features

What are the aspects of your desktop you most value? Are you a gamer? If so, I bet you care about low-latency graphics and hotkey support. Do you watch lots of video? Then screen-tearing needs to be eradicated. Have you a laptop with multiple GPUs and need it to intelligently switch GPUs? But what else is important to you?

We’d love to hear what you have to say, as it’ll help us decide our future direction. Please, don’t be shy!

Thanks
-G

12 Likes

Hi greyback - I’m Drew DeVault, the author of the Sway Wayland compositor and wlroots library. These questions are things we’ve thought about and discussed a lot ourselves, too.

We’ve chosen an approach very similar to what you propose:

Server/Compositor/Window Manager is a single process. Panels/Docks/Desktop are all separate processes, but have privileged access to Wayland extension(s) to give them special APIs so they can control the window stack, position themselves on screen, etc.

I think this strikes the right balance, too.

I don’t want Docks/Panels that only support Mir’s Wayland, they should work on all, but how to do that…

We are working on a single Wayland protocol that simply and elegantly solves this problem: surface-layers. I would put a link here but this forum doesn’t allow new users to post >2 links. surface-layers defines a number of layers of the desktop (similar to Weston’s layers if you’re familiar with Weston internals) that privledged clients can arrange surfaces on and describe how they interact with input. It supports uses-cases like:

  • wallpapers
  • desktop icons
  • docks/panels
  • screen lockers (and screensavers)
  • on screen keyboards
  • notifications
  • etc

We’re still working on the details and proving it out in the wlroots reference compositor, rootston. A major goal of our current work is making modular compositor components a reality, and I’d love to work with you guys on acheiving that goal. There are other protocols that need to be written and implemented, but I think it’s very possible.

Or is a tiling window manager more your thing?

I’m not sure that you’ll have much success entering this niche. Tiling users and stacking users expect a fundamentally different experience and have fundamentally different goals/requirements. That being said, we should avoid stepping on each others toes and work together to design Wayland protocols that are suitable for both use-cases (this has been a problem for Sway in the past).

3 Likes

Hi Drew,
good to meet you! I can imagine that you’ve thought about this stuff a lot, thanks for chiming in.

It is reassuring to see you came to a similar conclusion for a good desktop architecture. We also spent a lot of time discussing it internally, but the Monolithic approach was just too convenient for Unity8 for rapid development. But as time went on we saw the drawbacks. I’m glad you’ve had more foresight than we had!

I presume this is the surface-layers Wayland extension you’re referring to:
https://github.com/swaywm/sway-protocols/blob/master/unstable/surface-layers.xml
From an initial look, it certainly looks capable of doing most things a desktop needs.

We had something similar-ish in Unity8 internally, a stack of layers that the panel, notifications, launcher and OSK lived in - it worked well, with some funkiness about having to move OSK up/down the layers depending on the UI state. Let me think about all our old use-cases and see if your proposal is sufficient. I’ll comment in the repo!

It seems we have matching goals, and to avoid diversifying the Wayland ecosystem further, we should definitely collaborate on this. I’ll check out your reference compositor and have a play!

And re tiling WMs, yeah they attract a very different customer, but we hope to keep Mir’s server APIs generic enough so that it doesn’t obstruct that goal. It may be a delicate balancing act, or not possible at all. But if people seem to care about it, we’ll try.

Thanks for that Drew! Looking forward to chatting more :slight_smile:
-Gerry

2 Likes

A highly modular approach X11-like is desirable for me. Let the window manager (KWin, Marco, Compiz) deal with the eye candy stuff. Extend the areas where Wayland lacks, like gamma API and we get a true Xorg sucessor.

2 Likes

Hey @mrnhmath
thanks for the input! Unfortunately it appears the trend isn’t going your way!

I like David Stone’s video here where he motivates the reasoning for combining the server and WM/compositor in Wayland:
https://www.youtube.com/watch?v=GWQh_DmDLKQ

But this is a thread where we listen, so your vote has been logged! Cheers!
-Gerry

1 Like

Hi Gerry! It’s great to read from you! :slight_smile:

I would like to start with a question, first: does adding wayland support to Mir mean that both Mir and Wayland clients will be able to connect and draw their windows, or am I fantasising?

As for the architecture, I also believe that having server/compositor/window manager in the same process is a good idea; I’m not sure if the panels/docks/desktop they all need to be in separate processes, but for sure even if they are in a single process they should not be in the same one as the server.

As for customisation, my vote would go for a design which offers an API to people writing the compositor/window manager. I’ve not experience in the field so it’s likely that what I’m going to say doesn’t make much sense, but I would like to have a library which implements the server/compositor/window manager’s dirty work and takes care of all the communication and synchronisation between these components, while not specifying how the UI should look like and instead providing an API to people willing to write their compositor/window manager on top of the mir/wayland protocol. So that one could write a WM using QML, the way you did with unity8, and someone else could do the same using Gtk, and the two would look completely different.

Does this make any sense? :slight_smile:

Maybe it doesn’t need to be completely modular, but some modularity would be good. I like that in KDE Plasma, plasmashell and KWin run in separate processes instead of in a single process like GNOME. If the plasmashell process crashes, it recovers almost instantly and windows remain unaffected. If KWin crashes, windows can’t be moved, but it can be restarted from Krunner.

Definitely +1 on some of the modularity arguments, but really I just want my wobbly windows back. I don’t care if I make the rest of the desktop boring and flat, wobbly windows give valuable feedback and make using the computer more fun.

Enlightenment have already made the Wayland Wobbly Windows extension, so that could be a start.

Hey Alberto!
wonderful to hear from you too! Hope you’re doing well!

You’re not fantasising, Mir currently supports both Mir and Wayland clients at the same time. Once the Wayland support is complete, I expect we’ll deprecate and eventually remove the Mir client support entirely. There’s no point in keeping it at that stage, any toolkit that has Mir support also has Wayland support.

Good point about panels/docks/desktop not needing to be separate processes, it could be just one. We should definitely keep that in mind. You’re right to isolate them from the server.

It is something we learned from Unity8 the hard way - the whole compositing thread could block due to a dock or panel doing a blocking operation (synchronous DBus calls especially hurt us sometimes). So for definite we need to help third parties avoid that pitfall.

Yep, that’s exactly what we want to offer with Mir. I think you said it perfectly:

An off-shoot question of that would be: perhaps we should write that Gtk-Mir-server library? That would add a powerful graphics engine to Mir, with APIs that look familiar to Gtk developers. It could allow customization points for drawing, animations and behaviours, so a desktop author could easily impose their own visual identity, trusting that everything else is handled sensibly by Mir.

It would be a lot more light-weight than QtMir-server is. For smaller desktop environments, that might be a compelling offering. Just an idea!
Take care!
-Gerry

Hey noahadvs,
yep, that’s exactly what inspired my partially-modular idea. There’s nothing worse than a crash loosing your entire login session - literally the worst user experience possible! So with a clever design, we can avoid that as much as possible.

Note in your example, I suspect you’re referring to an X11 setup - there KWin is the compositor/window manager, but the X server is still there in the background. That’s why when KWin dies, X still draws your windows (and you kinda can still type in the focused window), but as KWin is gone you cannot manage your windows (no move, or change focus). But if X crashes in this setup, you would lose everything. X really is robust, perhaps because it has a simpler remit.

Thanks for the input!
-Gerry

Noted! Desktops have got rather serious these days. We’ll try to make Mir relax a little :wink:
-Gerry

Aspect of my desktop that I value most (when it’s there):
Responsiveness. Low latencies both most of the time and in the 0.1% worst times.
Whatever design helps you achieve and maintain responsiveness almost all of the time in almost any situation has my vote.

2 Likes

I think eye candy is important. I’d love wobbly windows and fire effects, but if that is not possible, there should at least be animations for open/close/minimize/maximize.

Whatever approach works best is fine, as long as there is a proper security permissions system for privileged API. I don’t want trivial keylogging like in Xorg.

There’s a spectrum here: at one end there’s fixed “standard” animations that would apply on any Mir-based desktop, at the other the desktop has the power and responsibility to draw anything. Somewhere in between is the “sweet spot” where customization is possible, but isn’t too demanding.

Support for “proper security” for privileged API is integrated into Mir. We will to continue to apply that as we implement Wayland protocols.

1 Like

Thanks @harelamnon, that’s a timely reminder.

A lot of the work over the past half year has been about getting things to work (Wayland bring-up, Fedora bring-up, migration off Launchpad). I don’t think that has impacted responsiveness but we haven’t measured for a while.

Our immediate priority is to fill out the Wayland Conformance test (as that will ensure we don’t break things when optimizing), but after that we should definitely spend a little time measuring aspects of performance.

2 Likes

As just a tech head, part of me wants it to be “FULLY MODULAR!!” so I can take it all apart, then the more sane side of me gets a hold and really likes the idea of having it as Gerry has laid out. I think going with it in the “two segments” Server/Compositor/Window Manager and Panels/Docks/Desktop would be the best approach as it would keep things sectioned off.

I think as long as the focus stays on compatibility with Wayland protocols, security and being “feature” complete for the Server/Compositor/Window Manager is the best approach.

I would like to see the Panels/Docks/Desktop being more of the “Plugin” approach allowing them to be changed out and customised if wanted/needed. Which I also know could and probably would be a lot more work, but Imo would keep most interested and allow for more “Customisation”.

@switches don’t worry about being prevented from doing things! This is software and it is open source: if there’s something in there you need to “take apart” then you can always download the source and do it. (If it works out you should also write a PR.)

But when getting started on (and most of the time when using) anything you don’t need all the options all the time - that’s confusing. A lot of design is about making the right things easy to do and that’s what @greyback is aiming to achieve.

Hi,

Here are 2 important points for me:

  • it must be stable, rock-solid. It’s a low-level and critical component so it must never break. On a side note, would it make sense to use rust-lang in the future ?

  • it must be fast

And some suggestions of features:

  • it should allow remote desktop : I should be able to see and act on someone else’s desktop, just like today with vncserver & vinagre. Today, I cannot install any wayland DE onto customer’s computer because there is nothing available (or did I missed something ?). There is only a new technology proposed by Redhat (pipewire), but it seems it’ll take months/years before it becomes stable and widespread. This is incredible because RealVNC already worked on that subject in 2013-2014 ! and even provided some prototypes (https://www.phoronix.com/scan.php?page=news_item&px=MTczODg). Could MIR implement that protocol in a not-so-distant future ?

  • it would be useful to provide a “low graphics mode” (but I don’t know what changes it would require on protocols/wayland clients). Ideally, that mode would be activated dynamically, for example by vncserver to reduce bandwidth (reduce animations, transparencies, desktop background)

Cheers,
Michel

1 Like

Summoning @ikey-doherty for his opinion, even though I think he settled for another solution for Budgie 11, but still, the more input the better

Looking forward to unity 8 on mir over wayland.

4 Likes