I found a simpler base list that we can use. Debian provides a tracker for each GNOME release based on what GNOME itself decides to release. It is simpler than trying to verify multiple files in a GNOME git repo and the GNOME git repo doesn’t handle build-depends well. The Debian tracker just provides a prettier view of the GNOME release metadata.
This allows the sets to be updated over time as GNOME makes changes to what they release.
everything that the gnome-games apt package directly depends on
Removals
vala
GNOME Shell
Starting with Ubuntu 22.04 LTS, we removed most GNOME Shell extensions but left ones that require some system integration. We are now doing a basic smoke test of those extensions as part of our SRUs. Here’s a recent example.
Mutter
As a shared critical component with Ubuntu Budgie, we also ensure that new Mutter SRUs are tested on Ubuntu Budgie (so far, the testing has been done directly by Ubuntu Budgie testers).
Apps Note
By the way, years ago, GNOME used to include a lot more optional apps in their release set but they reduced the set to a more basic set that they recommend to be installed by default. That’s how evolution and the GNOME games fell out of the set.
Diff from previous work
Here’s a diff of just the base set for 22.04 LTS compared to the earlier proposed list. It’s a little noisy because the earlier proposed list included the exceptions.
That looks like a good set to me. It includes some things that I don’t think we particularly care about (particularly gnome-console), but GNOME obviously cares about them and exercising the MRE exception for a package is an indication that we do care about it .
Having a diff over time would be nice, but I don’t think that’s a blocker.
I think this is a good methodology and list to use for the MRE documentation.
I think the next step will be to update the MRE wiki page to point to the Debian tracker and add the extra documented exceptions?
Thanks @raof! This looks good to me, though there are still some steps that I think we need to do to complete this. Maybe for the desktop team?
I’d like to be able to quickly determine if a particular package is part of the list or not, in order to actually process these SRUs. Right now that seems tricky.
Changes to the list expansion are expected and I assume a single member of the SRU team will decide if they make sense or not as MRE SRUs are reviewed. However this isn’t possible without being able to see the full expansion that was originally approved (ie. as it stands right now) and how it has changed. This is what I mean when I say I want an audit trail. Otherwise, we’d be in the odd position where upstream can effectively change our list without us noticing.
Right now it looks like specific test plans are being proposed individually in SRU bugs of MRE uploads. I suggest these are moved (following SRU team approval) to the wiki page. That way they wouldn’t have to be reviewed every time, which should reduce bikeshedding, save some copying and pasting and and make the review process smoother.
For (1) and (2) - maybe what we want is to our own script to parse the upstream metadata and map to source package names, and then commit the output (per release) to ubuntu-archive-tools? We don’t expect the list to change within a release, only between releases.
I guess the stretch goal here is for sru-review to grow support for emitting “This package falls under the $PACKAGE MRE; see $URL for details” on the relevant packages…
Thanks for working on moving things forward Chris, we will review the script and the list.
On the SRU verification would it work for the SRU team if the instructions are stored in https://wiki.ubuntu.com/DesktopTeam/TestPlans/ (where ‘sourcename’ is the name of the Ubuntu source package formatted wikistyle, GnomeTextEditor for gnome-text-editor is an existing example).
We would create the pages from now on as we SRU packages which don’t have one yet.
Having a place to keep test plans would be great - thanks @seb128 and @jbicha!
Just one note - we (SRU team) can definitely consider them case by case per SRU. But it would be better if each could be reviewed approved by the SRU team, if we could note that somewhere such that it is invalidated if there are significant changes. Then we could provide better consistency since if a test plan is already “SRU team approved” it won’t need to be reviewed again, and you wouldn’t be surprised with requests for changes or amendments if a different SRU team member reviews your upload. But then this needs to be done in a way that a future SRU team member can confirm that the test plan is the same one that was previously approved by a previous SRU team member.
I don’t mind what mechanism we use, but could we arrange that please? Since there will be a collection, perhaps we could review the first time a particular test plan is used, and then leave some kind of approval note?
I added a section to the bottom of https://wiki.ubuntu.com/DesktopTeam/TestPlans/gjs with an idea for how pages could be marked as reviewed. But it’s a decision for the SRU Team how those pages should be marked and reviewed.
I’ve refreshed the MRE list for lunar here. In doing so, the script needed additional handling of the libgweather->libgweather4 source transition, and I realised that the previous kinetic list included libgweather, which is not actually in the kinetic archives.
Hm. I was going to say that I think we’ve not yet satisfied Robbie’s “we can tell that the test plan is the same as the approved one” requirement, but on second thoughts, test plans on the Wiki satisfy that (in the same way that all the other test plans on the wiki do) - if the last edited tags match the review time and reviewer.
So, I think what is needed to complete the discussion here is for us, on the SRU team, to go through the test plans as we process GNOME MRE SRUs and approve them or request changes until we can approve them.
To get that started: a review of the Mutter test plan:
In general, I’d like something more structured than “verify things continue to work well”
As a Wayland compositor, Mutter is particularly hardware-dependent and a critical piece of infrastructure. I think we should call out specific testing on whatever set of platforms we think are appropriate (especially since around half the bugs on this particular mutter SRU are explicitly "mutter does $BAD_THING on $SPECIFIC_HARDWARE).
As a first cut of hardware we explicitly want to support, I suggest testing on:
A mesa driver (if we’re only doing one, i915 is probably the right choice; if we can do two, both i915 and amdgpu)
The most recent stable NVIDIA driver series (bonus points: most recent and oldest)
RPi? Definitely for the current SRU, as it contains an explicit fix for RPi behaviour; I’m not clear exactly how much the desktop team supports RPi-on-the-desktop, though.
The current SRU should probably have explicit testing on multi-monitor setups. I’d be inclined to request it as a part of the generic testing plan, too.
What other aspects of Mutter are high risk, or low cost to test? Scaling factors != 1.0 & 2.0? Mixed scaling? Video playback? Are gnome-shell extensions tied into Mutter, or is that just gnome-shell/gjs?
It should be fairly quick to exclude any really serious breakage on any given platform - just log in, bring up the spread, run a video player, run glmark2-es2-wayland (on Wayland sessions) should verify that a given platform hasn’t been critically regressed. I would therefore expect the major problem in testing to be access to hardware; what does the Desktop team have access to?
This looks pretty good to me. Again, I’d prefer something more concrete than “verify that things continue to work well” for Test Case 1, but I understand that it’s pretty easy to test and maybe low-value to precisely specify, so I’m OK with that.
It wasn’t immediately obvious to me where the list of extensions was for Test Case 2; I’ve edited the wiki page to make this more obvious (to me, at least!).
In reviewing a gjs SRU just now, I noticed that it didn’t link to the test plan @jbicha wrote up above and nor was that linked from the main GNOME exception documentation. So I’ve adjusted these.
I’ve also written up my idea of how we’ve agreed test plans and approvals will work. @jbicha please could you review?
On the gjs test plan itself, it looks fine to me but I’m not particularly familiar with the area. @raof please could you review it?
.