ISO Test cases are a bit stale, but what is wrong with them?

http://iso.qa.ubuntu.com/qatracker/milestones/408/builds/216037/testcases

Has many test cases for Ubuntu Desktop Focal. Similar pages exist for other flavours too.

However, I think many of them are stale, wrong, out-of-date, or even reference UI that is not available or have steps that no longer exist.

But which ones?

Can you please try the test cases and try to spot which steps are wrong and comment on here?

For example:

Was asking to click between Try or Install Lubuntu, but when booted with either UEFI or BIOS that screen is never shown. Buggy installer, or out of date instructions?

Was asked to perform default install with steps X, Y, Z from this testcase url, but that’s not valid as Rubuntu flavour has switched from Ubiquity to Calamares installer

Last time we released Focal, it was really hard to validate iso-testing bug reports, because a lot of test cases were out of date. In the run up to the 20.04.1 release, can you please help identify bad test instructions, such that we can fix them before 20.04.1 milestone testing?

Please comment in this thread.

3 Likes

Lubuntu’s are very stale, and subsequently newer testcases were written but stalled (many of the issues with me) prior to implementation.

My third attempt is found at https://code.launchpad.net/~guiverc/ubuntu-manual-tests/lubuntu-calamares

We’ve been using a checklist since 18.10 (https://phab.lubuntu.me/w/release-team/testing-checklist/) to tick off our testcases, and recording everything on the iso.qa site under our one install option (a line like the following will be added, if passed the checklist install will be updated with details entered there)
testcase: full disk, encryption, bios, no-internet

Far from ideal for drive-by or occasional testers…

As someone who is now only an occasional ISO tester I probably report issues without properly following the testcases as I know that most of them have needed updating for sometime now.

I know of one user that cared a great deal about the accuracy of the testcases but left the Ubuntu community sometime ago and is unlikely to return.

May be someone needs to step up and volunteer to head a project to lead a team of experienced testers get those testcases updated and rewritten so that they are relevant to the needs of the various development teams and the applications that they use?

I think the auto-resize test case was was something that always confused me as there is no option called ‘auto-resize’ in the installer and at one time the test case was written to be exactly the same as another test case. (I hope I’m remembering that correctly). May be better correlation between the wording on the website and that in the installer?

And with Groovy, it won’t ever be shown like that since GRUB2 will be used for any mode: Groovy to use GRUB2 for booting installer media in any modes on all architectures.

1 Like

I can confirm that the Ubuntu MATE ones are also stale and out of date.

Perhaps I am in error but the method for updating or revising these is a bit arcane. Having these in MD format on GitHub (as an example) would appear to make maintenance and contribution more generally available.

I am more than willing to work in a project on getting these test cases revived.

2 Likes

Why not just here on discourse? It works great for ubuntu.com/server/docs which is published out of https://discourse.ubuntu.com/c/server/guide

1 Like

No issues with during it here, I was just mentioning GitHub as an example :smiley:

1 Like