Ideas for promoting QA

Hi all!

I stumbled upon a jobb listing recently (Director of Chaos Engineering), which made me have a look at the QA work at Canonical. I have many years of experience in both programming (been programming since the age of seven), open source (involved since 2004) and quality assurance (8 years at developing aeronautical software in different capacities).

So, this topic is both a bit of a presentation myself (don’t worry, if you read this far you’ve already made it past that bit) and some thoughts on how QA can be improved in general in the open source community.

After reading through all topics here, I have noticed a few things. First of all, the issue here is the same as in any other distribution, i.e., dependency on upstream packages. Secondly, it is hard to find information, e.g., the information is here, on the wiki, on launchpad, etc. but it is difficult to find (requires lots of reading and scrolling and following links).

This first issue is tricky to solve and will probably take years of work before it is thoroughly resolved, but there are a few steps that would be a great starting point:

  1. Creating easy-to-use QA processes for any open source project and promoting it everywhere, especially upstream.
  2. Visualise QA in open source, with badges, certifications or the like.
  3. Talk about QA in a positive sense. Often QA is seen as a road block (especially by managers) to deliveries, but in reality it reduces time spent on fixing bugs. Start talking about QA as a positive thing will help!
  4. Automate and visualise tests of every single package that are currently in the Ubuntu deliveries. Just a huge colormap (with like 30.000 points) for instance would be super intuitive. Each color would define the current state, e.g., green = passing, red = failed, purple = no tests available, etc. Clicking on an item in the map should link directly to the upstream development repository/website.

The second issue is (perhaps) easier to remedy:

  1. Visualise how the ubuntu dependency tree look on a general level and point people to it. This will help when discussing where bugs should be fixed.
  2. Use more modern tools. I totally agree with @merlijn-sebrechts about making it fun (and easy to use). That will make QA, bug fixing, etc. much more attractive to the community!

I have a tonne of more specific ideas too about QA in open source, like automation, mutational testing, requirement- and test-driven development, AI-based test scripting, etc. but maybe that’s enough for now! :slight_smile:

What do you guys think?

2 Likes

Re-wright all of the QA Testing guidelines for her and now
in 2022. I am on the QA Testing “Ubuntu Quality Team”
The QA Testing requirements do not necessarily reflect the
needed requirements for the needed testing.
If you go through and actually test each flavor of Ubuntu.
Then you will see… That the testing requirements needed
to be slightly revised.
https://launchpad.net/~ubuntu-testing

Hi neerdoc,

Thank you, for the ideas that can make Ubuntu better.

Use more modern tools.

What tools do you have in mind?

Talk about QA in a positive sense.

Do you have a positive example of how you imagine this?

Bests,
Torsten

1 Like

Welcome to the community here on Discourse, and thanks for your suggestions! Making the QA experience better is something we definitely want to do, and you’ve got some excellent points here. Thanks so much for sharing them! :raised_hands:

I am going to tag in @bwyazel @franksmcb and @dwd-daniel - Britt is my counterpart in Community Ops, aka making community processes awesome, and Bill and Daniel are two of the community members who organize our testing efforts - aka, heroes in my eyes. :star_struck: I think these would be great people for starting a conversation - and please, if you’re a QA tester or have wanted to become a tester, join the conversation!

2 Likes

Agreed, re-writing the guidelines/processes for Ubuntu-internal testing is for sure one step to start with. Setting a good example for the open source communities in general by clarifying and promoting clear and easy to understand processes for developing software within Ubuntu would be a great beginning.

I have, for instance, read through the wiki https://wiki.ubuntu.com/QATeam and I cannot find any clear points as to how one would work with quality either from a practical or a procedural point-of-view.

1 Like

Hi @torsten.franz,

I would suggest looking at tools that have a proven track-record on attracting communities, such as gitlab, stackoverflow, etc.

There is also a strong point in being able to visualize projects, workflows, bugs and progress that is also greatly helped by tools such as taiga, but there are many others too.

CI/CD capability is another obvious candidate to facilitate drive and visualization on how tests are working for instance, which today is often integrated in larger development suites but can be run separately as well.

These are just examples, and I would suggest that a bit of work is put into researching exactly what tools, suites and methods would give us the best traction.

Regarding positive talking about QA; that is a more tricky question to answer straight off. I mean, talking positively about it is one thing, getting people to understand the benefits of working with QA is quite another. I do have one success story from a previous employer, which basically was a hands on test. We had lots of different small software projects, each 8-10 weeks in size, where we (from QA) convinced one team out of 8 to follow the QA process with an open mind and not to get stressed out that it took more time from the start to get going (this was a quite rigorous requirements based project) and then we compared the different projects. Only the one following the QA-process to the full extent was ready on time… which actually got some of the managers to understand that the QA-process is their friend!

So I guess actually getting proof and showing them off is a possible idea.

1 Like

Hey everyone, @madhens tagged me in. I am Daniel, one of the two owners of Ubuntu Hideout.

I would like to offer the perspective of someone aiming to generate community interest toward, and action in, testing the Ubuntu releases. It is somewhat at a tangent to the discussion so far, but I believe this is relevant for how “internal” and community QA/ testing engage. I have conflated beta-testing and QA here however I think that the two are similar enough from the community point of view that they should both be addressed together.

Here is a breakdown of how we have proceed so far, which has worked well.

Create engagement and a sense of reward

We appeal to the “distro-hopping”, “what’s new this time” and “community contribution/ collaboration” sensibilities in our members. We pitch the proposition as mutually beneficial from the beginning stating that the desired outcome is to enjoy oneself, evaluate what does and doesn’t work with the goal of uncovering or reproducing issues. Those issues can be addressed ahead of the release date to improve the release for themself and other users.

Barrier to entry

Two common questions we’ve seen since beginning our first community testing-cycle (21.10):

  • What’s involved/ how do I test?
  • Is it difficult?

Joining the testing and QA effort is straightforward for most users who have handled an ISO before. However, the first steps to taking part must be communicated clearly so the uninitiated may join in. We explain that beta-testing is generally no more difficult than booting/installing an ISO in a VM or on bare metal and proceeding to use the OS as one usually would in their daily tasks. We provide a list of requests and ideas for features and regressions to test. This answers all three questions, puts testing in the context of normal usage and makes it easy to get involved.

Our findings from this approach

  • There has been an encouraging volume of interest in the last couple of months leading up to the testing cycle
  • The number of members who have the beta-tester role has nearly doubled since the 21.10 test cycle
  • Enabling relaxed conversation between our testers enables them to spur each other on with ideas to test things others hadn’t considered, and drives its own sense of engagement

22.04 testing-cycle

So far this cycle, we’ve arranged testing to support Firefox Snap regression tests, Kubuntu general usage testing and Ubuntu Studio general usage testing. We’re waiting for additional information before launching testing efforts for each of the other flavours/ editions, but we aim to accommodate as many as possible. As more testing threads are opened for each of the other flavours/ editions we expect to see an increase in engagement.

Comment on the initial message

Creating easy-to-use QA processes for any open source project and promoting it everywhere, especially upstream.

This idea has appealed to me for some time. I think there would be great benefit to having prominent industry players buy into this.

Visualise QA in open source, with badges, certifications or the like.

Absolutely. This is the sense of reward and recognition that members will benefit from and promote their future engagement.

Talk about QA in a positive sense. Often QA is seen as a road block (especially by managers) to deliveries, but in reality it reduces time spent on fixing bugs. Start talking about QA as a positive thing will help!

I agree that QA is infamous for being thought of as a blocker and that it’s a marketting problem: QA is ostensibly designed to help maintain and improve reliability and reputation and to identify where those hard to reproduce bugs occur. Framing it correctly is always beneficial to everybody involved.

Automate and visualise tests of every single package that are currently in the Ubuntu deliveries. Just a huge colormap (with like 30.000 points) for instance would be super intuitive. Each color would define the current state, e.g., green = passing, red = failed, purple = no tests available, etc. Clicking on an item in the map should link directly to the upstream development repository/website.

Many members of the community, especially those not used to structured QA/ testing/ bug reporting would benefit from having a liaison to work with when reporting a problem - much like the help-desk idea fielded by @teward and @madhens.

2 Likes

I agree, the subjects are clearly closely enough related to use the same (or at least similar) approaches. After all, why reinvent the wheel? I very much like the idea of measurements as well. It makes it a lot easier to get a feel for if the efforts made are making a difference or not so that the next step gets easier to plan.

This is a really interesting view of the type of the QA reputation problem! I never thought of it as a marketing problem, but it makes perfect sense. One would probably get a very nice start of making an impact on the general view of QA being a blocker or not by engaging a marketing expert on the subject!

Next steps

From what I gather from the comments, the ideas that I tried to promote seems worthwhile to pursue. And after watching the Open Source and its social impact video, which I agree with deeply, I will definitely do this openly!

There are two of my suggestions that are easy to start on, which I thus will (as time permits):

  • Create an open source QA-process (which obviously will be open source itself).
  • Create an open source project to visualize the building/testing of all packages in Ubuntu.

Future work will include:

  • Talking to prominent industry players about the open source QA.
  • Create badges, certifications, etc. for easy application of the open source QA.
  • Discuss how to market QA, and how that could be funded.
1 Like