Debian ISO Testing

[Andrew M.A. Cater]

Dear Debian team,
I'm very pleased to be involved in the Debian project. Thank you for having
me. Let me introduce myself. I'm Martin, I am a developer based in
Manchester in the UK and I've recently decided to dive head first into the
wonderful world of FOSS. I have been in the Microsoft ecosystem for the best
part of two decades and only ever dabbled in Linux and Debian a little bit
during that time. I recently took part in the Debian 12.1 ISO release event
and I had fun, learned a lot of things and met some interesting people
(including the former DPL Sledge). I had some suggestions based on my
experience that I would like to share with you.
2) Recognition of testers. Although the small group over on #debian-cd on
OFTC were thankful for everyone's contributions including my own, it would
be really great for new testers like myself to have some kind of formal
project recognition. It may seem like a small thing but it makes a huge
difference to me personally. There are several ways of recognising the
testers...some easy some not so easy. Firstly, when publishing the release
notes, naming the testers (by their chosen names) that helped test the
images before publication would be a nice way to recognise us. Also, as
cheesy as this may seem or sound, awarding a certificate to recognise that
that person has participated in the Debian release testing process. Doesn't
have to be printed of course, it could be designed in Gimp or whathaveyou. I
have mocked a design which you can view here - > https://justpaste.it/de4hd
(NB: this is a heads up. This link is SFW...if you happen to be in the
office right now).

If it helps, you're now a Debian contributor by virtue of having edited the wiki and done some testing.

3) Overhaul of the wiki testing process. Rattus proposed some overhauling of
the ISO testing situation with the wiki. I am happy to work on that with you
Rattus if you get time and we have something where we don't need to wait for
the wiki to be released from another user. Although, I'm also happy to keep
it as is if that's what the community wants. If it's a nod to times of old,
I'm all for that too. I found the process quaint and reminded me of the old
days of locking Excel workbooks. :D
Thanks for reading this, if you want to chat, I'm in various channels on
Libera and OFTC on IRC or you can respond to this mail. :)
Thanks and regards,
Martin McCarthy.
==================================================
Remember when you were young? You shone like the sun.
Breathe, breathe in the air...don't be afraid to care.
Leave, but don't leave me...look around, choose your own ground.
==================================================

amacater on IRC :)

[Steve McIntyre]

Hi Martin!

I'm very pleased to be involved in the Debian project. Thank you for having
me. Let me introduce myself. I'm Martin, I am a developer based in Manchester
in the UK and I've recently decided to dive head first into the wonderful
world of FOSS. I have been in the Microsoft ecosystem for the best part of
two decades and only ever dabbled in Linux and Debian a little bit during
that time. I recently took part in the Debian 12.1 ISO release event and I
had fun, learned a lot of things and met some interesting people (including
the former DPL Sledge). I had some suggestions based on my experience that I
would like to share with you.
1) When I was conducting testing of Debian 12.1, I had an idea of writing a
"Testing Wizard" Bash/Perl/Python script to help with the testing process. A
simple shell script that will test the ISO against a predefined set of tests
and gather pertinent information (such as hardware information for the team).
The tester would run the script, all the automatic code would execute (such
as gathering hardware information, ISO name, boot time, etc) then the wizard
would guide the tester through a series of required tests and then possibly
some random ones thrown in (in order to increase chances of catching bugs
before publishing). I'd be interested to hear other people's opinions on it
and possibly we could have a session to discuss it in a bit more detail if
you're interested. If I'm overthinking it or over egging the pudding here, I
am also interested to hear your feedback there too. :-)

You're describing something that overlaps quite a bit with what we've
discussed already (your point 3 below). We'd like to have a system
set up to guide testing and help us test most effectively:

* minimise the number of tests we need to cover all the features
* track what hardware people are using
* lead people through the tests if needed, to make things easier

2) Recognition of testers. Although the small group over on #debian-cd on
OFTC were thankful for everyone's contributions including my own, it would be
really great for new testers like myself to have some kind of formal project
recognition. It may seem like a small thing but it makes a huge difference to
me personally. There are several ways of recognising the testers...some easy
some not so easy. Firstly, when publishing the release notes, naming the
testers (by their chosen names) that helped test the images before
publication would be a nice way to recognise us. Also, as cheesy as this may
seem or sound, awarding a certificate to recognise that that person has
participated in the Debian release testing process. Doesn't have to be
printed of course, it could be designed in Gimp or whathaveyou. I have mocked
a design which you can view here - > https://justpaste.it/de4hd (NB: this is
a heads up. This link is SFW...if you happen to be in the office right now).

As Andy C already mentioned, we're already keen to track and recognise
the efforts of volunteers in Debian. If you look at

https://contributors.debian.org/

you'll see that we track what people are doing in a number of ways. If
you follow the link to

https://contributors.debian.org/source/Release%20image%20testers/

that's the set of data for people who've helped test our images. For
you to appear there too, you just need to set up an account and log in
to claim your data. The data is in the system, we just deliberately do
*not* publically list people who haven't signed up. Privacy,
etc... :-)

3) Overhaul of the wiki testing process. Rattus proposed some overhauling of
the ISO testing situation with the wiki. I am happy to work on that with you
Rattus if you get time and we have something where we don't need to wait for
the wiki to be released from another user. Although, I'm also happy to keep
it as is if that's what the community wants. If it's a nod to times of old,
I'm all for that too. I found the process quaint and reminded me of the old
days of locking Excel workbooks. :D

*grin*

4) Automation. I read a thread on this mailing list recently talking about
openQA and whilst I am not opposed to automatic testing by machines, I would
insist that the manual testers are still retained as relying solely on
machines for QA can be problematic (machines are not perfect just like
humans). Not only that, the testers over on #debian-cd is a community and
having full automation would mean that community no longer meets regularly to
test stuff...and thus the community disperses and fades away over time. FOSS
to me is all about community so I would hope that openQA works alongside the
ISO testers rather than replace us. This is slightly off-topic but AI is
doing a lot of damage to corporate tech and I hope that the FOSS projects
don't fall victim to it too.

ACK.

[Philip Hands]

4) Automation. I read a thread on this mailing list recently talking
about openQA and whilst I am not opposed to automatic testing by
machines, I would insist that the manual testers are still retained as
relying solely on machines for QA can be problematic (machines are not
perfect just like humans). Not only that, the testers over on #debian-cd
is a community and having full automation would mean that community no
longer meets regularly to test stuff...and thus the community disperses
and fades away over time. FOSS to me is all about community so I would
hope that openQA works alongside the ISO testers rather than replace us.
This is slightly off-topic but AI is doing a lot of damage to corporate
tech and I hope that the FOSS projects don't fall victim to it too.

I don't think there's any danger of OpenQA doing this (speaking as the
person who set openqa.debian.net up) because the by-hand testing and
openQA are complementary rather than competing. They're aimed at
differing targets.

The automated tests are mostly about catching regressions, mostly during
development, and I'd hope that as the salsa integration comes together
they'll become something that can provide confidence to people as they
contribute to D-I.

My perception is that the by-hand testing is more about making sure
things are working across as wide a set of use cases on as wide a range
of hardware as possible, and is focussed around release time.

Writing automated tests that can notice things looking "weird" is rather
hard -- OpenQA is very good at noticing that someone changed the font
kerning very slightly (which is where many of our false positives come
from), but if something is broken on a bit of the screen that's not
specified as something to look out for in the test, it'll never notice.

Also, testing on real hardware is not exactly trivial with OpenQA, so
that has not been a priority, but even if we do get to the point where
we can do openQA testing of real hardware, I wouldn't expect that the
spectrum of hardware under test would be anywhere near as wide as
currently gets tested by hand, nor would I expect for it to be useful
(or even possible) to automate all of the tests done by hand.

Of course, if fleshing out the set of tests that OpenQA runs (on VM or
real hardware) renders some of the by-hand tests mostly redundant, that
doesn't seem terrible, especially if we concentrate on automating the
tests that are particularly tedious to do by hand. (suggestions welcome :-) )

Cheers, Phil.