Roland Clobus
2023-07-26 09:20:02 UTC
Hello Debian-cd Team and Phil,
Now that the busy period for releasing Debian 12.1 is over and some of
the live images have been verified by openQA (via me), would it make
sense to think about automating the tests for the officially released
Debian images, and the weekly builds as well?
My thoughts/ramblings:
* As soon as an image has been generated, and it has been made
accessible via an URL, openQA will be invoked and starts to download and
test the image (i.e. the generator will trigger openQA instead of openQA
polling)
** Phil will be able to generate API keys for openQA
** I've already implemented a similar setup on Jenkins [1] for the live
images [6]
** Phil has already implemented a similar setup for the netinst images,
using polling [2]
* By testing on virtualised hardware, at least many of the manual,
tediously repeating tests can be verified to work correctly, which could
make the tests on real hardware faster, because less needs to be tested
* Automated tests would automatically see e.g. kernel mismatches in the
installer [3]
** However, for the live images (based on testing and unstable) I've
implemented an automatic kernel selection, which saves additional
maintenance [4]
* The automated tests will show issues earlier, but that would require
regular monitoring/dashboarding
** I've tried to tags the issues that I've reported [5]
* For the medium to long term, would it make sense to shift these test
from debian.net machines to debian.org machines?
** The workload on osuosl3-amd64.d.n is already rather high
That's already a lot for a single mail,
with kind regards,
Roland Clobus
[1] https://jenkins.debian.net/view/live/
[2] https://openqa.debian.net/group_overview/10
[3]
https://openqa.debian.net/tests/overview?distri=debian&version=testing&build=20230724_1119-testing-amd64&groupid=10
[4]
https://salsa.debian.org/live-team/live-build/-/blob/master/scripts/build/installer_debian-installer#L309=
[5]
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=debian-qa%40lists.debian.org&tag=openqa&format=html#results
[6] https://openqa.debian.net/group_overview/14
Now that the busy period for releasing Debian 12.1 is over and some of
the live images have been verified by openQA (via me), would it make
sense to think about automating the tests for the officially released
Debian images, and the weekly builds as well?
My thoughts/ramblings:
* As soon as an image has been generated, and it has been made
accessible via an URL, openQA will be invoked and starts to download and
test the image (i.e. the generator will trigger openQA instead of openQA
polling)
** Phil will be able to generate API keys for openQA
** I've already implemented a similar setup on Jenkins [1] for the live
images [6]
** Phil has already implemented a similar setup for the netinst images,
using polling [2]
* By testing on virtualised hardware, at least many of the manual,
tediously repeating tests can be verified to work correctly, which could
make the tests on real hardware faster, because less needs to be tested
* Automated tests would automatically see e.g. kernel mismatches in the
installer [3]
** However, for the live images (based on testing and unstable) I've
implemented an automatic kernel selection, which saves additional
maintenance [4]
* The automated tests will show issues earlier, but that would require
regular monitoring/dashboarding
** I've tried to tags the issues that I've reported [5]
* For the medium to long term, would it make sense to shift these test
from debian.net machines to debian.org machines?
** The workload on osuosl3-amd64.d.n is already rather high
That's already a lot for a single mail,
with kind regards,
Roland Clobus
[1] https://jenkins.debian.net/view/live/
[2] https://openqa.debian.net/group_overview/10
[3]
https://openqa.debian.net/tests/overview?distri=debian&version=testing&build=20230724_1119-testing-amd64&groupid=10
[4]
https://salsa.debian.org/live-team/live-build/-/blob/master/scripts/build/installer_debian-installer#L309=
[5]
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=debian-qa%40lists.debian.org&tag=openqa&format=html#results
[6] https://openqa.debian.net/group_overview/14