Bug#1084789: d-i.debian.org: debian-testing-amd64-netinst.iso has multiple versions of module udebs

[Cyril Brulebois]

Package: d-i.debian.org
Severity: normal
Tags: d-i
Dear Debian Installer Maintainers,
debian-testing-amd64-netinst.iso (Trixie) has recently grown in size,
both daily and weekly builds.
linux-image is version 6.10.11-1, but there are multiple versions
* btrfs-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* btrfs-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* btrfs-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* btrfs-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
* crc-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crc-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crc-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crypto-dm-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crypto-dm-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
Is there any reason to have non-6.10.11-1 versions of the module udebs?
If not, then the image should be able to shrink by 9% or so.

Looping in debian-cd@, who is responsible for those images.


Cheers,

[Daniel Lewart]

linux-image is version 6.10.11-1, but there are multiple versions
* btrfs-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* btrfs-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* btrfs-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* btrfs-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
* crc-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crc-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crc-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crypto-dm-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crypto-dm-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
Is there any reason to have non-6.10.11-1 versions of the module udebs?

Now linux-image is version 6.11.5-1, and the bloat keeps increasing:
* btrfs-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* btrfs-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* btrfs-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* btrfs-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
* btrfs-modules-6.11.4-amd64-di_6.11.4-1_amd64.udeb
* btrfs-modules-6.11.5-amd64-di_6.11.5-1_amd64.udeb
* crc-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crc-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crc-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crypto-dm-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crypto-dm-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
* crypto-dm-modules-6.11.4-amd64-di_6.11.4-1_amd64.udeb
* crypto-dm-modules-6.11.5-amd64-di_6.11.5-1_amd64.udeb
...

Daniel Lewart
Urbana, Illinois

[Steve McIntyre]

Sorry this has taken so long - I've been swamped IRL :-(

linux-image is version 6.10.11-1, but there are multiple versions
* btrfs-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* btrfs-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* btrfs-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* btrfs-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
* crc-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crc-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crc-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crypto-dm-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crypto-dm-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
Is there any reason to have non-6.10.11-1 versions of the module udebs?

* btrfs-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* btrfs-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* btrfs-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* btrfs-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
* btrfs-modules-6.11.4-amd64-di_6.11.4-1_amd64.udeb
* btrfs-modules-6.11.5-amd64-di_6.11.5-1_amd64.udeb
* crc-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crc-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crc-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.10.9-amd64-di_6.10.9-1_amd64.udeb
* crypto-dm-modules-6.10.11-amd64-di_6.10.11-1_amd64.udeb
* crypto-dm-modules-6.10.12-amd64-di_6.10.12-1_amd64.udeb
* crypto-dm-modules-6.11.2-amd64-di_6.11.2-1_amd64.udeb
* crypto-dm-modules-6.11.4-amd64-di_6.11.4-1_amd64.udeb
* crypto-dm-modules-6.11.5-amd64-di_6.11.5-1_amd64.udeb
...

I think there's a problem here in (the archive|the kernel packaging),
as far as I can see. Look at

https://packages.debian.org/source/unstable/linux-signed-amd64

and you'll see that right now the linux-signed-amd64 (6.12.3+1) source
package claims to build modules for all of the following kernel
ABIs. Picking crypto-dm-modules-* as an example:

* crypto-modules-6.10.9-amd64-di
* crypto-modules-6.10.9-amd64-di
* crypto-modules-6.10.11-amd64-di
* crypto-modules-6.10.11-amd64-di
* crypto-modules-6.10.12-amd64-di
* crypto-modules-6.10.12-amd64-di
* crypto-modules-6.11.2-amd64-di
* crypto-modules-6.11.2-amd64-di
* crypto-modules-6.11.4-amd64-di
* crypto-modules-6.11.4-amd64-di
* crypto-modules-6.11.5-amd64-di
* crypto-modules-6.11.5-amd64-di
* crypto-modules-6.11.6-amd64-di
* crypto-modules-6.11.6-amd64-di
* crypto-modules-6.11.7-amd64-di
* crypto-modules-6.11.7-amd64-di
* crypto-modules-6.11.9-amd64-di
* crypto-modules-6.11.9-amd64-di
* crypto-modules-6.11.10-amd64-di
* crypto-modules-6.11.10-amd64-di
* crypto-modules-6.12.3-amd64-di
* crypto-modules-6.12.3-amd64-di

debian-cd just pulls in all the modules in a release, expecting that
to be a sensible set. WTH is happening here?

[Steve McIntyre]

Post by Steve McIntyre
I think there's a problem here in (the archive|the kernel packaging),
as far as I can see. Look at
https://packages.debian.org/source/unstable/linux-signed-amd64
and you'll see that right now the linux-signed-amd64 (6.12.3+1) source
package claims to build modules for all of the following kernel
* crypto-modules-6.10.9-amd64-di
* crypto-modules-6.10.9-amd64-di
* crypto-modules-6.10.11-amd64-di
* crypto-modules-6.10.11-amd64-di
* crypto-modules-6.10.12-amd64-di
* crypto-modules-6.10.12-amd64-di
* crypto-modules-6.11.2-amd64-di
* crypto-modules-6.11.2-amd64-di
* crypto-modules-6.11.4-amd64-di
* crypto-modules-6.11.4-amd64-di
* crypto-modules-6.11.5-amd64-di
* crypto-modules-6.11.5-amd64-di
* crypto-modules-6.11.6-amd64-di
* crypto-modules-6.11.6-amd64-di
* crypto-modules-6.11.7-amd64-di
* crypto-modules-6.11.7-amd64-di
* crypto-modules-6.11.9-amd64-di
* crypto-modules-6.11.9-amd64-di
* crypto-modules-6.11.10-amd64-di
* crypto-modules-6.11.10-amd64-di
* crypto-modules-6.12.3-amd64-di
* crypto-modules-6.12.3-amd64-di
debian-cd just pulls in all the modules in a release, expecting that
to be a sensible set. WTH is happening here?

Looking in the source package for linux-signed-amd64, I don't see
anything obviously wrong there. Any clues? :-(

[Cyril Brulebois]

Cc += ftp team (for input), kernel team (for information)

Post by Steve McIntyre
I think there's a problem here in (the archive|the kernel packaging),
as far as I can see. Look at
https://packages.debian.org/source/unstable/linux-signed-amd64
and you'll see that right now the linux-signed-amd64 (6.12.3+1) source
package claims to build modules for all of the following kernel
ABIs.

No, it does not.

Post by Steve McIntyre
* crypto-modules-6.10.9-amd64-di
* crypto-modules-6.10.9-amd64-di
* crypto-modules-6.10.11-amd64-di
* crypto-modules-6.10.11-amd64-di
* crypto-modules-6.10.12-amd64-di
* crypto-modules-6.10.12-amd64-di
* crypto-modules-6.11.2-amd64-di
* crypto-modules-6.11.2-amd64-di
* crypto-modules-6.11.4-amd64-di
* crypto-modules-6.11.4-amd64-di
* crypto-modules-6.11.5-amd64-di
* crypto-modules-6.11.5-amd64-di
* crypto-modules-6.11.6-amd64-di
* crypto-modules-6.11.6-amd64-di
* crypto-modules-6.11.7-amd64-di
* crypto-modules-6.11.7-amd64-di
* crypto-modules-6.11.9-amd64-di
* crypto-modules-6.11.9-amd64-di
* crypto-modules-6.11.10-amd64-di
* crypto-modules-6.11.10-amd64-di
* crypto-modules-6.12.3-amd64-di
* crypto-modules-6.12.3-amd64-di

That's just how packages.d.o represents things; the tracker page is
less misleading.

Post by Steve McIntyre
debian-cd just pulls in all the modules in a release, expecting that
to be a sensible set.

Any thoughts about my proposal to lift that expectation?

Post by Steve McIntyre
WTH is happening here?

A number of source versions are being kept for some reason:

$ rmadison linux-signed-amd64 -s unstable
linux-signed-amd64 | 6.10.9+1 | unstable | source
linux-signed-amd64 | 6.10.11+1 | unstable | source
linux-signed-amd64 | 6.10.12+1 | unstable | source
linux-signed-amd64 | 6.11.2+1 | unstable | source
linux-signed-amd64 | 6.11.4+1 | unstable | source
linux-signed-amd64 | 6.11.5+1 | unstable | source
linux-signed-amd64 | 6.11.6+1 | unstable | source
linux-signed-amd64 | 6.11.7+1 | unstable | source
linux-signed-amd64 | 6.11.9+1 | unstable | source
linux-signed-amd64 | 6.11.10+1 | unstable | source
linux-signed-amd64 | 6.12.3+1 | unstable | source

Note: this isn't a case of Extra-Source-Only fun.

(Another way to see this is to `apt-cache showsrc linux` in a sid
environment, you'll see the many versions.)

Binaries stay around and nothing seems to show up in the cruft report.


Cheers,

[Steve McIntyre]

Post by Cyril Brulebois
Any thoughts about my proposal to lift that expectation?

That looks eminently possible as a workaround at the very
least. Working on it now.

In fact, simpler answer: there was already code in
tools/generate_di_list to cope with udebs for multiple kernel versions
in the archive. The change of package names for udebs (no ABI included
now) broke a regexp there. I've fixed things up to cope with either
style, and that should do what we want here.

And yes: we should definitely ensure that the first image in each set
has linux-image-$foo and all dependencies. And fail the build
otherwise - this is critical, as you say!

Looking at this next.

[Steve McIntyre]

Post by Steve McIntyre

Post by Cyril Brulebois
Any thoughts about my proposal to lift that expectation?

That looks eminently possible as a workaround at the very
least. Working on it now.

In fact, simpler answer: there was already code in
tools/generate_di_list to cope with udebs for multiple kernel versions
in the archive. The change of package names for udebs (no ABI included
now) broke a regexp there. I've fixed things up to cope with either
style, and that should do what we want here.

I just triggered a manual netinst build now, and things look OK from
the log output. Grabbing the amd64 netinst to test locally now.

[Cyril Brulebois]

Post by Steve McIntyre

Post by Cyril Brulebois
Any thoughts about my proposal to lift that expectation?

That looks eminently possible as a workaround at the very
least. Working on it now.

In fact, simpler answer: there was already code in
tools/generate_di_list to cope with udebs for multiple kernel versions
in the archive. The change of package names for udebs (no ABI included
now) broke a regexp there. I've fixed things up to cope with either
style, and that should do what we want here.

OK, thanks. I had decided to wait a little for some possible feedback before
spending time looking into it on my own, I should have looked directly
 :/

Post by Steve McIntyre

And yes: we should definitely ensure that the first image in each set
has linux-image-$foo and all dependencies. And fail the build
otherwise - this is critical, as you say!

Looking at this next.

Thanks.


Cheers,

[Steve McIntyre]

Post by Cyril Brulebois

Post by Steve McIntyre

And yes: we should definitely ensure that the first image in each set
has linux-image-$foo and all dependencies. And fail the build
otherwise - this is critical, as you say!

Looking at this next.

Thanks.

Done and working. This already caused me to fix our s390x and mips64el
image generation - they've *never* had kernels pulled onto image
#1. Nobody is using these images... *sigh*

[Nicholas Geovanis]

Post by Steve McIntyre

Post by Cyril Brulebois

Post by Steve McIntyre

And yes: we should definitely ensure that the first image in each set
has linux-image-$foo and all dependencies. And fail the build
otherwise - this is critical, as you say!

Looking at this next.

Thanks.

Done and working. This already caused me to fix our s390x and mips64el
image generation - they've *never* had kernels pulled onto image
#1. Nobody is using these images... *sigh*

For the first time in truly 35 years I have administrator access to a z/OS
test LPAR. I was hoping to try Debian at some point. In my copious spare
time :-)

[Cyril Brulebois]

Post by Steve McIntyre
Done and working. This already caused me to fix our s390x and mips64el
image generation - they've *never* had kernels pulled onto image
#1.

OK. That matches the two archs I mentioned earlier as being possibly
tricky because of their kernel variations
 The bottom line is a bit meh
(regarding usage or lack thereof) but at least there's some overall
consistency, so yay?


Cheers,

[Debian Bug Tracking System]

forcemerge 1088605 1084789

Bug #1088605 [installation-reports] installation-reports: No installable kernel was found in the defined APT sources
Unable to merge bugs because:
package of #1084789 is 'cdimage.debian.org' not 'installation-reports'
Failed to forcibly merge 1088605: Did not alter merged bugs.