Bug#1031622: d-i regression in weekly builds: FEATURE_C12 unsupported by the installed e2fsck

[Cyril Brulebois]

Are d-i alphas and weekly builds built differently? Is it perhaps the
case that alphas are built from testing udebs, while weeklies are built
from unstable udebs?

Let's quote <https://www.debian.org/devel/debian-installer/index.en>:
- Or install the current weekly snapshot of Debian testing which uses

Current weekly snapshots

- If you prefer to use the latest and greatest, either to help us test
a future release of the installer or because of hardware problems or
other issues, try one of these daily built images which contain the
latest available version of installer components.

Current daily snapshots

We're in the first case:
https://cdimage.debian.org/cdimage/weekly-builds/amd64/iso-cd/

I don't know how to list the versions of the installed udebs and
mke2fs doesn't seem to have a --version option

You can check /var/lib/dpkg/status (or look at the MANIFEST.udebs file
in the d-i tree if you know where udebs came from).

Here, e2fsprogs-udeb is at 1.47.0-1, which explains the problem. But the
installer is clearly not “the same versions of the installer as the last
release” since it features linux 6.1.8-1


I had expected that weekly builds are expected to be able to install
testing. If that expectation is correct, then that means weekly builds
need to be built from udebs that will create a filesystem that is
considered acceptable by testing user-space (and bootloaders and kernels,
but this bug report is about user-space).

Closing this bug report as it's not a practical issue with the version
just released, and since it shouldn't be an issue with later releases
given grub2 in testing should cope just fine.

Note that my bug report is not about whether grub2 in testing copes
with the filesystem created by d-i: when I installed from the 2023-02-09
weekly build, grub successfully loaded my kernel and initramfs, so that
part at least seems fine. The issue I was having is that the *initramfs*
from the testing system I installed did not cope.
If I'm right about weeklies being built from unstable udebs, then I
a version of e2fsprogs that can fsck filesystems with metadata_csum_seed
and orphan_file migrates to testing; or e2fsprogs stops enabling those
features on at least the filesystems created in d-i (optionally all new
filesystems, but this particular bug report is about d-i only).

Cheers,

[Cyril Brulebois]

Hi Simon,

Post by Cyril Brulebois

Are d-i alphas and weekly builds built differently? Is it perhaps the
case that alphas are built from testing udebs, while weeklies are built
from unstable udebs?

- Or install the current weekly snapshot of Debian testing which uses

Current weekly snapshots

- If you prefer to use the latest and greatest, either to help us test
a future release of the installer or because of hardware problems or
other issues, try one of these daily built images which contain the
latest available version of installer components.

Current daily snapshots

https://cdimage.debian.org/cdimage/weekly-builds/amd64/iso-cd/

I don't know how to list the versions of the installed udebs and
mke2fs doesn't seem to have a --version option

You can check /var/lib/dpkg/status (or look at the MANIFEST.udebs file
in the d-i tree if you know where udebs came from).
Here, e2fsprogs-udeb is at 1.47.0-1, which explains the problem. But the
installer is clearly not “the same versions of the installer as the last
release” since it features linux 6.1.8-1


Having focussed my attention on d-i releases for so many years whenever
debian-cd was involved, it appears I totally missed a big change that
happened before I drew short straw for “who's doing d-i now?”, and that
was never documented on the website, so we've been lying for 12 years


Weekly build setup change:
https://salsa.debian.org/images-team/setup/-/commit/6dcb58e3259036b56925ef277010ce85037b7abd

Tentative fix in:
https://salsa.debian.org/webmaster-team/webwml/-/commit/a22870164df5007ae4f4e356dfe54983be0f1e9e

Already visible on:
https://www.debian.org/devel/debian-installer/index.en


Sorry for the confusion
 and thanks for insisting after my initial and
too hasty “it's a known bug, everything's fine” half-assessment



Cheers,

[Simon McVittie]

Post by Cyril Brulebois

Are d-i alphas and weekly builds built differently? Is it perhaps the
case that alphas are built from testing udebs, while weeklies are built
from unstable udebs?

In conclusion: yes, that guess was correct.

Post by Cyril Brulebois
https://salsa.debian.org/webmaster-team/webwml/-/commit/a22870164df5007ae4f4e356dfe54983be0f1e9e

Thanks for fixing that!

If I understand the situation correctly, that means the regression here is
indeed caused by the mke2fs in e2fsprogs/unstable defaulting to creating
a filesystem that cannot be fsck'd by the e2fsck in e2fsprogs/testing,
because orphan_file is a "compat" feature which can be ignored without
error by older kernels and other filesystem consumers like grub, but
due to the nature of a fsck tool, e2fsck is unwilling to tolerate "compat"
features that it doesn't understand?

(This is orthogonal to the handling of "incompat" features as discussed
(at some length) in #1030939.)

If that's the case, then I think because of the way our d-i/debian-cd
daily and weekly builds are done, filesystem maintainers need to make
sure that their filesystem-creation tools don't enable a new feature
by default until that feature is (at least minimally) supported by the
corresponding fsck tool *in testing*, and immediately enabling a new
feature as soon as the fsck tool in unstable supports it is too soon.

In that case this report should probably be reassigned to e2fsprogs,
as a request to stop enabling orphan_file by default until at least the
time that e2fsprogs (>= 1.47.0) has reached testing (but perhaps lower
risk to delay enabling it by default until post-bookworm).

Cyril, sorry if I've been saying "d-i" too often here: I don't know
which bits of this are a d-i responsibility and which are a debian-cd
responsibility. I reported this to installation-reports because I didn't
know which component was causing this, only that an installation that
I thought was intended to be a supported use-case had failed.

smcv

[Cyril Brulebois]

Post by Simon McVittie
If I understand the situation correctly, that means the regression here is
indeed caused by the mke2fs in e2fsprogs/unstable defaulting to creating
a filesystem that cannot be fsck'd by the e2fsck in e2fsprogs/testing,
because orphan_file is a "compat" feature which can be ignored without
error by older kernels and other filesystem consumers like grub, but
due to the nature of a fsck tool, e2fsck is unwilling to tolerate "compat"
features that it doesn't understand?

Just to be clear: mke2fs from e2fsprogs-udeb/unstable (the “installer is
based on Debian unstable” part) creating a file system that fsck from
e2fsprogs/testing (the “install Debian testing” part) doesn't understand.

(e2fsprogs/unstable as a binary package wasn't involved in your scenario.)

Post by Simon McVittie
If that's the case, then I think because of the way our d-i/debian-cd
daily and weekly builds are done, filesystem maintainers need to make
sure that their filesystem-creation tools don't enable a new feature
by default until that feature is (at least minimally) supported by the
corresponding fsck tool *in testing*, and immediately enabling a new
feature as soon as the fsck tool in unstable supports it is too soon.

That would seem sensible to me.

Post by Simon McVittie
In that case this report should probably be reassigned to e2fsprogs,
as a request to stop enabling orphan_file by default until at least the
time that e2fsprogs (>= 1.47.0) has reached testing (but perhaps lower
risk to delay enabling it by default until post-bookworm).

The immediate issue should go away for Bookworm anyway:
https://bugs.debian.org/1031325#202

And once the feature is turned off, the package might migrate, and
everything should be all set for when the feature is turned on again.
But feel free to reassign this bug report to keep track of it, there's
nothing to be done on the debian-boot/debian-cd side.

Post by Simon McVittie
Cyril, sorry if I've been saying "d-i" too often here: I don't know
which bits of this are a d-i responsibility and which are a debian-cd
responsibility.

That's fine, debian-cd has been Steve mostly, even if I've been getting
more involved over the last two release cycles; bugs reports generate the
same “bug ownership” feeling when they pop up in either side. Both
debian-installer and debian-cd (as in debian-cd.git and setup.git) are
inherently intertwined anyway.

(Except when I'm utterly confused by a longstanding documentation vs.
reality mismatch) I tend to have a vague idea of what's going on in both
areas to figure things out


Post by Simon McVittie
I reported this to installation-reports because I didn't know which
component was causing this, only that an installation that I thought
was intended to be a supported use-case had failed.

Everything you did was very fine. I just didn't realize we were actually
publishing images where we ship known bugs
 :(


Cheers,

[Simon McVittie]

Control: reassign -1 src:e2fsprogs 1.47.0-1
Control: retitle -1 mke2fs: should not enable orphan_file by default until trixie
Control: affects -1 installation-reports

Post by Cyril Brulebois

caused by the mke2fs in e2fsprogs/unstable ...

Just to be clear: mke2fs from e2fsprogs-udeb/unstable

Yes, sorry, I meant src:e2fsprogs but the relevant binary package is
indeed e2fsprogs-udeb.

Post by Cyril Brulebois

In that case this report should probably be reassigned to e2fsprogs,
as a request to stop enabling orphan_file by default until at least the
time that e2fsprogs (>= 1.47.0) has reached testing (but perhaps lower
risk to delay enabling it by default until post-bookworm).

Doing so now.

Post by Cyril Brulebois
https://bugs.debian.org/1031325#202

That's potentially somewhat orthogonal. If I understand it correctly,
#1031325 is mainly about the incompat feature "metadata_csum_seed" which
is unsupported by bullseye's grub, and because it's flagged as an incompat
feature, bullseye's grub assumes that it's invalid to do anything with
a filesystem with an incompat feature that it doesn't understand.

Meanwhile #1031622 is about a compat feature "orphan_file", which doesn't
need to be supported by the installed grub or kernel (because unknown
compat features are safe to ignore), but does need to be supported by the
installed e2fsck (because as a tool with in-depth ext[234] knowledge,
e2fsck is justifiably stricter about what filesystems it is willing to
work with).

I think the minimal-risk solution here is for bookworm's src:e2fsprogs to
revert the addition of both metadata_csum_seed and orphan_file to the
mke2fs defaults, so that filesystems are not created with those features
by default until trixie.

smcv