Bug#1056998: cdrom: Installation media changes after booting it

[Ram Reddy]

Package: cdrom
Severity: normal
X-Debbugs-Cc: ***@gmail.com

Dear Maintainer,
I have downloaded a debian 12.2.0 netinst and verified its integrity with
sha512sum -c --ignore-missing SHA512SUMS
and wrote it to a usb drive with
dd bs=2048 if=debian-12.2.0-amd64-netinst.iso of=/dev/sda oflag=sync
status=progress

I also verified that the contents were copied correctly with these two
commands:
cmp -n `stat -c '%s' debian-12.2.0-amd64-netinst.iso`
debian-12.2.0-amd64-netinst.iso /dev/sda

head -c $(stat -c '%s' debian-12.2.0-amd64-netinst.iso) /dev/sda | sha512sum

which returned the same checksum. I also safely removed the drive, unplugged
it,
and plugged it back in, and used those two commands again with the same output.
(to make sure the kernel buffer wasn't in use)

After booting the Installer from that drive, however, and navigating to the
Integrity Check section of the installer from its menu, it said that
./boot/grub/efi.img failed the MD5 checksum verification. Your installation
media or this file may have been corrupted.
After this, I chose to quit the installer and check the medias integrity again
with the earlier two commands. sha512sum showed a different checksum, and cmp
showed
debian-12.2.0-amd64-netinst.iso /dev/sda differ: byte 2303211, line 21165
showing that the content of the drive changed since it previously passed.

I also tried entering the installer, and aborting the installation without
using the
Integrity Test section. I did this because I thought that maybe mounting the
media
in the Integrity Test section of the installer somehow changed the data.
However,
after booting, cmp and sha512sum still showed the same
difference.

I also tried using ddrescue and verified that there were 0 issues with the
blocks
on the media, used a different usb drive, and tried 3 laptops. In all of these
cases
it was always byte 2303211, line 21165 which changed. To my knowledge, I don't
think
this is supposed to happen. I instead expected an outcome where the media would
stay
the same throughout the installation process. Lastly, the computer I was using
was able to get 10 passes in memtest without any errors.

Have a nice day, Ram

[Thomas Schmitt]

Hi,

[...] usb drive [...] debian-12.2.0-amd64-netinst.iso [...]
[...] Integrity Test section [...]
./boot/grub/efi.img failed the MD5 checksum verification. Your installation
media or this file may have been corrupted.
[...]
It was always byte 2303211, line 21165 which changed.

I'd say that corruption of ISO image or USB is quite unlikely.

It rather looks like some change was made to the FAT filesystem in the
EFI partition by the parts of the installer which ran in your experiments.
(I expect EFI firmware not to manipulate that filesystem ...)
The EFI partition of this particular ISO begins at byte 2291712. The
alteration thus happened 11499 bytes inside that partition and its
filesystem. In the original ISO i see a sequence of 0-bytes there.

It would be interesting to check whether any errors get reported if the
ISO is presented on a read-only CD or DVD instead of a writable USB stick.
(To be absolutely read-only it would have to be CD-R, unformatted CD-RW,
DVD+R, DVD-R, or unformatted DVD-RW. The others can be overwritten without
burn program.)

Explanation why the file in the read-only ISO filesystem is reported as
being altered:
The partition shares its block range with the ISO file /boot/grub/efi.img
which is mentioned in the ISO file /md5sum.txt . So the integrity check
detects the difference between recorded MD5 and the MD5 of the altered
FAT filesystem image file /boot/grub/efi.img.


Have a nice day :)

Thomas

[Ram Reddy]

Hello,

Post by Thomas Schmitt
I'd say that corruption of ISO image or USB is quite unlikely.
[ . . . ]
It would be interesting to check whether any errors get reported if the
ISO is presented on a read-only CD or DVD instead of a writable USB stick.

Thank you for the suggestion for trying a CD / DVD read only installation
media. Unfortunately I don't have one at hand, and there are none in
the stores here. The only option seems to be to order one online, which will
take a few days to ship.

When I get the disks, I'll test using the installer with them.
Thanks for the help!

Have a great day;
Ram

[Pete Batard]

Hi Ram,

In case you plugged your media at any time on a Windows system, please
be aware that Windows automatically creates a 'System Volume
Information/' directory, along with a bunch of small files, on any file
system it can handle, which would include the FAT16/FAT32 file systems
typically used by an ESP.

The end result of this is that, for media such as Debian that includes
an ESP in the form of an 'efi.img', validation checks of that efi.img
file are likely to fail if the media was ever plugged on Windows...

Now, I see no clear hint from your testing that you used any Windows
host in the process. But I thought I would mention that just in case.

Regards,

/Pete

Post by Ram Reddy
Hello,

Post by Thomas Schmitt
I'd say that corruption of ISO image or USB is quite unlikely.
[ . . . ]
It would be interesting to check whether any errors get reported if the
ISO is presented on a read-only CD or DVD instead of a writable USB

stick.
Thank you for the suggestion for trying a CD / DVD read only installation
media. Unfortunately I don't have one at hand, and there are none in
the stores here. The only option seems to be to order one online, which will
take a few days to ship.
When I get the disks, I'll test using the installer with them.
Thanks for the help!
Have a great day;
Ram

[Ram Reddy]

Hello,

Post by Thomas Schmitt
It would be interesting to check whether any errors get reported if the
ISO is presented on a read-only CD or DVD instead of a writable USB stick.

Hello, thank you for the help. The DVD+Rs arrived, along with the DVD
Drive. I tested the installer on one of my laptops, and found that its
contents didn't change. (which is to be expected, as DVD+Rs can't be
rewritten) I didn't bother testing that on different computers, because the
result would be the same. However, I have tested the USB drive on a few
other computers and here are my findings:

On the Lenovo Legion 7i Gen 5*, Lenovo Thinkpad X1 Carbon Gen 2*, Lenovo
Thinkpad X1 Carbon Gen 5 Intel*, and one Lenovo Thinkpad E14 Gen 5 all
showed the corruption error.
*These laptops have been tested for installer corruption twice (a week ago,
and just now)

However, on the Lenovo Yoga C740 and a different Lenovo Thinkpad E14 Gen 5
Intel showed no corruption issues.

Both the Thinkpad E14 Gen 5s had the same specifications and type number,
differing only in that the one with corruption of the installer has 24GB of
memory (16GB installed in the slot, 8GB soldered) and the other only has
8GB soldered. They both have the same BIOS version, R2AET32W(1.07). Again,
the ones that had corruption all had it at the same location. (byte
2303211, line 21165)

This seems to be really interesting because the corruption only happened on
certain computers, and it would stay that way on repeated attempts. Thank
you for the help.

Have a wonderful day,
Ram

[Thomas Schmitt]

Hi,

I tested the installer on one of my laptops, and found that its contents
didn't change.

Good to know that the software in the ISO still works from read-only DVDs.

On the Lenovo Legion 7i Gen 5*, Lenovo Thinkpad X1 Carbon Gen 2*, Lenovo
Thinkpad X1 Carbon Gen 5 Intel*, and one Lenovo Thinkpad E14 Gen 5 all
showed the corruption error. 
However, on the Lenovo Yoga C740 and a different Lenovo Thinkpad E14 Gen 5
Intel showed no corruption issues. 

This seems to indicate that the firmware has a stake in the problem ...

Both the Thinkpad E14 Gen 5s had the same specifications and type number,
differing only in that the one with corruption of the installer has 24GB of
memory (16GB installed in the slot, 8GB soldered) and the other only has 8GB
soldered. They both have the same BIOS version, R2AET32W(1.07).

... but the trigger would have to be very subtle.

This seems to be really interesting because the corruption only happened on
certain computers, and it would stay that way on repeated attempts.

I ponder how to distinguish a change made by the firmware itself from a
change made by the software in the ISO image.

Again, the
ones that had corruption all had it at the same location. (byte 2303211,
line 21165) 

Maybe we can learn more by comparing the files /boot/grub/efi.img of the
original ISO and of an altered USB stick.

Mount both and copy out the files, or use dd to extract them from the
partition interval that is marked in the ISO (ask by: fdisk -l).
With debian-12.2.0-amd64-netinst.iso that would be

iso=debian-12.2.0-amd64-netinst.iso

or (with the approriate letter instead of "X")

iso=/dev/sdX

and then

dd bs=512 if="$iso" of="$iso".esp skip=4476 count=18976

Both images will have about 9.5 MiB of size. I would be interested in
seeing the altered one, as i can extract the original on my own.
So if you have a way to upload the altered EFI partition image to a
publicly accessible place i would be happy to investigate it.

Much smaller and thus more bugreport-friendly would be a binary diff.
You would need something more detailed than plain cmp. cmp -l gives the
differing byte values one by line (i.e. lengthy if many bytes differ).
I made an own willful alteration (subtracting 4476 * 512 from 2303211 yields
the new problem byte address 11499):

cp debian-12.2.0-amd64-netinst.iso.esp debian-12.2.0-amd64-netinst.iso.esp.altered
echo ABCDE | dd conv=notrunc of=debian-12.2.0-amd64-netinst.iso.esp.altered bs=1 seek=11499 count=5

Then

cmp -l debian-12.2.0-amd64-netinst.iso.esp debian-12.2.0-amd64-netinst.iso.esp.altered

yields

11500 0 101
11501 0 102
11502 0 103
11503 0 104
11504 0 105

(Odd enough: Addresses are decimal starting at 1, not at 0. Values are
octal.)

Less useful is cmp -b:

debian-12.2.0-amd64-netinst.iso.esp debian-12.2.0-amd64-netinst.iso.esp.altered differ: byte 11500, line 278 is 0 ^@ 101 A

(Only the "A" is from my change. There seem to be pairs of octal value and
character meaning.)

I have my own unpublished differ for analyzing ISO entrails. So only the
big battleship "diffoscope" comes to my mind as public binary differ.
Probably there are leaner programs for that job.


Have a nice day :)

Thomas

[Nicholas Geovanis]

Post by Thomas Schmitt
.....
This seems to indicate that the firmware has a stake in the problem ...

Post by Ram Reddy
Both the Thinkpad E14 Gen 5s had the same specifications and type number,
differing only in that the one with corruption of the installer has 24GB

of

Post by Ram Reddy
memory (16GB installed in the slot, 8GB soldered) and the other only has

8GB

Post by Ram Reddy
soldered. They both have the same BIOS version, R2AET32W(1.07).

... but the trigger would have to be very subtle.

Post by Ram Reddy
This seems to be really interesting because the corruption only happened

on

Post by Ram Reddy
certain computers, and it would stay that way on repeated attempts.

FWIW check the BIOS L[123] cache settings and consider changing them to
more conservative "slower" values if possible. And you have different RAM
models and configurations, could there be one DIMM in the mix that is
running overclocked?

Have a nice day :)

Post by Thomas Schmitt
Thomas

GrÌß Gott :-)

[Ram Reddy]

Hi,

Post by Thomas Schmitt
Maybe we can learn more by comparing the files /boot/grub/efi.img of the
original ISO and of an altered USB stick.
iso=/dev/sdX
and then
dd bs=512 if="$iso" of="$iso".esp skip=4476 count=18976

I tried doing that for the times the drive was modified, and they are here:
https://drive.google.com/file/d/1Zd6iufVRsfIu-qzC-tJx4FEvCOESOz4_/view?usp=sharing
Each modification was unique, so I put them all there. The files are for
the 3 most recent attempts I did. G = gen, and the number after - is the
attempt number. For the ones with no modifications, there's a text file
with unmodified in its name which contains "unmodified" in it.
unmodified.esp is the unmodified ESP in the installer.

However, I found some interesting results. I tried redoing the tests on
each laptop, and found that the modifications occurred somewhat randomly.
Laptop Model | Test 1 | Test 2 | Test 3 | Test 4 | Test 5
Lenovo Legion 7i gen 5 | modded | modded | modded | modded | modded
Thinkpad X1 Carbon gen 2 | modded | modded |unmodded|unmodded|unmodded
Thinkpad X1 Carbon gen 5 | modded | modded | modded |unmodded|unmodded
Thinkpad E14 Gen 5 Intel | modded |unmodded|unmodded|unmodded|unmodded
Thinkpad E14 Gen 5 Intel |unmodded|unmodded|unmodded|unmodded|unmodded
Lenovo Yoga C740 |unmodded|unmodded|unmodded|unmodded|unmodded
modded = modificationed
It did show some patterns, however.

Have an awe-inspiring day,
Ram

[Thomas Schmitt]

Hi,

Post by Ram Reddy
https://drive.google.com/file/d/1Zd6iufVRsfIu-qzC-tJx4FEvCOESOz4_/view?usp=sharing

I downloaded the tarball and compared the original FAT filesystem with the
various modified filesystem images.

--------------------------------------------------------------------------

In Legion7iG5-*_modified.esp the suspect lost its ID card at the crime
scene:
At byte 39072 (0x98a0) the changes go from 0-bytes to the text "LENOVO".
At byte 9711680 (0x943040) i see a change from 0-bytes to "BIOS".

Diffing the result of "find" on the mounted unmodified.esp filesystem and
Legion7iG5-*_modified.esp shows that a new branch of directoriies with a

Post by Ram Reddy
./efi/Lenovo
./efi/Lenovo/BIOS
./efi/Lenovo/BIOS/SelfHealing.fd

The file is empty.

--------------------------------------------------------------------------

In ThinkpadX1CarbonG5-0_modified.esp there is no company name to see in
the changed bytes. I see UTF-16 strings "mation", "System", and
"Volum\000me". ASCII texts "SYSTEM~1", "WPSETT~1DAT". The latter might
possibly be "WPSettings.dat", which causes questions in the internet.
Most plausible seems an answer in the course of
https://answers.microsoft.com/en-us/insider/forum/all/whats-wpsettingsdat-generated-by/e11bca97-8c76-4662-8897-774ea3d5691a
"The WPSettings.dat file is generated by the Storage Service (StorSvc).
It seems that WPSettings.dat means the data files of Windows Phone's
Store Settings saved on the drives, [...]"

Diffing the result of "find" on the mounted unmodified.esp filesystem and
ThinkpadX1CarbonG5-0_modified.esp shows that a new directory with a new
file is in the modified filesystem:
./System Volume Information
./System Volume Information/WPSettings.dat
The file has 12 bytes of binary salad:
Hex: 0c 00 00 00 2e 42 6b 82 5d 88 0e c5
Char: . B k ]
Dec: 12 0 0 0 46 66 107 130 93 136 14 197

--------------------------------------------------------------------------

While it makes some sense to me that Lenovo Legion BIOS adds some Lenovo
stuff to the EFI System Partition, i really wonder why Lenovo Thinkpad
BIOS adds a Microsoft directory and file.

Whatever, i'd say that the software in the ISO and especially Debian
Installer are not suspicious to create directories with such names.


Have a nice day :)

Thomas

[Ram Reddy]

Hello,

Thomas Schmitt wrote:FWIW check the BIOS L[123] cache settings and
consider changing them to

Post by Nicholas Geovanis
more conservative "slower" values if possible. And you have different RAM

For changing the BIOS L[1/2/3] cache settings, these are laptop BIOSes
which usually have very few features (other than secure boot/tpm
options which are required for laptops with Windows), so there was no
option for that in the BIOSes. However, when I get home I could try
changing the cache settings on the Legion 7i gen 5, as it has a key
combination in the BIOS to enter an "Advanced Mode" with overclocking
settings and more, which likely has cache settings.

Post by Nicholas Geovanis
models and configurations, could there be one DIMM in the mix that is >

running overclocked?

None of the memory modules were overclocked, and they were all
operating at stock speeds. The Legion 7i gen 5 got 10 passes in
memtest with 0 errors, and both of the Thinkpad X1 Carbon laptops, and
the Yoga C740, had fully soldered memory. One of the Thinkpad E14 gen
5 laptops also had unchanged memory. The laptops without any changed
memory also showed similar results, so I doubt that it has anything to
do with memory.

Have an exceptionally great day,
Ram

[Thomas Schmitt]

Hi,

FWIW check the BIOS L[123] cache settings [...]

This was posted by Nicholas Geovanis <***@gmail.com>, not by me.

My suspicion is that the Lenovo Legion EFI BIOS intentionally creates
directories and empty file "/efi/Lenovo/BIOS/SelfHealing.fd".
I do not see much benefit in investigating this further.

More riddling is the directory and file found after using the USB stick
on the Lenovo Thinkpad machines. The internet says that
"/System Volume Information/WPSettings.dat" is part of Microsoft services.
So it might be that the USB stick got in contact with MS-Windows while
or before it was plugged into the Thinkpads.
If this suspicion can be excluded then i'd say that Thinkpad has some
strange BIOS habits.

What remains for Debian in this bug report is the observation that the
integrity check of the Debian installation ISO says "Your installation
media or this file may have been corrupted" when just a harmless change
has been made in the filesystem of the EFI partition.
Some special handling of /boot/grub/efi.img during the integrity check
would cause less surprise to the user.
Like:

"Possibly some software or firmware made changes in the EFI System
Partition. This may be considered harmless unless booting of this
Debian ISO encounters difficulties or other files show checksum
mismatches, too."


Have a nice day :)

Thomas

[Ram Reddy]

Hi,

Post by Thomas Schmitt
At byte 39072 (0x98a0) the changes go from 0-bytes to the text "LENOVO".
At byte 9711680 (0x943040) i see a change from 0-bytes to "BIOS".
[...]
Diffing the result of "find" on the mounted unmodified.esp filesystem and
Legion7iG5-*_modified.esp shows that a new branch of directoriies with a

Post by Ram Reddy
./efi/Lenovo
./efi/Lenovo/BIOS
./efi/Lenovo/BIOS/SelfHealing.fd

The file is empty.

This seems to make a lot of sense, since for the Legion 7i gen 5
whenever I started the system on the USB drive, it showed text
Self Healing BIOS Backup Progressing... __%
Or something like that. I probably should have said something about
that earlier, but never thought about it. Sorry.

Post by Thomas Schmitt
In ThinkpadX1CarbonG5-0_modified.esp there is no company name to see in
the changed bytes. I see UTF-16 strings "mation", "System", and
"Volum\000me". ASCII texts "SYSTEM~1", "WPSETT~1DAT". The latter might
possibly be "WPSettings.dat", which causes questions in the internet.

I think that this could possibly be explained by flaws in my testing.
Some of the computers were using Windows, and for those computers I
plugged in the USB drive, shut down the computer, and booted the USB
drive from the BIOS. In the time the USB drive was connected to the
computer when Windows was in use, it may have written files to it. I
realized this could be a problem towards the end of the testing, which
is why I started shutting down Windows first before testing the USB.
This can be shown in the bottom right results on the table, which are
the most recent ones, and the ones where I shut down windows first had
no modifications.

So it seems to be that the only modifications of the installer caused
by firmware is with the Legion 7i gen 5's self healing BIOS backup. I
don't think there is any problem with the debian installer itself, and
I don't think there's anything to change in it. When I get home, I'll
look for an option to disable the bios backup, and then install debian
on my computer. Thank you for your help.

Maybe this problem could be listed on the Debian Installer FAQ? I
think it could save some time for others finding it.

Have an awesome day,
Ram

[Thomas Schmitt]

Hi,

trying to get our mails in sync again. We seem to have reached consensus
in our understanding of the problem.

might possibly be "WPSettings.dat", which causes questions in the
internet.

I think that this could possibly be explained by flaws in my testing. Some
of the computers were using Windows, and for those computers I plugged in
the USB drive, shut down the computer, and booted the USB drive from the
BIOS.

This explains the MS-Windows file.

the ones where I shut down windows first had no modifications.

So the Thinkpad firmwares are exonerated.

it seems to be that the only modifications of the installer caused by
firmware is with the Legion 7i gen 5's self healing BIOS backup.

This suspect can be considered to have confessed.

--------------------------------------------------------------------------

As for a better message from Integrity Check:

I find in the ISO in file /install.amd/initrd.gz the lines:
"The file ${FILE} failed the MD5 checksum verification. Your installation"
"media or this file may have been corrupted."

But i failed to find these texts in debian-cd or debian-installer
Even searching for just "corrupted" yields only a remark in debian-cd.

Does any of the bystanders know from what package this message text stems ?


Have a nice day :)

Thomas

[Steve McIntyre]

Hey Thomas!

Post by Thomas Schmitt
"The file ${FILE} failed the MD5 checksum verification. Your installation"
"media or this file may have been corrupted."
But i failed to find these texts in debian-cd or debian-installer
Even searching for just "corrupted" yields only a remark in debian-cd.
Does any of the bystanders know from what package this message text stems ?

That's cdrom-checker: https://salsa.debian.org/installer-team/cdrom-checker

[Thomas Schmitt]

Hi,

Post by Thomas Schmitt
Does any of the bystanders know from what package this message text stems ?

So that would be indirectly put into the code at

https://sources.debian.org/src/cdrom-checker/1.62/main.c/?hl=152#L152

by "Template: cdrom-checker/mismatch" in:

https://sources.debian.org/src/cdrom-checker/1.62/debian/cdrom-checker.templates/?hl=43#L43

Using my micking instincts i guess my proposal would be to create a
new template like:

Template: cdrom-checker/mismatch-efi
Type: error
# :sl3:
_Description: Integrity test failed on EFI boot image
The EFI boot image file ${FILE} failed the MD5 checksum verification.
Possibly some software or firmware made changes in the EFI System
Partition. This may be considered harmless unless booting of this
Debian ISO encounters difficulties or other files show checksum
mismatches, too.

and to add a case in main.c, line 151, like (shown with reduced
indentation):

if(status != 0) {
if(strcmp(filename, "./boot/grub/efi.img") == 0) {
debconf_subst(debconf, "cdrom-checker/mismatch-efi", "FILE", filename);
debconf_input(debconf, "high", "cdrom-checker/mismatch-efi");
} else {
debconf_subst(debconf, "cdrom-checker/mismatch", "FILE", filename);
debconf_input(debconf, "high", "cdrom-checker/mismatch");
}
} else {
debconf_input(debconf, "critical", "cdrom-checker/passed");
}

A native speaker can probably find a better text and my guessing of the
inner code relations may be wrong, of course. Even the path
"./boot/grub/efi.img" is guessed from Ram Reddy's reported error message.


Have a nice day :)

Thomas