missing files from different formats of the .iso files

[griffin tucker]

here's the truth table for debian-11.6.0 source .iso:

present: o
missing: x

DLBD BD DVD file
--- --- --- ---
o x x pepperflashplugin-nonfree_1.8.8.dsc
x o x frogatto_1.3.1+dfsg-5.dsc
x o x openjazz_20190106-3.dsc
x o x torbrowser-launcher_0.3.3-6.dsc
o o x alien-arena_7.66+dfsg-6.dsc
x x o cbedic_4.0-4.diff.gz
x x o crafty-bitmaps_1.0-1.debian.tar.gz
x x o esix_1-3.dsc
x x o freespace2-launcher-wxlauncher_0.11.0+dfsg-3.debian.tar.xz
x x o game-data-packager_67.dsc
x x o hts-voice-nitech-jp-atr503-m001_1.05-5.dsc
x x o lutris_0.5.8.3-2.dsc
x x o magma_2.5.4+ds-3.debian.tar.xz
x x o nvidia-settings-legacy-390xx_390.144-1.dsc
x x o openjazz_20190106-3.dsc
x x o publicfile-installer_0.15.dsc
x x o r4d_1.7-1.debian.tar.xz
x x o spectemu_0.94a-20.dsc
x x o thunar-dropbox-plugin_0.3.1-1.dsc
x x o torbrowser-launcher_0.3.3-6.dsc
x x o vmware-manager_0.2.0-4.dsc

is there a reason for these few files being missing, depending on
which format the .iso is in?

i can't help but feel i'm missing something obvious for why the
formats slightly differ in content

[griffin tucker]

i figured this out. i think.

there's a bug in jigdo that makes it occasionally skip files from the
source and then it will request the online source. i'm not sure why.

obviously there's no missing files depending on the format, which is why i
didn't get a response from anyone.

On Fri, 21 Apr 2023 at 19:29, griffin tucker <

Post by griffin tucker
present: o
missing: x
DLBD BD DVD file
--- --- --- ---
o x x pepperflashplugin-nonfree_1.8.8.dsc
x o x frogatto_1.3.1+dfsg-5.dsc
x o x openjazz_20190106-3.dsc
x o x torbrowser-launcher_0.3.3-6.dsc
o o x alien-arena_7.66+dfsg-6.dsc
x x o cbedic_4.0-4.diff.gz
x x o crafty-bitmaps_1.0-1.debian.tar.gz
x x o esix_1-3.dsc
x x o freespace2-launcher-wxlauncher_0.11.0+dfsg-3.debian.tar.xz
x x o game-data-packager_67.dsc
x x o hts-voice-nitech-jp-atr503-m001_1.05-5.dsc
x x o lutris_0.5.8.3-2.dsc
x x o magma_2.5.4+ds-3.debian.tar.xz
x x o nvidia-settings-legacy-390xx_390.144-1.dsc
x x o openjazz_20190106-3.dsc
x x o publicfile-installer_0.15.dsc
x x o r4d_1.7-1.debian.tar.xz
x x o spectemu_0.94a-20.dsc
x x o thunar-dropbox-plugin_0.3.1-1.dsc
x x o torbrowser-launcher_0.3.3-6.dsc
x x o vmware-manager_0.2.0-4.dsc
is there a reason for these few files being missing, depending on
which format the .iso is in?
i can't help but feel i'm missing something obvious for why the
formats slightly differ in content

[Thomas Schmitt]

Hi,

Post by griffin tucker
there's a bug in jigdo that makes it occasionally skip files

In that case the missing files woule be present but filled with zeros
when you mount the incompletely composed ISO.
Can you confirm ?


Not picking a file of a jigdo image may have various reasons.
Temporary download problems should become visible by error messages.
Consider to catch the output of jigdo-lite in a file for inspection.

Persistent problems could be (possibly among others):

- File is missing on the chosen Debian mirror.

In this case it is supposed to be fetched from fallback servers which
are mentioned at the end of the .jigdo file:

$ gunzip <debian-11.6.0-source-DVD-1.jigdo | tail -5
T__clXB4rmLIbFvoblKNqA=Debian:pool/main/b/bin-prot/bin-prot_0.14.0-1.debian.tar.xz

[Servers]
Debian=http://us.cdimage.debian.org/cdimage/snapshot/Debian/
Debian=http://snapshot.debian.org/archive/debian/20221218T124218Z/ --try-last

Assuming that bin-prot_0.14.0-1.debian.tar.xz would be missing, you
could check by a web browser whether
http://us.cdimage.debian.org/cdimage/snapshot/Debian/pool/main/b/bin-prot/bin-prot_0.14.0-1.debian.tar.xz
exists.


- The checksum of the file in .jidgo does not match the checksum of the
file on the mirror server.

The checksum is listed in .jigdo before the '=', encoded in a variant of
base64.
For decoding append enough "=" to get a number of characters which is
divisible by 4, replace "-" and "_" by their base64 equivalents, decode
base64, format to hex, and remove od's decorations:

$ echo 'T__clXB4rmLIbFvoblKNqA'== | sed -e 's/-/+/g' -e 's/_/\//g' | base64 -d | od -t x1 | sed -e 's/^.......//' -e 's/ //g'
4fffdc957078ae62c86c5be86e528da8

In this case the checksum is MD5. Expect to see SHA256 in newer .jigdo.

Post by griffin tucker
from the source
and then it will request the online source. i'm not sure why.

What is the difference between "the source" and "the online source" ?
I'd normally expect that all jigdo sources of files are online.
Do you have your own local Debian mirror or a set of older ISOs ?


Have a nice day :)

Thomas

[griffin tucker]

hi,

Post by Thomas Schmitt
Hi,

Post by griffin tucker
there's a bug in jigdo that makes it occasionally skip files

In that case the missing files woule be present but filled with zeros
when you mount the incompletely composed ISO.
Can you confirm ?

sorry, i must've misexplained; the .iso's are made from the same
version, but a different format, (eg. between DVD, BD, and DLBD)

jigdo-lite runs in batch mode, and usually the reconstruction of the
first .iso is completely reconstructed from a directory of mounted
.iso's from another format, however, the next few .iso's will
consistently have 1 file missing that it couldn't find from the
mounted .iso's, when it should, and i am unable to determine the
pattern or reason for this

jigdo-lite will write everything except that missing file which is
written with 0's, then request an online source, which it does
successfully, then it fills in the 0's and verifies the md5 also
successfully

there is no corruption to the reconstructed .iso's.

Post by Thomas Schmitt
Not picking a file of a jigdo image may have various reasons.
Temporary download problems should become visible by error messages.
Consider to catch the output of jigdo-lite in a file for inspection.
- File is missing on the chosen Debian mirror.
In this case it is supposed to be fetched from fallback servers which
$ gunzip <debian-11.6.0-source-DVD-1.jigdo | tail -5
T__clXB4rmLIbFvoblKNqA=Debian:pool/main/b/bin-prot/bin-prot_0.14.0-1.debian.tar.xz
[Servers]
Debian=http://us.cdimage.debian.org/cdimage/snapshot/Debian/
Debian=http://snapshot.debian.org/archive/debian/20221218T124218Z/ --try-last
Assuming that bin-prot_0.14.0-1.debian.tar.xz would be missing, you
could check by a web browser whether
http://us.cdimage.debian.org/cdimage/snapshot/Debian/pool/main/b/bin-prot/bin-prot_0.14.0-1.debian.tar.xz
exists.
- The checksum of the file in .jidgo does not match the checksum of the
file on the mirror server.
The checksum is listed in .jigdo before the '=', encoded in a variant of
base64.
For decoding append enough "=" to get a number of characters which is
divisible by 4, replace "-" and "_" by their base64 equivalents, decode
$ echo 'T__clXB4rmLIbFvoblKNqA'== | sed -e 's/-/+/g' -e 's/_/\//g' | base64 -d | od -t x1 | sed -e 's/^.......//' -e 's/ //g'
4fffdc957078ae62c86c5be86e528da8
In this case the checksum is MD5. Expect to see SHA256 in newer .jigdo.

i look forward to the sha256 or sha512 implementation

Post by Thomas Schmitt

Post by griffin tucker
from the source
and then it will request the online source. i'm not sure why.

What is the difference between "the source" and "the online source" ?
I'd normally expect that all jigdo sources of files are online.
Do you have your own local Debian mirror or a set of older ISOs ?

no local mirror, just a set of the same version of debian in a
different format, as explained above

i tried with v12.0.0 source this time, and got similar results

i'm running it in a virtual machine, if that makes a difference
(afaik, it shouldn't)

Post by Thomas Schmitt
Have a nice day :)

likewise

Post by Thomas Schmitt
Thomas

[Thomas Schmitt]

Hi,

Post by griffin tucker
jigdo-lite runs in batch mode, and usually the reconstruction of the
first .iso is completely reconstructed from a directory of mounted
.iso's from another format, however, the next few .iso's will
consistently have 1 file missing that it couldn't find from the
mounted .iso's, when it should, and i am unable to determine the
pattern or reason for this

Could be a problem of jigdo-lite with this special file presentation.

Does the missing file have a particular position in the .jigdo file
which you use for composing the new ISO ?
gunzip < *.jigdo | less

Does it exist in the mounted ISOs and does its checksum match ?
echo '...char.salad.from.jigdo...'== | sed -e 's/-/+/g' -e 's/_/\//g' |
base64 -d | od -t x1 | sed -e 's/^.......//' -e 's/ //g'

Post by griffin tucker

Post by Thomas Schmitt
In this case the checksum is MD5. Expect to see SHA256 in newer .jigdo.

i look forward to the sha256 or sha512 implementation

In this case, the MD5 is not used as verifier but as access key.
When it gets found in the .template file it is looked up in the .jigdo
to get the download URL of the file.
Verification should be done by the overall checksum of the ISO as
published in the SHA256SUMS or SHA512SUMS files which accompany the .jigdo
and .template files.

So i don't see much reason to switch internally to SHA256.
But MD5 has a bad reputation in Debian and so the Jigdo format was
enhanced to use SHA256 instead. Interestingly, even
debian-12.0.0-amd64-netinst.jigdo still uses the shorter MD5s.
(xorriso-1.5.4 in Debian 12 is capable of producing SHA256 in the .jigdo
files. The development version xorriso-1.5.3 can do it since end of 2019.)

Post by griffin tucker
i'm running it in a virtual machine, if that makes a difference
(afaik, it shouldn't)

That would be indeed astonishing. I rather bet on the mounted ISOs as
trigger. (But i have no stake in jigdo reconstruction, only in its
production as developer of xorriso.)


---------------------------------------------------------------------
Off topic:

I wonder what those unusual headers with changing names in your mails
mean:
X-Coil-Desert: 4e292ddc276fab1b_1687051700717_2547565298
X-Obese-Wide-Eyed: 37ca251540d1f045_1687078216293_605541665
X-Ruddy-Name: 05f4b58c06a9aab5_1687078216565_3930339659
always between
X-MailChannels-Auth-Id: ...
and
X-MC-Loop-Signature: ...


Have a nice day :)

Thomas

[Thomas Schmitt]

Hi,

(Possibly your attachments did not make it to the mailing list.
I got only the mail which was sent directly to me.)

img-debian-12.0.0-source-bd_to_dvd_1_of_2-77429.asc
...
Found 2840 of the 2840 files required by the template
Successfully created `debian-12.0.0-source-DVD-1.iso'
...
Found 3646 of the 3647 files required by the template
Copied input files to temporary file `debian-12.0.0-source-DVD-2.iso.tmp'
...
Found 3907 of the 3908 files required by the template
Copied input files to temporary file `debian-12.0.0-source-DVD-3.iso.tmp'
...
Found 3482 of the 3483 files required by the template
Found 1268 of the 1269 files required by the template
Found 2224 of the 2225 files required by the template
Found 13101 of the 13102 files required by the template
Found 8045 of the 8046 files required by the template
Found 14341 of the 14342 files required by the template

This looks highly systematic and not like caused by some glitchy files in
the source pool.
I see the same failures in all second and further ISOs of a jigdo-lite
run. It seems not to be coupled to DVD or BD numbers higher than 1 but
only to the sequence of ISOs in a single jigdo-lite run.

Nevertheless, in the possibly incomplete log file
img-debian-12.0.0-source-dvd_to_dlbd-42245.asc i see already a missing
file with the debian-12.0.0-source-DLBD-1.iso file, which i would expect
to have been first. Was there another download before this in the same
jigdo-lite run ?

Consider to file a bug report towards package "jigdo-file".


-----------------------------------------------------------------------

Post by Thomas Schmitt
X-Coil-Desert: 4e292ddc276fab1b_1687051700717_2547565298
X-Obese-Wide-Eyed: 37ca251540d1f045_1687078216293_605541665
X-Ruddy-Name: 05f4b58c06a9aab5_1687078216565_3930339659

nfi, tbh.
i'm not a sysadmin and i wouldn't be surprised if it's some sort of
reference to an inside joke to those that monitor my communications.

Obviously they have room for such shenanigans but not for
X-Clacks-Overhead: GNU Terry Pratchett


Have a nice day :)

Thomas

[Thomas Schmitt]

Hi,

there is already a bug report about a similar problem

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988120

It points to file
jigdo-file-cache.db
and mentions a positive one-time effect, which would match the observation
that the first medium nearly always succeeds.
It works if jigdo-file-cache.db is deleted before the jigdo-lite run.

If you do so at the beginning of your script, then this would nicely
explain why further ISOs fail and what is their connection at all.
(I began to wonder how first and further ISOs are different to jigdo-lite.)


The bug report mentions SHA256 as checksums in the .jigdo file, which
i only see for the overall image but not for the single data files.
But maybe it is about SHA256 in the jigdo-file-cache.db.

This comment proposes a patch:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988120#20
and gives a code analysis.


Have a nice day :)

Thomas