[[e-mail]: testingcds 3sidamd64 (amd64 dvd) has failed; log included]

[Steve McIntyre]

Hey Raphaël!

This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

----- Forwarded message from cdimage creator role <debian-***@casulana.debian.org> -----

From: cdimage creator role <debian-***@casulana.debian.org>
To: ***@einval.com
Date: Mon, 08 Aug 2022 18:29:33 +0000
Subject: testingcds 3sidamd64 (amd64 dvd) has failed; log included
Message-Id: <E1oL7VN-002UJb-***@casulana.debian.org>

Error when trying to create cds for 3sidamd64
Mon Aug 8 03:20:26 UTC 2022: Using CONF from /home/debian-cd/build.bookworm/CONF.sh.
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
make[1]: Entering directory '/srv/cdbuilder.debian.org/git/setup/bookworm/debian-cd'
rm -rf /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/CD[1234567890]*
rm -rf /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/tasks
rm -f /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/*.filelist*
rm -f /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/packages-stamp /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/upgrade-stamp /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/checksum-check
echo "Cleaning the build directory"
Cleaning the build directory
rm -rf /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/apt
rm -rf /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64
make[1]: Leaving directory '/srv/cdbuilder.debian.org/git/setup/bookworm/debian-cd'
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
make[1]: Entering directory '/srv/cdbuilder.debian.org/git/setup/bookworm/debian-cd'
mkdir -p /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64
mkdir -p /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm
mkdir -p /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/apt

...

Generating the complete list of packages to be included in /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list...
perl -ne 'chomp; next if /^\s*$/; \
print "$_\n" if not $seen{$_}; $seen{$_}++;' \
/srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/rawlist \

/srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list

/home/debian-cd/build.bookworm/debian-cd/tools/check_backports_packages /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list.backports
if [ -f /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list.backports ]; then mv /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list.backports /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list; fi
for ARCH in amd64; do \
ARCH=$ARCH /home/debian-cd/build.bookworm/debian-cd/tools/sort_deps /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list; \
done
Running sort_deps to sort packages for amd64:
Generating dependency tree with apt-cache depends...
Adding standard, required, important and base packages first
S/R/I/B packages take 124138086 bytes
Adding the rest of the requested packages
Deep recursion on subroutine "main::add_missing" at /home/debian-cd/build.bookworm/debian-cd/tools/sort_deps line 1139, <LIST> line 52916.
Deep recursion on subroutine "main::add_missing" at /home/debian-cd/build.bookworm/debian-cd/tools/sort_deps line 1142, <LIST> line 52916.
Killed
make[1]: *** [Makefile:484: image-trees] Error 137
make[1]: Leaving directory '/srv/cdbuilder.debian.org/git/setup/bookworm/debian-cd'


----- End forwarded message -----

[Steve McIntyre]

Post by Steve McIntyre
Hey Raphaël!
This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Checking on the build machine just now, I could see that a number of
sort_deps processes were chewing up ~40G of memory each. I'm going to
revert your commit for now and re-trigger the weekly build.

[Luna Jernberg]

Seems updated images are there now, thanks Sledge :)

Post by Steve McIntyre

Hey Raphaėl!
This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Checking on the build machine just now, I could see that a number of
sort_deps processes were chewing up ~40G of memory each. I'm going to
revert your commit for now and re-trigger the weekly build.
--
Steve McIntyre, Cambridge, UK.
"I used to be the first kid on the block wanting a cranial implant,
now I want to be the first with a cranial firewall. " -- Charlie Stross

[Steve McIntyre]

Post by Steve McIntyre
Hey Raphaël!
This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Sorry, I meant 8513b237afbdb59a4a59d1bc707d37f0aa9138f7 of course.

[Raphael Hertzog]

Hi Steve,

Post by Steve McIntyre
This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Ah, sure, that seems quite likely. I did not have the issue in my
tests, but I tested with a partial mirror as created by simple-cdd so I
quite likely did not hit the problematic recursive dependency.

Can you point me to the debian-cd configuration that I shall use to reproduce
the issue?

Cheers,

[Steve McIntyre]

Hey Raphael

Post by Raphael Hertzog

Post by Steve McIntyre
This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Ah, sure, that seems quite likely. I did not have the issue in my
tests, but I tested with a partial mirror as created by simple-cdd so I
quite likely did not hit the problematic recursive dependency.
Can you point me to the debian-cd configuration that I shall use to reproduce
the issue?

It's the weekly config that builds all the full media sets. The DVD
build on its own will show it:

#### B_amd64_dvd config start: ####
#### amd64.cfg start ####
export ARCH=amd64
export VARIANTS=xen
export MAXISOS=1
export MAXJIGDOS=ALL
export MAX_PKG_SIZE=999999999999
export DESKTOP=all
export INSTALLER_CD=3
export KERNEL_PARAMS="desktop=all"
export TASK=Debian-all
export DRYRUN_WAIT=0.5
export FORCE_CD_SIZE1=STICK4GB;;
#### dvd.cfg end ####
#### B_amd64_dvd config end: ####

[Raphael Hertzog]

Hello,

Post by Raphael Hertzog
Ah, sure, that seems quite likely. I did not have the issue in my
tests, but I tested with a partial mirror as created by simple-cdd so I
quite likely did not hit the problematic recursive dependency.
Can you point me to the debian-cd configuration that I shall use to reproduce
the issue?

So the issue was not trivial to reproduce. FWIW, it's only reproducible in
bullseye, for some reason buster was coping fine with it.

But I got it reproduced and the problem was due to packages which have
strong "Recommends" like libapreq2-dev which has "Recommends:
libapreq2-doc (= 2.13-7+b3)" which is not satisfiable. It would never stop
its recursion because the version found was not good for it.

So in the end I have a new patch in the hertzog/bug601203 branch. I'm
doing some further tests but it seems to solve the issue:
https://salsa.debian.org/images-team/debian-cd/-/commit/ca9ac8deac5c1436f4b311a22a34a56f236dfe05

But as I investigated I found more things to fix, like quite
some dead code:
https://salsa.debian.org/images-team/debian-cd/-/commit/e77ade6033445571092e8663ad94a24c7e882b03

I'm going to push this soon but I would love if someone else could
do a test run and ensure it doesn't break anything else.

I have created a merge request to make it easier to review the code if
anyone wants to do it:
https://salsa.debian.org/images-team/debian-cd/-/merge_requests/26

Cheers,

[Steve McIntyre]

Hey Raphael!

Post by Raphael Hertzog
Hello,

Post by Raphael Hertzog
Ah, sure, that seems quite likely. I did not have the issue in my
tests, but I tested with a partial mirror as created by simple-cdd so I
quite likely did not hit the problematic recursive dependency.
Can you point me to the debian-cd configuration that I shall use to reproduce
the issue?

So the issue was not trivial to reproduce. FWIW, it's only reproducible in
bullseye, for some reason buster was coping fine with it.

Hmmm, OK.

Post by Raphael Hertzog
But I got it reproduced and the problem was due to packages which have
libapreq2-doc (= 2.13-7+b3)" which is not satisfiable. It would never stop
its recursion because the version found was not good for it.

Argh.

Post by Raphael Hertzog
So in the end I have a new patch in the hertzog/bug601203 branch. I'm
https://salsa.debian.org/images-team/debian-cd/-/commit/ca9ac8deac5c1436f4b311a22a34a56f236dfe05
But as I investigated I found more things to fix, like quite
https://salsa.debian.org/images-team/debian-cd/-/commit/e77ade6033445571092e8663ad94a24c7e882b03
I'm going to push this soon but I would love if someone else could
do a test run and ensure it doesn't break anything else.
I have created a merge request to make it easier to review the code if
https://salsa.debian.org/images-team/debian-cd/-/merge_requests/26

I can have a look, but not *right* now I'm afraid - swamped with other
stuff. Hopefully by the end of the week...