Hi Steve,
Post by Steve McIntyreIt looks like James (with some help from Thomas) has worked out a
quick way to change things to make things better for you, which is
good! (Thanks, guys! I'm about to test the change locally.)
Yes, I'm happy to see this development as well. And I have to stress
out, since this appears to be gist of your query, that mine is far from
a self centred goal, but something that aims at providing a *choice* to
people, when it comes to the manner in which they should create their
boot media, the reasons for which I will elaborate on hereafter.
Post by Steve McIntyre*However* you're the *only* person I've seen complaining about this
problem with "file system transposition".
I will venture that this is because, and this is not criticism per se,
most distributions currently make that process quite difficult or
fraught with so many hurdles that people who would want to use FST will
not yet bother trying it. For one thing, outside of Debian, most
distributions still seem to be relying on labels to locate the
installation media, which of course renders the process of using FST not
as straightforward as it should be, especially if the label used doesn't
meet the FAT limitations of using 11 uppercase characters (with further
additional restrictions), as this may result in a need to manually edit
GRUB/Syslinux configuration files.
Which means that, currently, unless you are using Debian (or Arch, which
does make an effort to use labels are FAT compatible), you are better of
not trying to use FST, as you are taking a gamble as to whether your
media is actually going to work unless you know precisely what you're doing.
And this gamble is precisely what I aim at trying to eliminate moving
forward, with the help of distro maintainers.
In other words, as a person invested in the means of providing people
with *choice* on how they should create their media, especially on
Windows platforms, I am trying to make the current problematic situation
evolve for the better. Which means that, yes, you will find me at the
forefront of trying to guide distributions into establishing FST as a
viable alternative to dd, and, sadly, as a potentially lone voice in
doing so.
But I have to be 100% clear about this, even if it would be easy to draw
the conclusion that, as the purveyor of Rufus, a boot media creation
application which tries not to use 'dd' as default (for the reasons that
will be explained below) I might have some clear vested interest, my
goal here, even if it may seem to be counter productive, is to genuinely
allow people to create their Linux installation media *without* having
to use third party applications like Rufus, since this is something
that, as opposed to BIOS, UEFI certainly makes achievable. This is
because I do consider that people who want to install a new OS should be
given the choice on how they should proceed with how they do so. But
currently, because ISOHybrid + 'dd' appears to be largely being seen as
the ultimate panacea by distro maintainers (and I can certainly see the
reasons why, because it certainly does make life easier for distro
maintainers), I do have to point out that it does come with some
drawbacks, along with my concerns at how little traction there appears
to be in terms of wanting to establish an alternative to only using dd
with ISOHybrids. That's because, from where I stand, I see this "dd uber
alles" stance as ultimately damaging for end users in the long run, even
if it does appear that I am currently have to be the one person that has
to try to convince distro maintainers that they may want to apply
careful consideration before putting all of their eggs in the ISOHybrid
as 'dd' basket...
Post by Steve McIntyreI genuinely have not seen
issues raised by people doing this, except where (it seems) Rufus is
working this way.
I will assert that this is because you are looking at this the wrong way
around. The underlying problem is that, and this always seems to come as
a shock to people who may be Linux-centric, people trying to create boot
media through ISOHybrid + dd *are* actually experiencing issues that are
effectively preventing them from installing Linux systems altogether.
And the annoying thing is that, since these are mostly Windows users,
very few of these issues actually make their way towards the attention
of Linux distro maintainers...
So, to starts us, here's a non-exhaustive list of reports I compiled
some time ago, to demonstrate this very point (and I'm going to stop at
12 examples, because I will assume that, even if I could add more, you
will get the point that this is a *real* issue faced by users, and not
something that should be dismissed as a low impact problem):
*
https://old.reddit.com/r/linuxquestions/comments/j4nolf/creating_a_bootable_usb_drive_for_linux/
*
https://old.reddit.com/r/ManjaroLinux/comments/gofq71/problem_with_rufus_310_and_manjaro_2001/
*
https://old.reddit.com/r/ManjaroLinux/comments/gjdpi4/cannot_create_bootable_usb_usb_size_shrinks_after/
*
https://old.reddit.com/r/ManjaroLinux/comments/c58bb8/manjaro_boot_media_shows_up_as_miso_efi/
*
https://old.reddit.com/r/techsupport/comments/499b5c/usb_stick_capacity_shrunk_to_2mb/
*
https://superuser.com/questions/752874/16-gb-usb-flash-drive-capacity-down-to-938-mb
*
https://www.diskgenius.com/how-to/how-to-restore-usb-drive-back-to-full-capacity.php
*
https://www.quora.com/After-making-my-32-GB-pen-drive-Kali-Linux-bootable-its-size-reduced-to-712-KB-Is-it-normal-If-not-then-how-can-I-fix-it
*
https://askubuntu.com/questions/289971/usbs-storage-capacity-reduced-to-2-mb-from-16-gb
*
https://askubuntu.com/questions/611325/capacity-of-pen-drive-shown-is-less-than-the-actual
*
https://askubuntu.com/questions/586118/16gb-pen-drive-showing-2mb-space-after-formatting-on-windows-7
*
https://forums.tomshardware.com/threads/usb-flash-drive-8gb-is-now-only-1gb.660997/
Note that some of these links may mention Rufus, because I mostly got
them while looking for issues that people may run into while running
Rufus. But since it's about 'dd' mode creation from ISOHybrid, and not
FST, the application being used here is irrelevant, as you will have no
trouble getting similar reports from people using balenaEtcher or any
'dd' like application you can use on Windows.
The core of the issue is that, because ISOHybrid media written in DD
mode on Windows usually results in a multipartioned device, with the
larger partition usually using a file system that Windows will not mount
and the smaller partition being the ESP, users are being thrown off by
the result, and are considering that the media creation process did not
complete successfully. Thus, they will not even try to use their media
to install Linux.
And I know that it is very tempting to say that "This is a user problem"
or that "These people should just plow through and it's their fault for
not doing so", but that doesn't it make less of an issue.
I could also mention that some people want to make sure that, if they
use a computer that supports UEFI and Legacy, they do install their OS
in pure UEFI mode, which is something that can easily be sorted out by
ensuring that the media is partitioned as GPT, but which an ISOHybrid
almost never is (and, I should add, if it is, then, because it will not
have the backup GPT at the expected position when written in dd mode,
then any GPT based ISOHYbrid media written in dd mode will actually be
"broken" in terms of UEFI compliance... and some UEFI firmwares might be
pedantic enough not to boot a media with a broken backup GPT). So, there
again, we are facing an issue with using ISOHybrid as 'dd', that can
easily be sorted out by switching to FST.
Then there are some opportunities of doing things, like installing
Debian ARM64 in full UEFI mode on a Raspberry Pi 4 using a single media
[1] (and I think this is part of a conversation you and I were already
involved in the past) that will never ever be achievable if treating
ISOHybrid only as a glorified dd image. And there again, I know that one
might be tempted to dismiss the idea of doing single media installation
on an SBC, when someone *should* just be able to use 2 drives anyway,
but if that is the case, I have to warn about idea once again of
*limiting* the end-user's choice (or deciding that all users, and
especially low priced SBC ones, should be wealthy enough to afford two
flash media).
I believe that we pretty much all came to FLOSS because we didn't like
to be limited by software in terms of choice. Which is why, even if it
appears that, (much to my dismay) I might be a single voice in doing so,
I feel it is my duty to continue to hammer the message that users should
be given the choice on how they create their installation media and
that, precisely because most distros are still placing major hurdles in
that respect, we must ensure that the hurdles that prevent choice are
being cleared.
I'll also conclude this section by stating that, from a technical
perspective, I can't help but feel a bit both amused and slightly
dismayed that, whereas UEFI was designed from the ground up to do away
with the necessity, enacted by BIOS, of creating boot media at the
sector level, to instead allow for the creation of boot media purely at
the file system level, we've now seen a push from many Linux distro
maintainers to completely ignore this most welcome UEFI improvement, and
instead do a complete 180 on it by more or less enforcing the writing of
boot media at the sector level, through the promotion of ISOHybrid as dd
only...
Post by Steve McIntyreLet's be clear: "transposition" == "copying things
onto a less capable filesystem".
I'd reframe this, because we're really only dealing with UEFI here, as:
"transposition" == "copying things onto the file system that is mandated
by UEFI as supported for boot"
The fact that it is indeed less capable is an unfortunate design
decision by the UEFI committee (which is hard to fathom where they had
plenty of more capable file systems to pick from at the time, especially
as FAT did come with licensing issues that Microsoft had to provide an
exception for). But this is neither something that you, nor I, nor the
end-user, has any choice with in the first place. In other words,
transposition is not a choice of a lesser file system. It's simply the
hand we are being dealt with by UEFI.
Post by Steve McIntyreI'm normally really impressed by your
skills and knowledge, but I don't understand why you seem to be so
obsessed by this.
I just want future users to be able to create Debian installation media
that they can use for their purpose, which, unlike what some people here
may think, may mean providing them with a *choice* to use either dd of
FST when creating the media, since each mode has its advantages and
drawbacks (and let me also be clear here, since it may appear that I am
bashing ISOHybrid as dd, I am in no way implying that one is "better"
than the other, as I very much see both as having their place). But
right now, because I would say Linux-centric folks appear to be less
alert to the current plight of Windows users, I feel like I have no
choice to raise my voice to make this plight heard, and the need for FST
support, as an alternative to dd, be understood. And it is very
disheartening for me to be faced with Linux-centric person after
Linux-centric person not coming to the logical conclusion that:
1. ISOHybrid as DD has its limits, and some of it are negatively
impacting the very same users that ISOHybrid as DD was supposed to help
in the first place.
2. Treating FST on an equal standing as DD will benefit users, and
therefore Linux distributions that support it, in the long run.
Post by Steve McIntyreWith some tweaks like this MR, simply copying files onto FAT32 can
clearly be made to work for UEFI-only media. That's fine. But it still
* BIOS boot
BIOS boot breakage is a given and is out of scope, since I am only
considering UEFI FST (which I will assert, considering that almost all
modern systems are UEFI based, should be fair).
Rufus can actually work some magic, to some extent, to make BIOS based
systems also work with FST, but that's well beyond the scope of what I
consider we should concern ourselves with (and, in case this is a
question, I am certainly not asking for anything from Debian that is
directly related to BIOS FST boot)
It seems you aren't aware that the default behaviour of Windows is to
create a 'System Volume Information' folder on any file system it
recognises.
Which means that, if you create an ISOHybrid using 'dd' on Windows, and
it includes a ESP, Windows will create a 'System Volume Information'
folder, which will break the full image checksum, since the ESP content
will have been altered the minute you created the media on Windows.
This is a Windows behaviour that is enabled by default and that you
can't disable without a reboot.
On the other hand, individual file checksums, which is what Ubuntu (and
others) have been using to validate their media, does work quite well
with both FST and DD.
Furthermore, the expectation (or at least my expectation) is that we're
moving towards trusted validation of binaries that are being loaded
during the boot sequence, which implicitly includes individual file
checksums, and makes part of the need for whole image checksums
redundant (and which, again, will work regardless of whether you're
using a "limited" file system like FAT or a more "capable" one).
Post by Steve McIntyreand those are important for me and a lot our users.
They are important for me too. That's why, if you want BIOS boot, I
would encourage you to use Rufus, as it will take care of creating a
Debian bootable media that just works, regardless of whether you are
*choosing* to use FST or DD
Post by Steve McIntyreSeriously, just
using DD or similar gives people a verifiable, known-good copy of our
installer image that will boot on as many machines as possible and
work well in the debian-installer environment. That's my primary goal
here.
Then I hope some of the points I highlighted above, and that should help
appreciate that ISOHybrid as DD might not be the foolproof panacea that
quite a few distro maintainers appear to think it is, will hit home.
Regards,
/Pete
[1] https://forums.raspberrypi.com//viewtopic.php?f=50&t=282839