| Message ID | 20200903030252.641239-1-allan@archlinux.org |
|---|---|
| State | Accepted, archived |
| Headers | show |
| Series | [pacman-dev] repo-add: add --include-sigs option | expand |
On 9/2/20 11:02 PM, Allan McRae wrote: > Pacman now downloads the signature files for all packages when present in a > repository. That makes distributing signatures within repository databases > redundant and costly. > > Do not distribute the package signature files within the repo databases by > default and add an --include-sigs to revert to the old behaviour. As I've mentioned on the list before, I would like an --ignore-sigs option and continue to distribute sigs by default for pacman 6.0 In pacman 6.1 we'll switch by default to ignoring them, and let people use --include-sigs to revert to the old behavior. Ignoring sigs right out of the gate means the default behavior of repo-add is to be unusable for people upgrading from pacman N-1. For example, Arch Linux would most certainly need to use the option to provide backwards compat while upgrading. So do third-party repositories. Also: this option cannot be added to scripts ahead of time, since repo-add will error on an unknown option, and it cannot be added after the fact, since some packages will be broken in the meantime. I don't see what the rush is here to add behavior that no one will want to use. - It makes sense to make this configurable now that it's useful to be able to ignore them. - At the same time, defaults should be based on what is more likely for people to want.
On 4/9/20 12:40 pm, Eli Schwartz wrote: > On 9/2/20 11:02 PM, Allan McRae wrote: >> Pacman now downloads the signature files for all packages when present in a >> repository. That makes distributing signatures within repository databases >> redundant and costly. >> >> Do not distribute the package signature files within the repo databases by >> default and add an --include-sigs to revert to the old behaviour. > > As I've mentioned on the list before, I would like an --ignore-sigs > option and continue to distribute sigs by default for pacman 6.0 > > In pacman 6.1 we'll switch by default to ignoring them, and let people > use --include-sigs to revert to the old behavior. > > Ignoring sigs right out of the gate means the default behavior of > repo-add is to be unusable for people upgrading from pacman N-1. For > example, Arch Linux would most certainly need to use the option to > provide backwards compat while upgrading. So do third-party repositories. > > Also: this option cannot be added to scripts ahead of time, since > repo-add will error on an unknown option, and it cannot be added after > the fact, since some packages will be broken in the meantime. > > I don't see what the rush is here to add behavior that no one will want > to use. > - It makes sense to make this configurable now that it's useful to be > able to ignore them. > - At the same time, defaults should be based on what is more likely for > people to want. > I really do not like the idea of adding an option, just to remove it in the next release. But we won't actually be able to remove it for at least two releases, as you have just made the case that people won't be able to change their scripts on release. Given pacman-6.0 is likely a few months out, can we do a 5.2.3 release including something like: diff --git a/doc/repo-add.8.asciidoc b/doc/repo-add.8.asciidoc index 8de4485b..19e2336a 100644 --- a/doc/repo-add.8.asciidoc +++ b/doc/repo-add.8.asciidoc @@ -70,6 +70,10 @@ repo-add Options Remove old package files from the disk when updating their entry in the database. +*\--include-sigs*:: + Dummy option for forward compatibility with pacman-6.0. + Include package PGP signatures in the repository database (if available) + Example ------- diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index b0b3505d..ee010dba 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -43,6 +43,7 @@ LOCKFILE= CLEAN_LOCK=0 USE_COLOR='y' PREVENT_DOWNGRADE=0 +INCLUDE_SIGS=0 # Import libmakepkg source "$LIBRARY"/util/message.sh @@ -631,6 +632,9 @@ while (( $# )); do -p|--prevent-downgrade) PREVENT_DOWNGRADE=1 ;; + --include-sigs) + INCLUDE_SIGS=1 + ;; *) args+=("$1") ;;
On 4/9/20 12:55 pm, Allan McRae wrote: > On 4/9/20 12:40 pm, Eli Schwartz wrote: >> On 9/2/20 11:02 PM, Allan McRae wrote: >>> Pacman now downloads the signature files for all packages when present in a >>> repository. That makes distributing signatures within repository databases >>> redundant and costly. >>> >>> Do not distribute the package signature files within the repo databases by >>> default and add an --include-sigs to revert to the old behaviour. >> >> As I've mentioned on the list before, I would like an --ignore-sigs >> option and continue to distribute sigs by default for pacman 6.0 >> >> In pacman 6.1 we'll switch by default to ignoring them, and let people >> use --include-sigs to revert to the old behavior. >> >> Ignoring sigs right out of the gate means the default behavior of >> repo-add is to be unusable for people upgrading from pacman N-1. For >> example, Arch Linux would most certainly need to use the option to >> provide backwards compat while upgrading. So do third-party repositories. >> >> Also: this option cannot be added to scripts ahead of time, since >> repo-add will error on an unknown option, and it cannot be added after >> the fact, since some packages will be broken in the meantime. >> >> I don't see what the rush is here to add behavior that no one will want >> to use. >> - It makes sense to make this configurable now that it's useful to be >> able to ignore them. >> - At the same time, defaults should be based on what is more likely for >> people to want. >> > > I really do not like the idea of adding an option, just to remove it in > the next release. But we won't actually be able to remove it for at > least two releases, as you have just made the case that people won't be > able to change their scripts on release. > > Given pacman-6.0 is likely a few months out, can we do a 5.2.3 release > including something like: > Any feedback on this option? > > diff --git a/doc/repo-add.8.asciidoc b/doc/repo-add.8.asciidoc > index 8de4485b..19e2336a 100644 > --- a/doc/repo-add.8.asciidoc > +++ b/doc/repo-add.8.asciidoc > @@ -70,6 +70,10 @@ repo-add Options > Remove old package files from the disk when updating their entry in the > database. > > +*\--include-sigs*:: > + Dummy option for forward compatibility with pacman-6.0. > + Include package PGP signatures in the repository database (if available) > + > > Example > ------- > diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in > index b0b3505d..ee010dba 100644 > --- a/scripts/repo-add.sh.in > +++ b/scripts/repo-add.sh.in > @@ -43,6 +43,7 @@ LOCKFILE= > CLEAN_LOCK=0 > USE_COLOR='y' > PREVENT_DOWNGRADE=0 > +INCLUDE_SIGS=0 > > # Import libmakepkg > source "$LIBRARY"/util/message.sh > @@ -631,6 +632,9 @@ while (( $# )); do > -p|--prevent-downgrade) > PREVENT_DOWNGRADE=1 > ;; > + --include-sigs) > + INCLUDE_SIGS=1 > + ;; > *) > args+=("$1") > ;; > . >
On 09/21/20 at 03:19pm, Allan McRae wrote: > On 4/9/20 12:55 pm, Allan McRae wrote: > > On 4/9/20 12:40 pm, Eli Schwartz wrote: > >> On 9/2/20 11:02 PM, Allan McRae wrote: > >>> Pacman now downloads the signature files for all packages when present in a > >>> repository. That makes distributing signatures within repository databases > >>> redundant and costly. > >>> > >>> Do not distribute the package signature files within the repo databases by > >>> default and add an --include-sigs to revert to the old behaviour. > >> > >> As I've mentioned on the list before, I would like an --ignore-sigs > >> option and continue to distribute sigs by default for pacman 6.0 > >> > >> In pacman 6.1 we'll switch by default to ignoring them, and let people > >> use --include-sigs to revert to the old behavior. > >> > >> Ignoring sigs right out of the gate means the default behavior of > >> repo-add is to be unusable for people upgrading from pacman N-1. For > >> example, Arch Linux would most certainly need to use the option to > >> provide backwards compat while upgrading. So do third-party repositories. > >> > >> Also: this option cannot be added to scripts ahead of time, since > >> repo-add will error on an unknown option, and it cannot be added after > >> the fact, since some packages will be broken in the meantime. > >> > >> I don't see what the rush is here to add behavior that no one will want > >> to use. > >> - It makes sense to make this configurable now that it's useful to be > >> able to ignore them. > >> - At the same time, defaults should be based on what is more likely for > >> people to want. > >> > > > > I really do not like the idea of adding an option, just to remove it in > > the next release. But we won't actually be able to remove it for at > > least two releases, as you have just made the case that people won't be > > able to change their scripts on release. > > > > Given pacman-6.0 is likely a few months out, can we do a 5.2.3 release > > including something like: > > > > Any feedback on this option? I would suggest just allowing the user to specify either way (--include-sigs/--no-include-sigs, --include-sigs={yes,no}, etc). Then uses can specify whatever they want without having to worry about what we set as a default. > > > > diff --git a/doc/repo-add.8.asciidoc b/doc/repo-add.8.asciidoc > > index 8de4485b..19e2336a 100644 > > --- a/doc/repo-add.8.asciidoc > > +++ b/doc/repo-add.8.asciidoc > > @@ -70,6 +70,10 @@ repo-add Options > > Remove old package files from the disk when updating their entry in the > > database. > > > > +*\--include-sigs*:: > > + Dummy option for forward compatibility with pacman-6.0. > > + Include package PGP signatures in the repository database (if available) > > + > > > > Example > > ------- > > diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in > > index b0b3505d..ee010dba 100644 > > --- a/scripts/repo-add.sh.in > > +++ b/scripts/repo-add.sh.in > > @@ -43,6 +43,7 @@ LOCKFILE= > > CLEAN_LOCK=0 > > USE_COLOR='y' > > PREVENT_DOWNGRADE=0 > > +INCLUDE_SIGS=0 > > > > # Import libmakepkg > > source "$LIBRARY"/util/message.sh > > @@ -631,6 +632,9 @@ while (( $# )); do > > -p|--prevent-downgrade) > > PREVENT_DOWNGRADE=1 > > ;; > > + --include-sigs) > > + INCLUDE_SIGS=1 > > + ;; > > *) > > args+=("$1") > > ;; > > . > >
On 21/9/20 3:51 pm, Andrew Gregory wrote: > On 09/21/20 at 03:19pm, Allan McRae wrote: >> On 4/9/20 12:55 pm, Allan McRae wrote: >>> On 4/9/20 12:40 pm, Eli Schwartz wrote: >>>> On 9/2/20 11:02 PM, Allan McRae wrote: >>>>> Pacman now downloads the signature files for all packages when present in a >>>>> repository. That makes distributing signatures within repository databases >>>>> redundant and costly. >>>>> >>>>> Do not distribute the package signature files within the repo databases by >>>>> default and add an --include-sigs to revert to the old behaviour. >>>> >>>> As I've mentioned on the list before, I would like an --ignore-sigs >>>> option and continue to distribute sigs by default for pacman 6.0 >>>> >>>> In pacman 6.1 we'll switch by default to ignoring them, and let people >>>> use --include-sigs to revert to the old behavior. >>>> >>>> Ignoring sigs right out of the gate means the default behavior of >>>> repo-add is to be unusable for people upgrading from pacman N-1. For >>>> example, Arch Linux would most certainly need to use the option to >>>> provide backwards compat while upgrading. So do third-party repositories. >>>> >>>> Also: this option cannot be added to scripts ahead of time, since >>>> repo-add will error on an unknown option, and it cannot be added after >>>> the fact, since some packages will be broken in the meantime. >>>> >>>> I don't see what the rush is here to add behavior that no one will want >>>> to use. >>>> - It makes sense to make this configurable now that it's useful to be >>>> able to ignore them. >>>> - At the same time, defaults should be based on what is more likely for >>>> people to want. >>>> >>> >>> I really do not like the idea of adding an option, just to remove it in >>> the next release. But we won't actually be able to remove it for at >>> least two releases, as you have just made the case that people won't be >>> able to change their scripts on release. >>> >>> Given pacman-6.0 is likely a few months out, can we do a 5.2.3 release >>> including something like: >>> >> >> Any feedback on this option? > > I would suggest just allowing the user to specify either way > (--include-sigs/--no-include-sigs, --include-sigs={yes,no}, etc). > Then uses can specify whatever they want without having to worry about > what we set as a default. > The problem is more the transition. I would like the default to be not to include the signatures in the repo database. So does pacman need to manage the transition from having signatures in a database to not, or do the users need to manage that? With my patch (or any variant the does not include signatures by default), users upgrading to repo-add v6.0 would need to adjust their repo management utilities to add a signature include option immediately, as their users may still be using pacman-5.x. Thinking of Arch here, a dbscripts update would need launched on the server at the same time as updating repo-add. I am OK with that - some updates need done in concert. But Eli was not. Allan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Mon, 21 Sep 2020, Allan McRae wrote: > On 21/9/20 3:51 pm, Andrew Gregory wrote: >> >> I would suggest just allowing the user to specify either way >> (--include-sigs/--no-include-sigs, --include-sigs={yes,no}, etc). >> Then uses can specify whatever they want without having to worry about >> what we set as a default. >> > > The problem is more the transition. I would like the default to be not > to include the signatures in the repo database. So does pacman need to > manage the transition from having signatures in a database to not, or do > the users need to manage that? > > With my patch (or any variant the does not include signatures by > default), users upgrading to repo-add v6.0 would need to adjust their > repo management utilities to add a signature include option immediately, > as their users may still be using pacman-5.x. How about adding the options --include-sigs and --no-include-sigs now without changing the default behaviour? Then, everyone could adopt their scripts to include that option. Afterwards, pacman v6.0 can change the default to not include signatures without breaking setups. I believe, this is, what Andrew meant, anyways. > > Thinking of Arch here, a dbscripts update would need launched on the > server at the same time as updating repo-add. I am OK with that - some > updates need done in concert. But Eli was not. > > Allan > regards, Erich -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE3p92iMrPBP64GmxZCu7JB1Xae1oFAl9oVAwACgkQCu7JB1Xa e1pLAw/+O8DrpZIKDuiUAtu+jTeovJnheLvNWEErYtJDtddXnYZhqotvdCZA5Ycl gdVBLejDwVfWraMQYBljfY85aUstJyIwEnQz6nmpKM1ywwIpJEdBZPdzW7BPQpiJ aPg9+GWJNKKj/IrfTCc0hNVrJuYQ2lPCchZfsAZLv3VSPj18BQNr1m7j4uzSjrPe hGzeRJBflUXV8QgrMBxF9UN4W25hQojnZ1SA88Dz8xoLzm7ffG5Fa6DlXgVXf8zb pqu9li949uL3W8teeZKn6b1Va8g3CxT30aZ96qI1Mzsmx8SuK1BGrbDek5OiD0r8 gqDhvCvV4McvcrTPK4cWZgWcnbEtBbU+mgk/mKNg9xleUqGo0dxVna6IIJ3W5jwL Mk7f72CAFxOMSJ4UwVIAgMXUlOQOFIZ7mtEtzWeYNPB5TA9qY+VS6aXKNdz0hzwP rCWwvYm2nJ18N2LqyUgfZ5jVnzOUg716c7H92k+45KdD/cQC6JS83zXuzQrauOan jVZTDzqSPDLfFzT71W8nsXnLlRCX12vfBP0fCOIHjB0WJz7g7B5hnFh7nmIIYwcC Dus7V8zvhGtSfFNTq973kE0Mfd/IFjSisNwtR+4G34EefkmjbEaoPH7ytCjA6lxl urGuaHejq9vdzmuwDj6An+9Io4vJVaAEI3R9IBk3dEu2naJPeGI= =iaM7 -----END PGP SIGNATURE-----
On 9/21/20 3:02 AM, Allan McRae wrote: > On 21/9/20 3:51 pm, Andrew Gregory wrote: >> I would suggest just allowing the user to specify either way >> (--include-sigs/--no-include-sigs, --include-sigs={yes,no}, etc). >> Then uses can specify whatever they want without having to worry about >> what we set as a default. >> > > The problem is more the transition. I would like the default to be not > to include the signatures in the repo database. So does pacman need to > manage the transition from having signatures in a database to not, or do > the users need to manage that? > > With my patch (or any variant the does not include signatures by > default), users upgrading to repo-add v6.0 would need to adjust their > repo management utilities to add a signature include option immediately, > as their users may still be using pacman-5.x. > > Thinking of Arch here, a dbscripts update would need launched on the > server at the same time as updating repo-add. I am OK with that - some > updates need done in concert. But Eli was not. I'm concerned both about the need to time the adjustment of the option, and about the desire for what I see as sane defaults. My preference is to provide both options, but change the default in pacman 6.0.1. While we're hacking on repo-add options, we could go ahead and make it use parseopts, because the current handling is gross. Also I would like an elephant (usually I would request a pony, but this felt more apropos if we're talking about repo-add).
On 23/9/20 12:27 pm, Eli Schwartz wrote: > On 9/21/20 3:02 AM, Allan McRae wrote: >> On 21/9/20 3:51 pm, Andrew Gregory wrote: >>> I would suggest just allowing the user to specify either way >>> (--include-sigs/--no-include-sigs, --include-sigs={yes,no}, etc). >>> Then uses can specify whatever they want without having to worry about >>> what we set as a default. >>> >> >> The problem is more the transition. I would like the default to be not >> to include the signatures in the repo database. So does pacman need to >> manage the transition from having signatures in a database to not, or do >> the users need to manage that? >> >> With my patch (or any variant the does not include signatures by >> default), users upgrading to repo-add v6.0 would need to adjust their >> repo management utilities to add a signature include option immediately, >> as their users may still be using pacman-5.x. >> >> Thinking of Arch here, a dbscripts update would need launched on the >> server at the same time as updating repo-add. I am OK with that - some >> updates need done in concert. But Eli was not. > > I'm concerned both about the need to time the adjustment of the option, > and about the desire for what I see as sane defaults. > > My preference is to provide both options, but change the default in > pacman 6.0.1. We are not making a behaviour change in a point release. > While we're hacking on repo-add options, we could go ahead and make it > use parseopts, because the current handling is gross. Also I would like > an elephant (usually I would request a pony, but this felt more apropos > if we're talking about repo-add). I'm not touching repo-add beyond the most minor of changes. A sync db writing backend should be added to libalpm and repo-add replaced with something more easily extended. Allan
Hi On Thu, Sep 3, 2020 at 7:41 PM Eli Schwartz <eschwartz@archlinux.org> wrote: > > On 9/2/20 11:02 PM, Allan McRae wrote: > > Pacman now downloads the signature files for all packages when present in a > > repository. That makes distributing signatures within repository databases > > redundant and costly. > > > > Do not distribute the package signature files within the repo databases by > > default and add an --include-sigs to revert to the old behaviour. > > As I've mentioned on the list before, I would like an --ignore-sigs > option and continue to distribute sigs by default for pacman 6.0 I agree with Eli here. "Using embedded signatures" should stay default as long as we support clients with pacman 5.x version. Otherwise we are going to hit problems when a repo maintainer updated their system to pacman 6.x and started distributing optimized databases without signatures while some clients still expect embedded sigs. So I vote for including sigs by default in pacman 6.0 release, and then flip the default later (during 6.0.1 or 6.1 release).
On 29/9/20 6:53 am, Anatol Pomozov wrote: > Hi > > On Thu, Sep 3, 2020 at 7:41 PM Eli Schwartz <eschwartz@archlinux.org> wrote: >> >> On 9/2/20 11:02 PM, Allan McRae wrote: >>> Pacman now downloads the signature files for all packages when present in a >>> repository. That makes distributing signatures within repository databases >>> redundant and costly. >>> >>> Do not distribute the package signature files within the repo databases by >>> default and add an --include-sigs to revert to the old behaviour. >> >> As I've mentioned on the list before, I would like an --ignore-sigs >> option and continue to distribute sigs by default for pacman 6.0 > > I agree with Eli here. "Using embedded signatures" should stay default > as long as we support clients with pacman 5.x version. > > Otherwise we are going to hit problems when a repo maintainer updated > their system to pacman 6.x and started distributing optimized > databases without signatures while some clients still expect embedded > sigs. > > So I vote for including sigs by default in pacman 6.0 release, and > then flip the default later (during 6.0.1 or 6.1 release). I'll repeat, we will never switch a default in a maintenance release. And the x.y.1 release usually happens within a couple of weeks as bugs are found. So that seems a pointless transition period. Moving it to 6.1 means a likely 18 month wait given our usual release cycle. I think we all agree the default should be not to include signatures in the repo database. The question is about timing. I don't think timing is up to when pacman makes a new release. It is up to the distro or repo provider, and providing a flag to revert to the old behaviour is all that is needed. I am of the opinion if people update a major part of their package provision infrastructure without reading the changes (which will be clearly stated in the release notes), then that is their problem, not ours. A
diff --git a/doc/repo-add.8.asciidoc b/doc/repo-add.8.asciidoc index 8de4485b..9b903ab1 100644 --- a/doc/repo-add.8.asciidoc +++ b/doc/repo-add.8.asciidoc @@ -70,6 +70,8 @@ repo-add Options Remove old package files from the disk when updating their entry in the database. +*\--include-sigs*:: + Include package PGP signatures in the repository database (if available) Example ------- diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 7182d1b8..c153e7bb 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -42,6 +42,7 @@ LOCKFILE= CLEAN_LOCK=0 USE_COLOR='y' PREVENT_DOWNGRADE=0 +INCLUDE_SIGS=0 # Import libmakepkg source "$LIBRARY"/util/compress.sh @@ -260,7 +261,7 @@ db_write_entry() { fi # compute base64'd PGP signature - if [[ -f "$pkgfile.sig" ]]; then + if (( INCLUDE_SIGS )) && [[ -f "$pkgfile.sig" ]]; then if grep -q 'BEGIN PGP SIGNATURE' "$pkgfile.sig"; then error "$(gettext "Cannot use armored signatures for packages: %s")" "$pkgfile.sig" return 1 @@ -622,6 +623,9 @@ while (( $# )); do -p|--prevent-downgrade) PREVENT_DOWNGRADE=1 ;; + --include-sigs) + INCLUDE_SIGS=1 + ;; *) args+=("$1") ;;
Pacman now downloads the signature files for all packages when present in a repository. That makes distributing signatures within repository databases redundant and costly. Do not distribute the package signature files within the repo databases by default and add an --include-sigs to revert to the old behaviour. Signed-off-by: Allan McRae <allan@archlinux.org> --- doc/repo-add.8.asciidoc | 2 ++ scripts/repo-add.sh.in | 6 +++++- 2 files changed, 7 insertions(+), 1 deletion(-)