| Message ID | 20200822131245.136978-1-daan.j.demeyer@gmail.com |
|---|---|
| State | Under Review |
| Headers | show |
| Series | [pacman-dev,v4] libalpm: Add support for trigger dropins | expand |
On 8/22/20 9:12 AM, Daan De Meyer wrote: > In some scenarios, instead of adding their own hooks, packages want to > augment hooks already installed by other packages with extra triggers. > Currently, this requires modifying the existing hook file which is error > prone and hard to manage for package managers. > > A concrete example where packages would want to extend an existing hook > is using systemd's kernel-install script on Arch Linux. An example > pacman hook for kernel-install could be the following: > > ``` > [Trigger] > Operation = Install > Operation = Upgrade > Type = Path > Target = usr/lib/modules/*/vmlinuz > Target = usr/lib/kernel/install.d/* > > [Trigger] > Operation = Install > Operation = Upgrade > Type = Package > Target = systemd > > [Action] > Description = Adding kernel and initramfs images to /boot... > When = PostTransaction > Exec = /etc/pacman.d/scripts/mkosi-kernel-add > NeedsTargets > ``` > > This hook would run on installation and upgrades of kernel packages and > every package that installs kernel-install scripts (which includes > dracut and mkinitcpio). It's also fairly generic in that it doesn't > include specifics of other packages except its source package (and > implicitly the install directory of kernel packages). > > However, both mkinitcpio and dracut use the concept of hooks that > can be installed by other packages to extend their functionality. In > mkinitcpio these are stored in /usr/lib/initcpio. When a package is > installed that installs extra hooks for mkinitcpio, the kernel-install > hook will not trigger. We can fix this by adding /usr/lib/initcpio/* to > the kernel-install pacman hook but this requires either modifying the > kernel-install hook when installing mkinitpcio or adding all possible > directories for all possible packages that want to add triggers to the > kernel-install hook in the systemd package itself. Neither of these are > attractive solutions. The first one isn't because we'd have to modify > the hook when installing packages, the second isn't because it would be > a huge maintenance burden and doesn't include for AUR or private > repository packages. This all seems fairly specific to the kernel itself. As a matter of curiosity... why can't a mkinitcpio hook declare it is triggered on /usr/lib/initcpio and a dracut hook declare it is triggered on /usr/lib/dracut/modules.d/ ? In fact, the current mkinitcpio hook does precisely this, and is provided by the mkinitcpio package. It's all statically coded ahead of time, AFAICT the only reason to do it this way is to incorrectly put a kernel-install hook into the systemd package and make the mkinitcpio/dracut packages dynamically load themselves into said hook. There is currently no kernel-install pacman hook, and even if one did want to declare kernel-install the official way to manage kernels on Arch Linux, the systemd package is the wrong place to put it as systemd is installed on many Arch systems that don't have a kernel and should not run kernel-install hooks. If two packages provide mutually exclusive services, I think it's reasonable for them to also provide mutually exclusive pacman hooks geared to that package specifically. It's not like you can even use kernel-install with both dracut and mkinitcpio installed, since they both try to create the same initramfs. More likely, since kernel-install is only catering to a fairly narrow use case while being installed as part of the base system due to it being bundled as part of systemd itself, it will be the responsibility of people who actually like it to manually (no package) install their own hook to /etc/pacman.d statically coding the desired directory to watch, which will never ever ever change after the initial installation process. It's entirely reasonable for people to create their own custom hooks for their own system, augmenting the packaged hooks which are only supposed to be used for cases where it's unambiguously correct to run those routines. That is why there is a hookdir in /etc. I have a bunch of hooks which are totally unfit for general packaging but I wouldn't wish to live without. Can I hear an argument for where this functionality is generically useful, and in a case which isn't simply incorrect packaging?
> If two packages provide mutually exclusive services, I think it's > reasonable for them to also provide mutually exclusive pacman hooks > geared to that package specifically. It's not like you can even use > kernel-install with both dracut and mkinitcpio installed, since they > both try to create the same initramfs. You actually can since they don't put it in the same place. You're also technically not forced to have dracut generate an initramfs. You could have it installed to play around with unified kernel images while mkinitcpio is installed to get the usual kernels/initramfs split images. And if you do have both installed and they both use kernel-install (let's not focus too much here on whether kernel-install is a good idea for Arch or not, that's a discussion for the Arch mailing list), it makes sense to me that they both reuse the same hook instead of having to almost exactly the same hook between the two packages, just with an extra Target line to include their own module directory. > There is currently no kernel-install pacman hook, and even if one did > want to declare kernel-install the official way to manage kernels on > Arch Linux, the systemd package is the wrong place to put it as systemd > is installed on many Arch systems that don't have a kernel and should > not run kernel-install hooks. On systems without kernels the kernel-install hook would be a noop. One small annoyance is that it would still pop up in pacman's output even though it's not doing anything but that's something minor imo. > Can I hear an argument for where this functionality is generically > useful, and in a case which isn't simply incorrect packaging? If hook based packages switch to non-shell glob based patterns and instead have packages that install hooks also install their own trigger, a user can selectively disable some of these triggers. e.g. a user has a package installed that comes with a dracut hook but isn't interested in the hook and symlinks it to /dev/null in /etc so it doesn't run. However, when the package is updated, the pacman dracut hook will still trigger because of the updated dracut hook installed by the package. If the pacman dracut hook required external hooks to install their own triggers, the user could disable the hook in dracut and disable the trigger in pacman, resulting in the hook never being executed and also not triggering extra dracut runs by the dracut pacman hook. Adding more triggers is mostly useful for hook based packages depending on other hook based packages (kernel-install and mkinitcpio/dracut is a single example of this) where the nested hook based package can add its own hook directory as a trigger to the main package pacman hook. Assuming a package stores some stuff in a non-standard path and needs another package to run on that non-standard place when it's updated, it can add a drop-in to add that path to the list of triggers for that package's pacman hook. I realize that these are mostly niceties and I'm not solving some big massive issue packaging issue by adding this. Still, since there's not that much added complexity in the implementation and the user facing part (imo) so adding this seems worth it to me to make some of the mentioned use cases a little easier to handle. Daan On Sun, 23 Aug 2020 at 07:35, Eli Schwartz <eschwartz@archlinux.org> wrote: > > On 8/22/20 9:12 AM, Daan De Meyer wrote: > > In some scenarios, instead of adding their own hooks, packages want to > > augment hooks already installed by other packages with extra triggers. > > Currently, this requires modifying the existing hook file which is error > > prone and hard to manage for package managers. > > > > A concrete example where packages would want to extend an existing hook > > is using systemd's kernel-install script on Arch Linux. An example > > pacman hook for kernel-install could be the following: > > > > ``` > > [Trigger] > > Operation = Install > > Operation = Upgrade > > Type = Path > > Target = usr/lib/modules/*/vmlinuz > > Target = usr/lib/kernel/install.d/* > > > > [Trigger] > > Operation = Install > > Operation = Upgrade > > Type = Package > > Target = systemd > > > > [Action] > > Description = Adding kernel and initramfs images to /boot... > > When = PostTransaction > > Exec = /etc/pacman.d/scripts/mkosi-kernel-add > > NeedsTargets > > ``` > > > > This hook would run on installation and upgrades of kernel packages and > > every package that installs kernel-install scripts (which includes > > dracut and mkinitcpio). It's also fairly generic in that it doesn't > > include specifics of other packages except its source package (and > > implicitly the install directory of kernel packages). > > > > However, both mkinitcpio and dracut use the concept of hooks that > > can be installed by other packages to extend their functionality. In > > mkinitcpio these are stored in /usr/lib/initcpio. When a package is > > installed that installs extra hooks for mkinitcpio, the kernel-install > > hook will not trigger. We can fix this by adding /usr/lib/initcpio/* to > > the kernel-install pacman hook but this requires either modifying the > > kernel-install hook when installing mkinitpcio or adding all possible > > directories for all possible packages that want to add triggers to the > > kernel-install hook in the systemd package itself. Neither of these are > > attractive solutions. The first one isn't because we'd have to modify > > the hook when installing packages, the second isn't because it would be > > a huge maintenance burden and doesn't include for AUR or private > > repository packages. > > This all seems fairly specific to the kernel itself. As a matter of > curiosity... why can't a mkinitcpio hook declare it is triggered on > /usr/lib/initcpio and a dracut hook declare it is triggered on > /usr/lib/dracut/modules.d/ ? > In fact, the current mkinitcpio hook does precisely this, and is > provided by the mkinitcpio package. > > It's all statically coded ahead of time, AFAICT the only reason to do it > this way is to incorrectly put a kernel-install hook into the systemd > package and make the mkinitcpio/dracut packages dynamically load > themselves into said hook. > > There is currently no kernel-install pacman hook, and even if one did > want to declare kernel-install the official way to manage kernels on > Arch Linux, the systemd package is the wrong place to put it as systemd > is installed on many Arch systems that don't have a kernel and should > not run kernel-install hooks. > > If two packages provide mutually exclusive services, I think it's > reasonable for them to also provide mutually exclusive pacman hooks > geared to that package specifically. It's not like you can even use > kernel-install with both dracut and mkinitcpio installed, since they > both try to create the same initramfs. > > More likely, since kernel-install is only catering to a fairly narrow > use case while being installed as part of the base system due to it > being bundled as part of systemd itself, it will be the responsibility > of people who actually like it to manually (no package) install their > own hook to /etc/pacman.d statically coding the desired directory to > watch, which will never ever ever change after the initial installation > process. > > It's entirely reasonable for people to create their own custom hooks for > their own system, augmenting the packaged hooks which are only supposed > to be used for cases where it's unambiguously correct to run those > routines. That is why there is a hookdir in /etc. I have a bunch of > hooks which are totally unfit for general packaging but I wouldn't wish > to live without. > > Can I hear an argument for where this functionality is generically > useful, and in a case which isn't simply incorrect packaging? > > -- > Eli Schwartz > Bug Wrangler and Trusted User >
On 8/23/20 6:32 AM, Daan De Meyer wrote: >> If two packages provide mutually exclusive services, I think it's >> reasonable for them to also provide mutually exclusive pacman hooks >> geared to that package specifically. It's not like you can even use >> kernel-install with both dracut and mkinitcpio installed, since they >> both try to create the same initramfs. > > You actually can since they don't put it in the same place. You're > also technically not forced to have dracut generate an initramfs. You > could have it installed to play around with unified kernel images > while mkinitcpio is installed to get the usual kernels/initramfs split > images. One initcpio is generated by /usr/lib/kernel/install.d/50-mkinitcpio.install and one by /usr/lib/kernel/install.d/50-dracut.install, if both are installed then they both write a kernel to BOOT/MACHINE-ID/KERNEL-VERSION/initrd I don't understand the purpose of having these both autogenerate the same file. Actually I don't understand the potential purpose of having them both autogenerate different files at the same time either... > And if you do have both installed and they both use > kernel-install (let's not focus too much here on whether > kernel-install is a good idea for Arch or not, that's a discussion for > the Arch mailing list), it makes sense to me that they both reuse the > same hook instead of having to almost exactly the same hook between > the two packages, just with an extra Target line to include their own > module directory. Why does it matter how similar the hooks are, these are very small files... That's just not a good argument IMO. That's an argument for adding additional features to pacman, with the intended use of allowing a single package to do the same thing twice, after throwing out the efforts of the first time! This isn't about whether kernel-install is a good thing (people use it, their reason for using it is because they like it, therefore for them it's presumably a good thing...). It's about whether kernel-install using *both* dracut *and* mkinitcpio at the same time is a good thing. I'm confident it isn't. I'm more than happy for user choice, so if people like hooks for kernel-install they're more than welcome to use it. But there is *zero* reason for competing implementations of the same thing to do the same work twice, and we should not be adding additional features to pacman for that sole use case. >> There is currently no kernel-install pacman hook, and even if one did >> want to declare kernel-install the official way to manage kernels on >> Arch Linux, the systemd package is the wrong place to put it as systemd >> is installed on many Arch systems that don't have a kernel and should >> not run kernel-install hooks. > > On systems without kernels the kernel-install hook would be a noop. > One small annoyance is that it would still pop up in pacman's output > even though it's not doing anything but that's something minor imo. If you want to add no-op hooks on your own system, that's fine, but I believe it's quite rare to do so in the official repos at the moment... because generally they are installed in the same package that uses them. I'm not sure I see a compelling argument to install things in the wrong package just because they won't get used unless you install the right package. It's not a small annoyance which is worth it due to $benefit. It's a small annoyance with absolutely no reason to exist, and no benefits, and the sheer lack of a point would make me, as a user, more annoyed than I would be otherwise. >> Can I hear an argument for where this functionality is generically >> useful, and in a case which isn't simply incorrect packaging? > > If hook based packages switch to non-shell glob based patterns and > instead have packages that install hooks also install their own > trigger, a user can selectively disable some of these triggers. e.g. a > user has a package installed that comes with a dracut hook but isn't > interested in the hook and symlinks it to /dev/null in /etc so it > doesn't run. However, when the package is updated, the pacman dracut > hook will still trigger because of the updated dracut hook installed > by the package. If the pacman dracut hook required external hooks to > install their own triggers, the user could disable the hook in dracut > and disable the trigger in pacman, resulting in the hook never being > executed and also not triggering extra dracut runs by the dracut > pacman hook. I'm pretty sure we don't want to force every single package providing a dracut/modules.d/ file to also install additional pacman hook files instead of using globs for their intended purpose, as this would then mess up the expected functionality of those other packages, and in all 3 cases for dracut, they're installed by the upstream build system, not the Arch packager -- so this would force the packagers to maintain an absolute filelist by hand. Maybe it would be interesting to look into having NoExtract'ed files not trigger hooks, instead. Users could prevent installation of dracut files they don't want to use, and as a bonus, if you suddenly decide to use it you track this by starting to install the file, rather than just building the initrd and then forgetting the hook doesn't watch for rebuilds anymore and having it not rebuild when it should. > Adding more triggers is mostly useful for hook based packages > depending on other hook based packages (kernel-install and > mkinitcpio/dracut is a single example of this) where the nested hook > based package can add its own hook directory as a trigger to the main > package pacman hook. Assuming a package stores some stuff in a > non-standard path and needs another package to run on that > non-standard place when it's updated, it can add a drop-in to add that > path to the list of triggers for that package's pacman hook. You don't need to re-explain to me "what" it does, this is evident in the initial patch submission. :) But again -- I don't see how this is useful for anything other than one arguably wrongly-implemented kernel package (which is not actually implemented yet). My argument is you "shouldn't" do this. ... I'm also not sure what "nested" hooks have to do with this, since pacman hooks aren't nested and kernel-install would only be one hook that (re)builds an initramfs as part of its internal functionality. Do you want to be able to have a hook that is also triggered when another named hook is run? That's a different proposal, but color me intrigued. > I realize that these are mostly niceties and I'm not solving some big > massive issue packaging issue by adding this. Still, since there's not > that much added complexity in the implementation and the user facing > part (imo) so adding this seems worth it to me to make some of the > mentioned use cases a little easier to handle. I'm definitely up for improving niceties, there's no problem there. :) My only concern here is that I don't think this is a nicety -- I think it's an incorrect approach.
Lets take a step back here... I don't really care about the kernel use case, but more whether this could be more generally used. Here are other examples I came up with. Font caches: A package could drop a config file in /etc/font/conf.d/ to add another directory that is to be used when building font caches. They would also want to add a .trigger file for the hook. OK... I just came up with one other example! But I did not look too hard at what hooks on my system actually do. Back to the question of whether we support this. A quick skim of the patch shows me it is quite long, but relatively simple. (it seems to be doing more than just adding .trigger support - also adding drop in directories - and this seems two patches to me...) So, I am leaning in the direction of this being a useful addition. Further discussion of the usefulness should not focus around the kernel approach, but rather the general idea of the feature. Allan
So looking at general usefulness, I think it mostly comes down to two things: - Selectively disable triggers for existing packages. This would require packages to switch to drop-in based triggers which the user can then override or an additional patch would be needed to allow a negative glob in another trigger to override a normal glob in a previous trigger. So if a package has a glob /* trigger and a drop-in trigger has a glob !/usr/* trigger, those would combine to search everything except /usr. I'm not sure if we want to allow triggers to influence each other though. I'm guessing this is not possible today and it might require some changes to get this work if triggers are currently architected to be independent of each other (which might very well be a good thing to keep things simple). - Add triggers for additional directories/files to packages This is useful for NeedsTargets based triggers since you can pass additional targets to the hook and if the hook can deal with those extra targets, dropping in new directories/files for the hook to work on becomes really easy. It can also be useful for hooks without NeedsTargets if aside from the added trigger, you also configure the extra directory to work on in a configuration directory for the package somewhere. So for the font cache example, you could add the extra directory somewhere to a configuration file of the program that builds font caches and add a trigger drop-in for that extra directory to the pacman hook as well and things should just work. If the font cache builder program works with NeedsTargets, you only need to add the trigger and it should just work. One thing to be aware of with this approach is that we might not want to encourage adding extra directories like this to search for files. At least in systemd and Arch Linux, the preference is to use store files in a fixed set of directories instead of adding new directories all over the place. If the idea is to store files in existing directories, making it easy to use alternative directories might not be something we want to do. Then again, this is rather specific to my experience with systemd/Arch. pacman might be used in other scenarios where doing this is perfectly valid and encouraged. Daan On Mon, 24 Aug 2020 at 02:22, Allan McRae <allan@archlinux.org> wrote: > > Lets take a step back here... > > I don't really care about the kernel use case, but more whether this > could be more generally used. Here are other examples I came up with. > > Font caches: > A package could drop a config file in /etc/font/conf.d/ to add another > directory that is to be used when building font caches. They would also > want to add a .trigger file for the hook. > > OK... I just came up with one other example! But I did not look too > hard at what hooks on my system actually do. > > > Back to the question of whether we support this. A quick skim of the > patch shows me it is quite long, but relatively simple. (it seems to be > doing more than just adding .trigger support - also adding drop in > directories - and this seems two patches to me...) > > So, I am leaning in the direction of this being a useful addition. > > Further discussion of the usefulness should not focus around the kernel > approach, but rather the general idea of the feature. > > Allan
Em agosto 23, 2020 22:22 Allan McRae escreveu: > Lets take a step back here... > > I don't really care about the kernel use case, but more whether this > could be more generally used. Here are other examples I came up with. > > Font caches: > A package could drop a config file in /etc/font/conf.d/ to add another > directory that is to be used when building font caches. They would also > want to add a .trigger file for the hook. > > OK... I just came up with one other example! But I did not look too > hard at what hooks on my system actually do. > > > Back to the question of whether we support this. A quick skim of the > patch shows me it is quite long, but relatively simple. (it seems to be > doing more than just adding .trigger support - also adding drop in > directories - and this seems two patches to me...) > > So, I am leaning in the direction of this being a useful addition. > > Further discussion of the usefulness should not focus around the kernel > approach, but rather the general idea of the feature. > Without getting much into the kernel stuff, I also think this is a good addition for pacman. I think that, the discussion regarding the kernel stuff could go into arch-general though. I'd be happy to revisit the whole kernel-install discussion too. Regards, Giancarlo Razzolini
On 08/24/20 at 11:22am, Allan McRae wrote: > Lets take a step back here... > > I don't really care about the kernel use case, but more whether this > could be more generally used. Here are other examples I came up with. > > Font caches: > A package could drop a config file in /etc/font/conf.d/ to add another > directory that is to be used when building font caches. They would also > want to add a .trigger file for the hook. > > OK... I just came up with one other example! But I did not look too > hard at what hooks on my system actually do. > > > Back to the question of whether we support this. A quick skim of the > patch shows me it is quite long, but relatively simple. (it seems to be > doing more than just adding .trigger support - also adding drop in > directories - and this seems two patches to me...) > > So, I am leaning in the direction of this being a useful addition. > > Further discussion of the usefulness should not focus around the kernel > approach, but rather the general idea of the feature. I'm still a bit skeptical. I can't say I'm particularly familiar with fonts and their configuration, but is there really a valid use case for that kind of per-package configuration that we need to support? More abstractly, I'm not fond of how much this ties packages together. One of the ideas of hooks was for the package that does the work to handle all of the necessary details so the triggering packages wouldn't have to worry about it. With this approach, they not only have to know to adjust the hook, they also have to know the name of the hook and how the hook works. Changes to the hook then require an adjustment and rebuild of any package modifying it. This is not an entirely new problem, user overrides are also dependant on hook naming, but this does exacerbate it in a way that I think will be easy for packagers to miss and cause breakage. What about adding Include support to hooks? Then hooks that need this type of functionality could explicitly include trigger files from a particular directory, insulating the process from simple hook renaming and hopefully making it more obvious when changes to the hook will require changes to any package that modifies it.
On 8/24/20 4:21 PM, Andrew Gregory wrote: > I'm still a bit skeptical. I can't say I'm particularly familiar with > fonts and their configuration, but is there really a valid use case > for that kind of per-package configuration that we need to support? I kind of agree, because while it's true font conf files can add new search paths it's also not unreasonable to just use one unified path, as fontconfig already recursively searches there and you can organize fonts by subdirectory. Nevertheless, thanks Allan for mentioning it, because I really wanted to know if this could be suitably generic for non-kernel use and this gives a much better framework for discussion... > More abstractly, I'm not fond of how much this ties packages together. > One of the ideas of hooks was for the package that does the work to > handle all of the necessary details so the triggering packages > wouldn't have to worry about it. With this approach, they not only > have to know to adjust the hook, they also have to know the name of > the hook and how the hook works. Changes to the hook then require an > adjustment and rebuild of any package modifying it. This is not an > entirely new problem, user overrides are also dependant on hook > naming, but this does exacerbate it in a way that I think will be easy > for packagers to miss and cause breakage. > > What about adding Include support to hooks? Then hooks that need this > type of functionality could explicitly include trigger files from > a particular directory, insulating the process from simple hook > renaming and hopefully making it more obvious when changes to the hook > will require changes to any package that modifies it. I think you're right and Include support would be a better way to go here.
> What about adding Include support to hooks? Then hooks that need this > type of functionality could explicitly include trigger files from > a particular directory, insulating the process from simple hook > renaming and hopefully making it more obvious when changes to the hook > will require changes to any package that modifies it. Just to make sure I understand correctly, does Include support imply that hooks opt in to reading triggers from a specific directory and that we can rename the hook itself without changing the drop-in directory for that hook to avoid breakage? If so, that sounds totally reasonable and even preferable since it also avoids packages adding triggers to hooks that can't deal with extra targets when NeedsTargets is used. I can adapt the patch to use this approach. Do we pass a full path to Include or just a directory name? The advantage of just a directory name is that it could be created in any of the hook directories (hook installed in /usr/share/libalpm/hooks and the added triggers in /etc/pacman.d/hooks/<directory> for example). It doesn't necessarily have to be just a directory name, we could interpret it as a relative path and you could still put it in each of the hook directories but that might just be adding too much complexity. Daan On Mon, 24 Aug 2020 at 22:18, Eli Schwartz <eschwartz@archlinux.org> wrote: > > On 8/24/20 4:21 PM, Andrew Gregory wrote: > > I'm still a bit skeptical. I can't say I'm particularly familiar with > > fonts and their configuration, but is there really a valid use case > > for that kind of per-package configuration that we need to support? > > I kind of agree, because while it's true font conf files can add new > search paths it's also not unreasonable to just use one unified path, as > fontconfig already recursively searches there and you can organize fonts > by subdirectory. > > Nevertheless, thanks Allan for mentioning it, because I really wanted to > know if this could be suitably generic for non-kernel use and this gives > a much better framework for discussion... > > > More abstractly, I'm not fond of how much this ties packages together. > > One of the ideas of hooks was for the package that does the work to > > handle all of the necessary details so the triggering packages > > wouldn't have to worry about it. With this approach, they not only > > have to know to adjust the hook, they also have to know the name of > > the hook and how the hook works. Changes to the hook then require an > > adjustment and rebuild of any package modifying it. This is not an > > entirely new problem, user overrides are also dependant on hook > > naming, but this does exacerbate it in a way that I think will be easy > > for packagers to miss and cause breakage. > > > > What about adding Include support to hooks? Then hooks that need this > > type of functionality could explicitly include trigger files from > > a particular directory, insulating the process from simple hook > > renaming and hopefully making it more obvious when changes to the hook > > will require changes to any package that modifies it. > > I think you're right and Include support would be a better way to go here. > > -- > Eli Schwartz > Bug Wrangler and Trusted User >
On 08/24/20 at 10:34pm, Daan De Meyer wrote: > > What about adding Include support to hooks? Then hooks that need this > > type of functionality could explicitly include trigger files from > > a particular directory, insulating the process from simple hook > > renaming and hopefully making it more obvious when changes to the hook > > will require changes to any package that modifies it. > > Just to make sure I understand correctly, does Include support imply > that hooks opt in to reading triggers from a specific directory and > that we can rename the hook itself without changing the drop-in > directory for that hook to avoid breakage? If so, that sounds totally > reasonable and even preferable since it also avoids packages adding > triggers to hooks that can't deal with extra targets when NeedsTargets > is used. I can adapt the patch to use this approach. > > Do we pass a full path to Include or just a directory name? The > advantage of just a directory name is that it could be created in any > of the hook directories (hook installed in /usr/share/libalpm/hooks > and the added triggers in /etc/pacman.d/hooks/<directory> for > example). It doesn't necessarily have to be just a directory name, we > could interpret it as a relative path and you could still put it in > each of the hook directories but that might just be adding too much > complexity. Include should take a glob, just like pacman.conf.
On 31/8/20 6:23 am, Andrew Gregory wrote: > On 08/24/20 at 10:34pm, Daan De Meyer wrote: >>> What about adding Include support to hooks? Then hooks that need this >>> type of functionality could explicitly include trigger files from >>> a particular directory, insulating the process from simple hook >>> renaming and hopefully making it more obvious when changes to the hook >>> will require changes to any package that modifies it. >> >> Just to make sure I understand correctly, does Include support imply >> that hooks opt in to reading triggers from a specific directory and >> that we can rename the hook itself without changing the drop-in >> directory for that hook to avoid breakage? If so, that sounds totally >> reasonable and even preferable since it also avoids packages adding >> triggers to hooks that can't deal with extra targets when NeedsTargets >> is used. I can adapt the patch to use this approach. >> >> Do we pass a full path to Include or just a directory name? The >> advantage of just a directory name is that it could be created in any >> of the hook directories (hook installed in /usr/share/libalpm/hooks >> and the added triggers in /etc/pacman.d/hooks/<directory> for >> example). It doesn't necessarily have to be just a directory name, we >> could interpret it as a relative path and you could still put it in >> each of the hook directories but that might just be adding too much >> complexity. > > Include should take a glob, just like pacman.conf. > I'm trying to figure out how includes would address this problem. We have a hook that runs when its triggers are met. We want to have a package add additional triggers, by dropping in a file into our config, but do not want to duplicate the function of the hook, or to directly edit the hook. An Include would provide a place where we can drop more config files with (e.g.) more triggers, but that requires the original hook to be set up such that it is extendable. Just adding additional triggers does not require the original hook to do anything. Now, just writing as I think here, so don't take this as a good idea... Unless a hooks name defined their include directory? I.e. fontconfig.hook would include fontconfig.d/* automatically. Long story short, I'm not sold on how Includes would work! Allan
On 09/04/20 at 12:22pm, Allan McRae wrote: ... > I'm trying to figure out how includes would address this problem. > > We have a hook that runs when its triggers are met. We want to have a > package add additional triggers, by dropping in a file into our config, > but do not want to duplicate the function of the hook, or to directly > edit the hook. An Include would provide a place where we can drop more > config files with (e.g.) more triggers, but that requires the original > hook to be set up such that it is extendable. Just adding additional > triggers does not require the original hook to do anything. Requiring the hook to opt-in to this behavior is the point. Once you make a hook extensible, making any changes to that hook become much more difficult because those changes could render any extensions useless, and the hook author doesn't have any way to know if anything else is depending on the current implementation. Requiring the hook to opt-in gives the author a way to indicate that the hook is suitable for extension and won't change in a way that breaks those extensions in the future. Essentially we're allowing a hook to choose to provide a public interface instead of automatically doing it for all hooks. And, of course, the fact that we already have code to deal with Includes is a nice plus.
One idea is that we don't need to necessarily do this with includes. We could simply add an option (e.g. EnableDropinDirectories) that enables the drop-in directories for a particular hook. It doesn't account for renaming the hook but in my opinion, I don't think that will be a massive issue in practice. Daan On Sat, 5 Sep 2020 at 02:31, Andrew Gregory <andrew.gregory.8@gmail.com> wrote: > > On 09/04/20 at 12:22pm, Allan McRae wrote: > ... > > I'm trying to figure out how includes would address this problem. > > > > We have a hook that runs when its triggers are met. We want to have a > > package add additional triggers, by dropping in a file into our config, > > but do not want to duplicate the function of the hook, or to directly > > edit the hook. An Include would provide a place where we can drop more > > config files with (e.g.) more triggers, but that requires the original > > hook to be set up such that it is extendable. Just adding additional > > triggers does not require the original hook to do anything. > > Requiring the hook to opt-in to this behavior is the point. Once you > make a hook extensible, making any changes to that hook become much > more difficult because those changes could render any extensions > useless, and the hook author doesn't have any way to know if anything > else is depending on the current implementation. Requiring the hook > to opt-in gives the author a way to indicate that the hook is suitable > for extension and won't change in a way that breaks those extensions > in the future. Essentially we're allowing a hook to choose to provide > a public interface instead of automatically doing it for all hooks. > And, of course, the fact that we already have code to deal with > Includes is a nice plus.
diff --git a/doc/alpm-hooks.5.asciidoc b/doc/alpm-hooks.5.asciidoc index 916d43bb..caa55a79 100644 --- a/doc/alpm-hooks.5.asciidoc +++ b/doc/alpm-hooks.5.asciidoc @@ -38,6 +38,12 @@ linkman:pacman.conf[5] (the default is +{sysconfdir}/pacman.d/hooks+). The file names are required to have the suffix ".hook". Hooks are run in alphabetical order of their file name, where the ordering ignores the suffix. +Extra triggers can be added in drop-in directories in the configured hook +directories. Drop-in directories for a hook must be named after the hook with +the ".d" suffix added. For example, "dracut.hook.d" would be the drop-in +directory for "dracut.hook". Trigger files in drop-in directories are required +to have the suffix ".trigger" and can only contain '[Trigger]' sections. + TRIGGERS -------- @@ -96,7 +102,8 @@ OVERRIDING HOOKS Hooks may be overridden by placing a file with the same name in a higher priority hook directory. Hooks may be disabled by overriding them with -a symlink to '/dev/null'. +a symlink to '/dev/null'. The same logic applies to trigger files in drop-in +directories. EXAMPLES -------- diff --git a/lib/libalpm/hook.c b/lib/libalpm/hook.c index aca8707e..19963dab 100644 --- a/lib/libalpm/hook.c +++ b/lib/libalpm/hook.c @@ -44,6 +44,7 @@ struct _alpm_trigger_t { enum _alpm_hook_op_t op; enum _alpm_trigger_type_t type; alpm_list_t *targets; + char *name; }; struct _alpm_hook_t { @@ -66,6 +67,7 @@ static void _alpm_trigger_free(struct _alpm_trigger_t *trigger) { if(trigger) { FREELIST(trigger->targets); + free(trigger->name); free(trigger); } } @@ -146,16 +148,16 @@ static int _alpm_hook_validate(alpm_handle_t *handle, return ret; } -static int _alpm_hook_parse_cb(const char *file, int line, +#define error(...) _alpm_log(handle, ALPM_LOG_ERROR, __VA_ARGS__); return 1; +#define warning(...) _alpm_log(handle, ALPM_LOG_WARNING, __VA_ARGS__); + +static int _alpm_trigger_parse_cb(const char *file, int line, const char *section, char *key, char *value, void *data) { struct _alpm_hook_cb_ctx *ctx = data; alpm_handle_t *handle = ctx->handle; struct _alpm_hook_t *hook = ctx->hook; -#define error(...) _alpm_log(handle, ALPM_LOG_ERROR, __VA_ARGS__); return 1; -#define warning(...) _alpm_log(handle, ALPM_LOG_WARNING, __VA_ARGS__); - if(!section && !key) { error(_("error while reading hook %s: %s\n"), file, strerror(errno)); } else if(!section) { @@ -166,8 +168,6 @@ static int _alpm_hook_parse_cb(const char *file, int line, struct _alpm_trigger_t *t; CALLOC(t, sizeof(struct _alpm_trigger_t), 1, return 1); hook->triggers = alpm_list_add(hook->triggers, t); - } else if(strcmp(section, "Action") == 0) { - /* no special processing required */ } else { error(_("hook %s line %d: invalid section %s\n"), file, line, section); } @@ -205,6 +205,37 @@ static int _alpm_hook_parse_cb(const char *file, int line, } else { error(_("hook %s line %d: invalid option %s\n"), file, line, key); } + } + + return 0; +} + +static int _alpm_hook_parse_cb(const char *file, int line, + const char *section, char *key, char *value, void *data) +{ + struct _alpm_hook_cb_ctx *ctx = data; + alpm_handle_t *handle = ctx->handle; + struct _alpm_hook_t *hook = ctx->hook; + + if(!section && !key) { + error(_("error while reading hook %s: %s\n"), file, strerror(errno)); + } else if(!section) { + error(_("hook %s line %d: invalid option %s\n"), file, line, key); + } else if(!key) { + /* beginning a new section */ + if(strcmp(section, "Trigger") == 0) { + if (_alpm_trigger_parse_cb(file, line, section, key, value, data) != 0) { + return 1; + } + } else if(strcmp(section, "Action") == 0) { + /* no special processing required */ + } else { + error(_("hook %s line %d: invalid section %s\n"), file, line, section); + } + } else if(strcmp(section, "Trigger") == 0) { + if (_alpm_trigger_parse_cb(file, line, section, key, value, data) != 0) { + return 1; + } } else if(strcmp(section, "Action") == 0) { if(strcmp(key, "When") == 0) { if(hook->when != 0) { @@ -249,12 +280,12 @@ static int _alpm_hook_parse_cb(const char *file, int line, } } -#undef error -#undef warning - return 0; } +#undef error +#undef warning + static int _alpm_hook_trigger_match_file(alpm_handle_t *handle, struct _alpm_hook_t *hook, struct _alpm_trigger_t *t) { @@ -466,6 +497,17 @@ static alpm_list_t *find_hook(alpm_list_t *haystack, const void *needle) return NULL; } +static alpm_list_t *find_trigger(alpm_list_t *haystack, const void *needle) { + while (haystack) { + struct _alpm_trigger_t *t = haystack->data; + if (t && t->name && strcmp(t->name, needle) == 0) { + return haystack; + } + haystack = haystack->next; + } + return NULL; +} + static ssize_t _alpm_hook_feed_targets(char *buf, ssize_t needed, alpm_list_t **pos) { size_t remaining = needed, written = 0;; @@ -529,6 +571,209 @@ static int _alpm_hook_run_hook(alpm_handle_t *handle, struct _alpm_hook_t *hook) } } +static int _alpm_hook_parse_drop_in_one(alpm_handle_t *handle, + struct _alpm_hook_t *hook, const char *dir) +{ + char path[PATH_MAX]; + size_t dirlen; + struct dirent *entry; + DIR *d; + size_t suflen = strlen(ALPM_TRIGGER_SUFFIX); + int ret = 0; + + if((dirlen = strlen(dir)) + 1 >= PATH_MAX) { + _alpm_log(handle, ALPM_LOG_ERROR, + _("could not open drop-in directory: %s: %s\n"), dir, + strerror(ENAMETOOLONG)); + return -1; + } + memcpy(path, dir, dirlen + 1); + + if(!(d = opendir(path))) { + if(errno == ENOENT) { + return 0; + } else { + _alpm_log(handle, ALPM_LOG_ERROR, + _("could not open drop-in directory: %s: %s\n"), path, + strerror(errno)); + return -1; + } + } + + while((errno = 0, entry = readdir(d))) { + struct _alpm_hook_cb_ctx ctx = { handle, hook }; + size_t name_len; + struct stat buf; + + if(strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0) { + continue; + } + + if((name_len = strlen(entry->d_name)) >= PATH_MAX - dirlen - 1) { + _alpm_log(handle, ALPM_LOG_ERROR, _("could not open file: %s%s: %s\n"), + path, entry->d_name, strerror(ENAMETOOLONG)); + ret = -1; + continue; + } + + /* Make sure path ends with a '/'. */ + if (dirlen > 0 && path[dirlen - 1] != '/') { + path[dirlen++] = '/'; + } + + memcpy(path + dirlen, entry->d_name, name_len + 1); + + if(name_len < suflen + || strcmp(entry->d_name + name_len - suflen, ALPM_TRIGGER_SUFFIX) != 0) { + _alpm_log(handle, ALPM_LOG_DEBUG, "skipping non-trigger file %s\n", + path); + continue; + } + + if(find_trigger(hook->triggers, entry->d_name)) { + _alpm_log(handle, ALPM_LOG_DEBUG, + "skipping overridden trigger %s\n", path); + continue; + } + + if(stat(path, &buf) != 0) { + _alpm_log(handle, ALPM_LOG_ERROR, + _("could not stat trigger file %s: %s\n"), path, + strerror(errno)); + ret = -1; + continue; + } + + if(S_ISDIR(buf.st_mode)) { + _alpm_log(handle, ALPM_LOG_DEBUG, "skipping directory %s\n", path); + continue; + } + + _alpm_log(handle, ALPM_LOG_DEBUG, "parsing trigger file %s\n", path); + + if(parse_ini(path, _alpm_trigger_parse_cb, &ctx) != 0 + || _alpm_hook_validate(handle, ctx.hook, path)) { + _alpm_log(handle, ALPM_LOG_ERROR, + _("parsing trigger file %s failed\n"), path); + ret = -1; + continue; + } + + STRDUP(hook->name, entry->d_name, ret = -1); + } + if(errno != 0) { + _alpm_log(handle, ALPM_LOG_ERROR, _("could not read directory: %s: %s\n"), + dir, strerror(errno)); + ret = -1; + } + + closedir(d); + + return ret; +} + +static int _alpm_hook_parse_drop_ins(alpm_handle_t *handle, alpm_list_t *hooks) +{ + alpm_list_t *i; + size_t suflen = strlen(ALPM_DROP_IN_SUFFIX); + int ret = 0; + + for(i = alpm_list_last(handle->hookdirs); i; i = alpm_list_previous(i)) { + char path[PATH_MAX]; + size_t dirlen; + struct dirent *entry; + DIR *d; + + if((dirlen = strlen(i->data)) >= PATH_MAX) { + _alpm_log(handle, ALPM_LOG_ERROR, _("could not open file: %s: %s\n"), + (char *)i->data, strerror(ENAMETOOLONG)); + ret = -1; + continue; + } + memcpy(path, i->data, dirlen + 1); + + if(!(d = opendir(path))) { + if(errno == ENOENT) { + continue; + } else { + _alpm_log(handle, ALPM_LOG_ERROR, + _("could not open directory: %s: %s\n"), path, + strerror(errno)); + ret = -1; + continue; + } + } + + while((errno = 0, entry = readdir(d))) { + alpm_list_t *j; + struct stat buf; + size_t name_len; + + if(strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0) { + continue; + } + + if((name_len = strlen(entry->d_name)) >= PATH_MAX - dirlen) { + _alpm_log(handle, ALPM_LOG_ERROR, _("could not open file: %s%s: %s\n"), + path, entry->d_name, strerror(ENAMETOOLONG)); + ret = -1; + continue; + } + memcpy(path + dirlen, entry->d_name, name_len + 1); + + if(name_len < suflen + || strcmp(entry->d_name + name_len - suflen, ALPM_DROP_IN_SUFFIX) != 0) { + _alpm_log(handle, ALPM_LOG_DEBUG, + "skipping non-drop-in directory %s\n", path); + continue; + } + + entry->d_name[name_len - 2] = '\0'; + j = find_hook(hooks, entry->d_name); + entry->d_name[name_len - 2] = '.'; + + if (!j) { + _alpm_log(handle, ALPM_LOG_DEBUG, + "skipping drop-in directory %s without a corresponding hook", + path); + continue; + } + + if(stat(path, &buf) != 0) { + _alpm_log(handle, ALPM_LOG_ERROR, + _("could not stat drop-in directory %s: %s\n"), path, + strerror(errno)); + ret = -1; + continue; + } + + if(!S_ISDIR(buf.st_mode)) { + _alpm_log(handle, ALPM_LOG_DEBUG, "skipping file %s\n", path); + continue; + } + + _alpm_log(handle, ALPM_LOG_DEBUG, "parsing drop-in directory %s\n", path); + + if (_alpm_hook_parse_drop_in_one(handle, j->data, path) != 0) { + _alpm_log(handle, ALPM_LOG_ERROR, + _("failed to read drop-in files in drop-in directory %s"), + path); + ret = -1; + continue; + } + } + if(errno != 0) { + _alpm_log(handle, ALPM_LOG_ERROR, _("could not read directory: %s: %s\n"), + (char *) i->data, strerror(errno)); + ret = -1; + } + + closedir(d); + } + + return ret; +} + int _alpm_hook_run(alpm_handle_t *handle, alpm_hook_when_t when) { alpm_event_hook_t event = { .when = when }; @@ -626,6 +871,10 @@ int _alpm_hook_run(alpm_handle_t *handle, alpm_hook_when_t when) closedir(d); } + if (_alpm_hook_parse_drop_ins(handle, hooks) != 0) { + ret = -1; + } + if(ret != 0 && when == ALPM_HOOK_PRE_TRANSACTION) { goto cleanup; } diff --git a/lib/libalpm/hook.h b/lib/libalpm/hook.h index ff5de4f2..961d3238 100644 --- a/lib/libalpm/hook.h +++ b/lib/libalpm/hook.h @@ -23,6 +23,8 @@ #include "alpm.h" #define ALPM_HOOK_SUFFIX ".hook" +#define ALPM_DROP_IN_SUFFIX ".hook.d" +#define ALPM_TRIGGER_SUFFIX ".trigger" int _alpm_hook_run(alpm_handle_t *handle, alpm_hook_when_t when);
In some scenarios, instead of adding their own hooks, packages want to augment hooks already installed by other packages with extra triggers. Currently, this requires modifying the existing hook file which is error prone and hard to manage for package managers. A concrete example where packages would want to extend an existing hook is using systemd's kernel-install script on Arch Linux. An example pacman hook for kernel-install could be the following: ``` [Trigger] Operation = Install Operation = Upgrade Type = Path Target = usr/lib/modules/*/vmlinuz Target = usr/lib/kernel/install.d/* [Trigger] Operation = Install Operation = Upgrade Type = Package Target = systemd [Action] Description = Adding kernel and initramfs images to /boot... When = PostTransaction Exec = /etc/pacman.d/scripts/mkosi-kernel-add NeedsTargets ``` This hook would run on installation and upgrades of kernel packages and every package that installs kernel-install scripts (which includes dracut and mkinitcpio). It's also fairly generic in that it doesn't include specifics of other packages except its source package (and implicitly the install directory of kernel packages). However, both mkinitcpio and dracut use the concept of hooks that can be installed by other packages to extend their functionality. In mkinitcpio these are stored in /usr/lib/initcpio. When a package is installed that installs extra hooks for mkinitcpio, the kernel-install hook will not trigger. We can fix this by adding /usr/lib/initcpio/* to the kernel-install pacman hook but this requires either modifying the kernel-install hook when installing mkinitpcio or adding all possible directories for all possible packages that want to add triggers to the kernel-install hook in the systemd package itself. Neither of these are attractive solutions. The first one isn't because we'd have to modify the hook when installing packages, the second isn't because it would be a huge maintenance burden and doesn't include for AUR or private repository packages. Instead, this commit adds support for drop-in directories that allow extra triggers to be added in separate files. When parsing hooks, we now also look in a .d directory for extra triggers (for mkinitcpio.hook, we look in mkinitcpio.hook.d). Trigger files are required to have the .trigger extension and follow the same ini format as regular hooks files. However, only [Trigger] sections are allowed. Drop-in directories for a hook can be put in any of the existing hook search directories. The same override rules that apply for normal hooks files apply for drop-in files as well. With this feature, packages can now install extra triggers for an existing hook into the trigger dropin directory for that package. These extra triggers can be tracked by the package manager and uninstalled along with the package. Signed-off-by: Daan De Meyer <daan.j.demeyer@gmail.com> --- doc/alpm-hooks.5.asciidoc | 9 +- lib/libalpm/hook.c | 267 ++++++++++++++++++++++++++++++++++++-- lib/libalpm/hook.h | 2 + 3 files changed, 268 insertions(+), 10 deletions(-)