| Message ID | 20190115133450.9676-1-maarten@de-vri.es |
|---|---|
| State | New |
| Headers | show |
| Series | [devtools,1/2] arch-nspawn, mkarchroot: Allow not sharing the cache directories. | expand |
On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote: > These patches make it possible to build withouth a shared pacman cache > using makechrootpkg. I need this myself because I'm building packages > for different repositories where some of them contain packages with the > same name, but different compile time configuration. I'd suggest using different pkgnames to be honest. :D > It also allows building packages for different Arch based distros on the > same host more easily. Arch Linux 32 has the same issue and solved it using this much simpler patch: https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03 The solution would then be to add the different cachedirs to the pacman.conf you use, which is anyways going to be different due to being i686-specific.
On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects <arch-projects@archlinux.org> wrote: > > On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote: > > These patches make it possible to build withouth a shared pacman cache > > using makechrootpkg. I need this myself because I'm building packages > > for different repositories where some of them contain packages with the > > same name, but different compile time configuration. > > I'd suggest using different pkgnames to be honest. :D I can see your point there, but many of the PKGBUILDs are automatically generated. It would be possible to add prefixes of course, but we'll get very long package names. And the repositories are never used together, so for the end-user it looks weird. > > > It also allows building packages for different Arch based distros on the > > same host more easily. > > Arch Linux 32 has the same issue and solved it using this much simpler > patch: > https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03 > > The solution would then be to add the different cachedirs to the > pacman.conf you use, which is anyways going to be different due to being > i686-specific. But this will still bind a directory from the host into the container right? It would require me to create separate cache directories on the host and create matching pacman.conf files. It would indeed be a simpler patch to devtools, but the whole process is harder. I do think these patches make sense. > > -- > Eli Schwartz > Bug Wrangler and Trusted User >
>>> It also allows building packages for different Arch based distros on the >>> same host more easily. >> >> Arch Linux 32 has the same issue and solved it using this much simpler >> patch: >> https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03 >> >> The solution would then be to add the different cachedirs to the >> pacman.conf you use, which is anyways going to be different due to being >> i686-specific. > > But this will still bind a directory from the host into the container right? > It would require me to create separate cache directories on the host > and create matching pacman.conf files. > It would indeed be a simpler patch to devtools, but the whole process is harder. This is inevitable if you want to have a cache which persist through cleaning the root/ chroot (the option is -c, IIRC) - which is IMHO a good thing to have. > > I do think these patches make sense. Only if you plan to use no (persistent) cache at all. regards, Erich
On 1/15/19 8:56 AM, Maarten de Vries wrote: > On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects > <arch-projects@archlinux.org> wrote: >> >> On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote: >>> These patches make it possible to build withouth a shared pacman cache >>> using makechrootpkg. I need this myself because I'm building packages >>> for different repositories where some of them contain packages with the >>> same name, but different compile time configuration. >> >> I'd suggest using different pkgnames to be honest. :D > > I can see your point there, but many of the PKGBUILDs are > automatically generated. > It would be possible to add prefixes of course, but we'll get very > long package names. > And the repositories are never used together, so for the end-user it > looks weird. Without being able to visualize what you're doing I cannot say whether it makes sense to me. ¯\_(ツ)_/¯ >>> It also allows building packages for different Arch based distros on the >>> same host more easily. >> >> Arch Linux 32 has the same issue and solved it using this much simpler >> patch: >> https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03 >> >> The solution would then be to add the different cachedirs to the >> pacman.conf you use, which is anyways going to be different due to being >> i686-specific. > > But this will still bind a directory from the host into the container right? > It would require me to create separate cache directories on the host > and create matching pacman.conf files. > It would indeed be a simpler patch to devtools, but the whole process is harder. > > I do think these patches make sense. You could just reuse the cache directory that is being used as the custom repo itself. You could even simply use this as an additional cache directory, and no packages will ever be downloaded to the main one.
On Tue, 15 Jan 2019 at 15:03, Eli Schwartz <eschwartz@archlinux.org> wrote: > > On 1/15/19 8:56 AM, Maarten de Vries wrote: > > On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects > > <arch-projects@archlinux.org> wrote: > >> > >> On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote: > >>> These patches make it possible to build withouth a shared pacman cache > >>> using makechrootpkg. I need this myself because I'm building packages > >>> for different repositories where some of them contain packages with the > >>> same name, but different compile time configuration. > >> > >> I'd suggest using different pkgnames to be honest. :D > > > > I can see your point there, but many of the PKGBUILDs are > > automatically generated. > > It would be possible to add prefixes of course, but we'll get very > > long package names. > > And the repositories are never used together, so for the end-user it > > looks weird. > > Without being able to visualize what you're doing I cannot say whether > it makes sense to me. ¯\_(ツ)_/¯ > > >>> It also allows building packages for different Arch based distros on the > >>> same host more easily. > >> > >> Arch Linux 32 has the same issue and solved it using this much simpler > >> patch: > >> https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03 > >> > >> The solution would then be to add the different cachedirs to the > >> pacman.conf you use, which is anyways going to be different due to being > >> i686-specific. > > > > But this will still bind a directory from the host into the container right? > > It would require me to create separate cache directories on the host > > and create matching pacman.conf files. > > It would indeed be a simpler patch to devtools, but the whole process is harder. > > > > I do think these patches make sense. > > You could just reuse the cache directory that is being used as the > custom repo itself. You could even simply use this as an additional > cache directory, and no packages will ever be downloaded to the main one. Hmm, that sounds useful indeed. Only problem for me here is that the host building the packages is not the one who stores them. Otherwise that would be very neat. Also, Erich makes a good point that having a persistent cache is indeed a good thing, also across cleaning the chroot. I suppose that maybe then these patches aren't needed at all. It might still be useful to have a command line argument to set the cache, but maintaining a modified pacman.conf isn't that problematic if I have to make sure the directories exist anyway. So, thanks for the quick feedback Eli and Erich, I'll solve this differently :) > > -- > Eli Schwartz > Bug Wrangler and Trusted User >
The commit you linked earlier[1], is that something that will find it's way into mainline devtools? [1] https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03 On Tue, 15 Jan 2019 at 15:11, Maarten de Vries <maarten@de-vri.es> wrote: > > On Tue, 15 Jan 2019 at 15:03, Eli Schwartz <eschwartz@archlinux.org> wrote: > > > > On 1/15/19 8:56 AM, Maarten de Vries wrote: > > > On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects > > > <arch-projects@archlinux.org> wrote: > > >> > > >> On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote: > > >>> These patches make it possible to build withouth a shared pacman cache > > >>> using makechrootpkg. I need this myself because I'm building packages > > >>> for different repositories where some of them contain packages with the > > >>> same name, but different compile time configuration. > > >> > > >> I'd suggest using different pkgnames to be honest. :D > > > > > > I can see your point there, but many of the PKGBUILDs are > > > automatically generated. > > > It would be possible to add prefixes of course, but we'll get very > > > long package names. > > > And the repositories are never used together, so for the end-user it > > > looks weird. > > > > Without being able to visualize what you're doing I cannot say whether > > it makes sense to me. ¯\_(ツ)_/¯ > > > > >>> It also allows building packages for different Arch based distros on the > > >>> same host more easily. > > >> > > >> Arch Linux 32 has the same issue and solved it using this much simpler > > >> patch: > > >> https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03 > > >> > > >> The solution would then be to add the different cachedirs to the > > >> pacman.conf you use, which is anyways going to be different due to being > > >> i686-specific. > > > > > > But this will still bind a directory from the host into the container right? > > > It would require me to create separate cache directories on the host > > > and create matching pacman.conf files. > > > It would indeed be a simpler patch to devtools, but the whole process is harder. > > > > > > I do think these patches make sense. > > > > You could just reuse the cache directory that is being used as the > > custom repo itself. You could even simply use this as an additional > > cache directory, and no packages will ever be downloaded to the main one. > > Hmm, that sounds useful indeed. Only problem for me here is that the > host building the packages is not the one who stores them. Otherwise > that would be very neat. > > Also, Erich makes a good point that having a persistent cache is > indeed a good thing, also across cleaning the chroot. > > I suppose that maybe then these patches aren't needed at all. It might > still be useful to have a command line argument to set the cache, but > maintaining a modified pacman.conf isn't that problematic if I have to > make sure the directories exist anyway. > > So, thanks for the quick feedback Eli and Erich, I'll solve this differently :) > > > > > -- > > Eli Schwartz > > Bug Wrangler and Trusted User > >
On 1/17/19 6:27 AM, Maarten de Vries wrote: > The commit you linked earlier[1], is that something that will find > it's way into mainline devtools? > > [1] https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03 Unlikely. Not that I'm opposed to it happening, it's just totally not worth my time to even try. The last time I tried to submit a patch for devtools was here: https://git.archlinux.org/devtools.git/commit/?id=5b3c14454a9c1ec00a3ef11f3f599281127b383d This is when it was finally merged: https://lists.archlinux.org/pipermail/arch-projects/2018-May/004897.html It was only merged at that time, because despite my having contributed a patch months before which make makechrootpkg work *at all* with pacman 5.1 on the host system, no one was actually reading this mailing list who had commit rights for devtools. Eventually, when pacman 5.1 was released, people started complaining in the private staff channel that they could no longer build packages, which is sort of awkward for the maintenance of the distribution, right? Anyway, I said "I have pending patches from months ago that were supposed to make this a non-issue", and I appealed directly to Allan in order to merge it, and he asked me for a commit id that I thought should be merged. And this is the story of how I actually managed to get some changes into devtools. By hunting down someone on IRC *after* the emergency has already happened. (Observant witnesses will note that there is another patch after that, also written by me. That doesn't count, as one of the devtools maintainers had something which bothered them enough to work on devtools, and asked about it in IRC, and I wrote the patch basically on request.) ... I have lots of changes I want to actually make practical use of on a personal level, and I *also* want to use modifications in order to build packages for an archlinux32 chroot. I'm well on the way to totally forking everything, and my changes are just going to get more significant. Attempting to upstream a collection of controversial as well as non-controversial changes, when my chances of even being heard in the first place are... dim... is not worth the time when I can simply sudo make install something that makes me happy and gets things done.
On Thu, 17 Jan 2019 at 17:15, Eli Schwartz <eschwartz@archlinux.org> wrote: > > On 1/17/19 6:27 AM, Maarten de Vries wrote: > > The commit you linked earlier[1], is that something that will find > > it's way into mainline devtools? > > > > [1] https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03 > Unlikely. Not that I'm opposed to it happening, it's just totally not > worth my time to even try. > > The last time I tried to submit a patch for devtools was here: > https://git.archlinux.org/devtools.git/commit/?id=5b3c14454a9c1ec00a3ef11f3f599281127b383d > > This is when it was finally merged: > https://lists.archlinux.org/pipermail/arch-projects/2018-May/004897.html > > It was only merged at that time, because despite my having contributed a > patch months before which make makechrootpkg work *at all* with pacman > 5.1 on the host system, no one was actually reading this mailing list > who had commit rights for devtools. > > Eventually, when pacman 5.1 was released, people started complaining in > the private staff channel that they could no longer build packages, > which is sort of awkward for the maintenance of the distribution, right? > Anyway, I said "I have pending patches from months ago that were > supposed to make this a non-issue", and I appealed directly to Allan in > order to merge it, and he asked me for a commit id that I thought should > be merged. > > And this is the story of how I actually managed to get some changes into > devtools. By hunting down someone on IRC *after* the emergency has > already happened. > > (Observant witnesses will note that there is another patch after that, > also written by me. That doesn't count, as one of the devtools > maintainers had something which bothered them enough to work on > devtools, and asked about it in IRC, and I wrote the patch basically on > request.) > > ... > > I have lots of changes I want to actually make practical use of on a > personal level, and I *also* want to use modifications in order to build > packages for an archlinux32 chroot. I'm well on the way to totally > forking everything, and my changes are just going to get more > significant. Attempting to upstream a collection of controversial as > well as non-controversial changes, when my chances of even being heard > in the first place are... dim... is not worth the time when I can simply > sudo make install something that makes me happy and gets things done. > I see. In that case I'll stick with a fork for now. Thanks again for the feedback :) -- Maarten
diff --git a/arch-nspawn.in b/arch-nspawn.in index b9c846e..c6cc3a4 100644 --- a/arch-nspawn.in +++ b/arch-nspawn.in @@ -52,7 +52,9 @@ shift 1 [[ -z $working_dir ]] && die 'Please specify a working directory.' -if [[ -z $cache_dir ]]; then +if [[ $cache_dir = '-' ]]; then + cache_dirs=() +elif [[ -z $cache_dir ]]; then cache_dirs=($(pacman -v 2>&1 | grep '^Cache Dirs:' | sed 's/Cache Dirs:\s*//g')) else cache_dirs=("$cache_dir") @@ -72,7 +74,9 @@ build_mount_args() { mount_args+=("--bind-ro=$host_mirror_path") fi - mount_args+=("--bind=${cache_dirs[0]}") + if [[ ${#cache_dirs[@]} -ge 1 ]]; then + mount_args+=("--bind=${cache_dirs[0]}") + fi for cache_dir in "${cache_dirs[@]:1}"; do mount_args+=("--bind-ro=$cache_dir") @@ -92,7 +96,9 @@ copy_hostconf () { cp -T "$file" "$working_dir$file" done - sed -r "s|^#?\\s*CacheDir.+|CacheDir = ${cache_dirs[*]}|g" -i "$working_dir/etc/pacman.conf" + if [[ ${#cache_dirs[@]} -ge 1 ]]; then + sed -r "s|^#?\\s*CacheDir.+|CacheDir = ${cache_dirs[*]}|g" -i "$working_dir/etc/pacman.conf" + fi } # }}} diff --git a/mkarchroot.in b/mkarchroot.in index 52e363f..81bc58a 100644 --- a/mkarchroot.in +++ b/mkarchroot.in @@ -51,7 +51,9 @@ shift 1 [[ -z $working_dir ]] && die 'Please specify a working directory.' -if [[ -z $cache_dir ]]; then +if [[ $cache_dir = '-' ]]; then + cache_dirs=() +elif [[ -z $cache_dir ]]; then cache_dirs=($(pacman -v "$cache_conf" 2>&1 | grep '^Cache Dirs:' | sed 's/Cache Dirs:\s*//g')) else cache_dirs=(${cache_dir})
These patches make it possible to build withouth a shared pacman cache using makechrootpkg. I need this myself because I'm building packages for different repositories where some of them contain packages with the same name, but different compile time configuration. It also allows building packages for different Arch based distros on the same host more easily. The first patch changes arch-nspawn and mkarchroot to accept `-c -` to disable bind mounting the host pacman cache in the container. The container will simply use whatever is configured in the container instead. The second patch allows the user set the pacman cache for makechrootpkg. Sadly -c was already taken, so here it's -C. The end result is that a user can run the following to build without a shared pacman cache: `makechrootpkg -C - -r $chroot` I considered a more generic -N ... to pass arguments to arch-nspawn, but arch-nspawn doesn't have that many options and it would get quite ugly: makechrootpkg -N -c -N - -r $chroot Signed-off-by: Maarten de Vries <maarten@de-vri.es> --- arch-nspawn.in | 12 +++++++++--- mkarchroot.in | 4 +++- 2 files changed, 12 insertions(+), 4 deletions(-)