[devtools,1/2] arch-nspawn, mkarchroot: Allow not sharing the cache directories.

Message ID 20190115133450.9676-1-maarten@de-vri.es
State New
Headers show
Series
  • [devtools,1/2] arch-nspawn, mkarchroot: Allow not sharing the cache directories.
Related show

Commit Message

Eli Schwartz via arch-projects Jan. 15, 2019, 1:34 p.m. UTC
These patches make it possible to build withouth a shared pacman cache
using makechrootpkg. I need this myself because I'm building packages
for different repositories where some of them contain packages with the
same name, but different compile time configuration.

It also allows building packages for different Arch based distros on the
same host more easily.

The first patch changes arch-nspawn and mkarchroot to accept `-c -` to
disable bind mounting the host pacman cache in the container.
The container will simply use whatever is configured in the container
instead.

The second patch allows the user set the pacman cache for makechrootpkg.
Sadly -c was already taken, so here it's -C.

The end result is that a user can run the following to build without
a shared pacman cache:

`makechrootpkg -C - -r $chroot`

I considered a more generic -N ... to pass arguments to arch-nspawn,
but arch-nspawn doesn't have that many options and it would get quite
ugly:

makechrootpkg -N -c -N - -r $chroot

Signed-off-by: Maarten de Vries <maarten@de-vri.es>
---
 arch-nspawn.in | 12 +++++++++---
 mkarchroot.in  |  4 +++-
 2 files changed, 12 insertions(+), 4 deletions(-)

Comments

Eli Schwartz via arch-projects Jan. 15, 2019, 1:41 p.m. UTC | #1
On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:
> These patches make it possible to build withouth a shared pacman cache
> using makechrootpkg. I need this myself because I'm building packages
> for different repositories where some of them contain packages with the
> same name, but different compile time configuration.

I'd suggest using different pkgnames to be honest. :D

> It also allows building packages for different Arch based distros on the
> same host more easily.

Arch Linux 32 has the same issue and solved it using this much simpler
patch:
https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03

The solution would then be to add the different cachedirs to the
pacman.conf you use, which is anyways going to be different due to being
i686-specific.
Eli Schwartz via arch-projects Jan. 15, 2019, 1:56 p.m. UTC | #2
On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects
<arch-projects@archlinux.org> wrote:
>
> On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:
> > These patches make it possible to build withouth a shared pacman cache
> > using makechrootpkg. I need this myself because I'm building packages
> > for different repositories where some of them contain packages with the
> > same name, but different compile time configuration.
>
> I'd suggest using different pkgnames to be honest. :D

I can see your point there, but many of the PKGBUILDs are
automatically generated.
It would be possible to add prefixes of course, but we'll get very
long package names.
And the repositories are never used together, so for the end-user it
looks weird.

>
> > It also allows building packages for different Arch based distros on the
> > same host more easily.
>
> Arch Linux 32 has the same issue and solved it using this much simpler
> patch:
> https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03
>
> The solution would then be to add the different cachedirs to the
> pacman.conf you use, which is anyways going to be different due to being
> i686-specific.

But this will still bind a directory from the host into the container right?
It would require me to create separate cache directories on the host
and create matching pacman.conf files.
It would indeed be a simpler patch to devtools, but the whole process is harder.

I do think these patches make sense.

>
> --
> Eli Schwartz
> Bug Wrangler and Trusted User
>
Erich Eckner Jan. 15, 2019, 2:01 p.m. UTC | #3
>>> It also allows building packages for different Arch based distros on the
>>> same host more easily.
>>
>> Arch Linux 32 has the same issue and solved it using this much simpler
>> patch:
>> https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03
>>
>> The solution would then be to add the different cachedirs to the
>> pacman.conf you use, which is anyways going to be different due to being
>> i686-specific.
> 
> But this will still bind a directory from the host into the container right?
> It would require me to create separate cache directories on the host
> and create matching pacman.conf files.
> It would indeed be a simpler patch to devtools, but the whole process is harder.

This is inevitable if you want to have a cache which persist through
cleaning the root/ chroot (the option is -c, IIRC) - which is IMHO a
good thing to have.

> 
> I do think these patches make sense.

Only if you plan to use no (persistent) cache at all.

regards,
Erich
Eli Schwartz via arch-projects Jan. 15, 2019, 2:03 p.m. UTC | #4
On 1/15/19 8:56 AM, Maarten de Vries wrote:
> On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects
> <arch-projects@archlinux.org> wrote:
>>
>> On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:
>>> These patches make it possible to build withouth a shared pacman cache
>>> using makechrootpkg. I need this myself because I'm building packages
>>> for different repositories where some of them contain packages with the
>>> same name, but different compile time configuration.
>>
>> I'd suggest using different pkgnames to be honest. :D
> 
> I can see your point there, but many of the PKGBUILDs are
> automatically generated.
> It would be possible to add prefixes of course, but we'll get very
> long package names.
> And the repositories are never used together, so for the end-user it
> looks weird.

Without being able to visualize what you're doing I cannot say whether
it makes sense to me. ¯\_(ツ)_/¯

>>> It also allows building packages for different Arch based distros on the
>>> same host more easily.
>>
>> Arch Linux 32 has the same issue and solved it using this much simpler
>> patch:
>> https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03
>>
>> The solution would then be to add the different cachedirs to the
>> pacman.conf you use, which is anyways going to be different due to being
>> i686-specific.
> 
> But this will still bind a directory from the host into the container right?
> It would require me to create separate cache directories on the host
> and create matching pacman.conf files.
> It would indeed be a simpler patch to devtools, but the whole process is harder.
> 
> I do think these patches make sense.

You could just reuse the cache directory that is being used as the
custom repo itself. You could even simply use this as an additional
cache directory, and no packages will ever be downloaded to the main one.
Eli Schwartz via arch-projects Jan. 15, 2019, 2:11 p.m. UTC | #5
On Tue, 15 Jan 2019 at 15:03, Eli Schwartz <eschwartz@archlinux.org> wrote:
>
> On 1/15/19 8:56 AM, Maarten de Vries wrote:
> > On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects
> > <arch-projects@archlinux.org> wrote:
> >>
> >> On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:
> >>> These patches make it possible to build withouth a shared pacman cache
> >>> using makechrootpkg. I need this myself because I'm building packages
> >>> for different repositories where some of them contain packages with the
> >>> same name, but different compile time configuration.
> >>
> >> I'd suggest using different pkgnames to be honest. :D
> >
> > I can see your point there, but many of the PKGBUILDs are
> > automatically generated.
> > It would be possible to add prefixes of course, but we'll get very
> > long package names.
> > And the repositories are never used together, so for the end-user it
> > looks weird.
>
> Without being able to visualize what you're doing I cannot say whether
> it makes sense to me. ¯\_(ツ)_/¯
>
> >>> It also allows building packages for different Arch based distros on the
> >>> same host more easily.
> >>
> >> Arch Linux 32 has the same issue and solved it using this much simpler
> >> patch:
> >> https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03
> >>
> >> The solution would then be to add the different cachedirs to the
> >> pacman.conf you use, which is anyways going to be different due to being
> >> i686-specific.
> >
> > But this will still bind a directory from the host into the container right?
> > It would require me to create separate cache directories on the host
> > and create matching pacman.conf files.
> > It would indeed be a simpler patch to devtools, but the whole process is harder.
> >
> > I do think these patches make sense.
>
> You could just reuse the cache directory that is being used as the
> custom repo itself. You could even simply use this as an additional
> cache directory, and no packages will ever be downloaded to the main one.

Hmm, that sounds useful indeed. Only problem for me here is that the
host building the packages is not the one who stores them. Otherwise
that would be very neat.

Also, Erich makes a good point that having a persistent cache is
indeed a good thing, also across cleaning the chroot.

I suppose that maybe then these patches aren't needed at all. It might
still be useful to have a command line argument to set the cache, but
maintaining a modified pacman.conf isn't that problematic if I have to
make sure the directories exist anyway.

So, thanks for the quick feedback Eli and Erich, I'll solve this differently :)

>
> --
> Eli Schwartz
> Bug Wrangler and Trusted User
>
Eli Schwartz via arch-projects Jan. 17, 2019, 11:27 a.m. UTC | #6
The commit you linked earlier[1], is that something that will find
it's way into mainline devtools?

[1] https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03

On Tue, 15 Jan 2019 at 15:11, Maarten de Vries <maarten@de-vri.es> wrote:
>
> On Tue, 15 Jan 2019 at 15:03, Eli Schwartz <eschwartz@archlinux.org> wrote:
> >
> > On 1/15/19 8:56 AM, Maarten de Vries wrote:
> > > On Tue, 15 Jan 2019 at 14:41, Eli Schwartz via arch-projects
> > > <arch-projects@archlinux.org> wrote:
> > >>
> > >> On 1/15/19 8:34 AM, Maarten de Vries via arch-projects wrote:
> > >>> These patches make it possible to build withouth a shared pacman cache
> > >>> using makechrootpkg. I need this myself because I'm building packages
> > >>> for different repositories where some of them contain packages with the
> > >>> same name, but different compile time configuration.
> > >>
> > >> I'd suggest using different pkgnames to be honest. :D
> > >
> > > I can see your point there, but many of the PKGBUILDs are
> > > automatically generated.
> > > It would be possible to add prefixes of course, but we'll get very
> > > long package names.
> > > And the repositories are never used together, so for the end-user it
> > > looks weird.
> >
> > Without being able to visualize what you're doing I cannot say whether
> > it makes sense to me. ¯\_(ツ)_/¯
> >
> > >>> It also allows building packages for different Arch based distros on the
> > >>> same host more easily.
> > >>
> > >> Arch Linux 32 has the same issue and solved it using this much simpler
> > >> patch:
> > >> https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03
> > >>
> > >> The solution would then be to add the different cachedirs to the
> > >> pacman.conf you use, which is anyways going to be different due to being
> > >> i686-specific.
> > >
> > > But this will still bind a directory from the host into the container right?
> > > It would require me to create separate cache directories on the host
> > > and create matching pacman.conf files.
> > > It would indeed be a simpler patch to devtools, but the whole process is harder.
> > >
> > > I do think these patches make sense.
> >
> > You could just reuse the cache directory that is being used as the
> > custom repo itself. You could even simply use this as an additional
> > cache directory, and no packages will ever be downloaded to the main one.
>
> Hmm, that sounds useful indeed. Only problem for me here is that the
> host building the packages is not the one who stores them. Otherwise
> that would be very neat.
>
> Also, Erich makes a good point that having a persistent cache is
> indeed a good thing, also across cleaning the chroot.
>
> I suppose that maybe then these patches aren't needed at all. It might
> still be useful to have a command line argument to set the cache, but
> maintaining a modified pacman.conf isn't that problematic if I have to
> make sure the directories exist anyway.
>
> So, thanks for the quick feedback Eli and Erich, I'll solve this differently :)
>
> >
> > --
> > Eli Schwartz
> > Bug Wrangler and Trusted User
> >
Eli Schwartz via arch-projects Jan. 17, 2019, 4:15 p.m. UTC | #7
On 1/17/19 6:27 AM, Maarten de Vries wrote:
> The commit you linked earlier[1], is that something that will find
> it's way into mainline devtools?
> 
> [1] https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03
Unlikely. Not that I'm opposed to it happening, it's just totally not
worth my time to even try.

The last time I tried to submit a patch for devtools was here:
https://git.archlinux.org/devtools.git/commit/?id=5b3c14454a9c1ec00a3ef11f3f599281127b383d

This is when it was finally merged:
https://lists.archlinux.org/pipermail/arch-projects/2018-May/004897.html

It was only merged at that time, because despite my having contributed a
patch months before which make makechrootpkg work *at all* with pacman
5.1 on the host system, no one was actually reading this mailing list
who had commit rights for devtools.

Eventually, when pacman 5.1 was released, people started complaining in
the private staff channel that they could no longer build packages,
which is sort of awkward for the maintenance of the distribution, right?
Anyway, I said "I have pending patches from months ago that were
supposed to make this a non-issue", and I appealed directly to Allan in
order to merge it, and he asked me for a commit id that I thought should
be merged.

And this is the story of how I actually managed to get some changes into
devtools. By hunting down someone on IRC *after* the emergency has
already happened.

(Observant witnesses will note that there is another patch after that,
also written by me. That doesn't count, as one of the devtools
maintainers had something which bothered them enough to work on
devtools, and asked about it in IRC, and I wrote the patch basically on
request.)

...

I have lots of changes I want to actually make practical use of on a
personal level, and I *also* want to use modifications in order to build
packages for an archlinux32 chroot. I'm well on the way to totally
forking everything, and my changes are just going to get more
significant. Attempting to upstream a collection of controversial as
well as non-controversial changes, when my chances of even being heard
in the first place are... dim... is not worth the time when I can simply
sudo make install something that makes me happy and gets things done.
Eli Schwartz via arch-projects Jan. 17, 2019, 4:24 p.m. UTC | #8
On Thu, 17 Jan 2019 at 17:15, Eli Schwartz <eschwartz@archlinux.org> wrote:
>
> On 1/17/19 6:27 AM, Maarten de Vries wrote:
> > The commit you linked earlier[1], is that something that will find
> > it's way into mainline devtools?
> >
> > [1] https://github.com/eli-schwartz/devtools/commit/c0681c0ec0a93a4a4eaf9b2fd85ce48a30702a03
> Unlikely. Not that I'm opposed to it happening, it's just totally not
> worth my time to even try.
>
> The last time I tried to submit a patch for devtools was here:
> https://git.archlinux.org/devtools.git/commit/?id=5b3c14454a9c1ec00a3ef11f3f599281127b383d
>
> This is when it was finally merged:
> https://lists.archlinux.org/pipermail/arch-projects/2018-May/004897.html
>
> It was only merged at that time, because despite my having contributed a
> patch months before which make makechrootpkg work *at all* with pacman
> 5.1 on the host system, no one was actually reading this mailing list
> who had commit rights for devtools.
>
> Eventually, when pacman 5.1 was released, people started complaining in
> the private staff channel that they could no longer build packages,
> which is sort of awkward for the maintenance of the distribution, right?
> Anyway, I said "I have pending patches from months ago that were
> supposed to make this a non-issue", and I appealed directly to Allan in
> order to merge it, and he asked me for a commit id that I thought should
> be merged.
>
> And this is the story of how I actually managed to get some changes into
> devtools. By hunting down someone on IRC *after* the emergency has
> already happened.
>
> (Observant witnesses will note that there is another patch after that,
> also written by me. That doesn't count, as one of the devtools
> maintainers had something which bothered them enough to work on
> devtools, and asked about it in IRC, and I wrote the patch basically on
> request.)
>
> ...
>
> I have lots of changes I want to actually make practical use of on a
> personal level, and I *also* want to use modifications in order to build
> packages for an archlinux32 chroot. I'm well on the way to totally
> forking everything, and my changes are just going to get more
> significant. Attempting to upstream a collection of controversial as
> well as non-controversial changes, when my chances of even being heard
> in the first place are... dim... is not worth the time when I can simply
> sudo make install something that makes me happy and gets things done.
>

I see. In that case I'll stick with a fork for now. Thanks again for
the feedback :)

-- Maarten

Patch

diff --git a/arch-nspawn.in b/arch-nspawn.in
index b9c846e..c6cc3a4 100644
--- a/arch-nspawn.in
+++ b/arch-nspawn.in
@@ -52,7 +52,9 @@  shift 1
 
 [[ -z $working_dir ]] && die 'Please specify a working directory.'
 
-if [[ -z $cache_dir ]]; then
+if [[ $cache_dir = '-' ]]; then
+	cache_dirs=()
+elif [[ -z $cache_dir ]]; then
 	cache_dirs=($(pacman -v 2>&1 | grep '^Cache Dirs:' | sed 's/Cache Dirs:\s*//g'))
 else
 	cache_dirs=("$cache_dir")
@@ -72,7 +74,9 @@  build_mount_args() {
 		mount_args+=("--bind-ro=$host_mirror_path")
 	fi
 
-	mount_args+=("--bind=${cache_dirs[0]}")
+	if [[ ${#cache_dirs[@]} -ge 1 ]]; then
+		mount_args+=("--bind=${cache_dirs[0]}")
+	fi
 
 	for cache_dir in "${cache_dirs[@]:1}"; do
 		mount_args+=("--bind-ro=$cache_dir")
@@ -92,7 +96,9 @@  copy_hostconf () {
 		cp -T "$file" "$working_dir$file"
 	done
 
-	sed -r "s|^#?\\s*CacheDir.+|CacheDir = ${cache_dirs[*]}|g" -i "$working_dir/etc/pacman.conf"
+	if [[ ${#cache_dirs[@]} -ge 1 ]]; then
+		sed -r "s|^#?\\s*CacheDir.+|CacheDir = ${cache_dirs[*]}|g" -i "$working_dir/etc/pacman.conf"
+	fi
 }
 # }}}
 
diff --git a/mkarchroot.in b/mkarchroot.in
index 52e363f..81bc58a 100644
--- a/mkarchroot.in
+++ b/mkarchroot.in
@@ -51,7 +51,9 @@  shift 1
 
 [[ -z $working_dir ]] && die 'Please specify a working directory.'
 
-if [[ -z $cache_dir ]]; then
+if [[ $cache_dir = '-' ]]; then
+	cache_dirs=()
+elif [[ -z $cache_dir ]]; then
 	cache_dirs=($(pacman -v "$cache_conf" 2>&1 | grep '^Cache Dirs:' | sed 's/Cache Dirs:\s*//g'))
 else
 	cache_dirs=(${cache_dir})