| Message ID | 20180428093108.8263-1-archlinux@thecybershadow.net |
|---|---|
| Headers | show |
| Series | makepkg: Allow placing local sources in subdirectories | expand |
On 04/28/2018 04:31 AM, Vladimir Panteleev wrote: > Currently, makepkg doesn't support (actually, mishandles) sources in a > subdirectory of the package directory. Though there aren't many uses > for such a feature, it is useful for an otherwise source-less package > of Arch- or site-specific scripts, to be installed on one or a few > systems, for which it doesn't make sense to have a separate source > repository. In this case, it would be nice to either move the source > files in a subdirectory, or allow pulling them from somewhere else on > the filesystem. > > Whether or not this is considered a valid usecase, the current > behavior is broken (makepkg computes broken paths and attempts to > operate on files that aren't there), so this probably ought to be > fixed regardless. Also https://bugs.archlinux.org/task/39718 I'm not sure we have a specified behavior, but I'm personally of the opinion this should be prohibited. IMHO there is no reason to hide files in subdirectories when the entire directory with the PKGBUILD is meant to be the package sources etc. If we did support it officially, we should make sure --source supports it too, and possibly decide whether we support this in the AUR as well. An argument could be made BTW, that these source paths are in fact malformed and should possibly be file://$(pwd)/ urls which are downloaded via curl according to the intent of the PKGBUILD author (these would then be downloaded from a local URI). The logical extension of that line of thought, is we should add a lint_pkgbuild check with a more informative error when no protocol is specified, but a non-filename is used. Yes, file:/// sources work fine. If it isn't specifically local:// then download_file is used, which does DLAGENTS lookup. We don't provide a "file" DLAGENTS protocol, which means we usually just silently exit that function without actually downloading anything... but nothing is stopping people from adding one in the PKGBUILD itself. curl supports downloading from file:///absolute/path/to/file ... I *really* dislike the inconsistency of permitting subdirs for local files (implying we think this is a reasonable use case) but not permitting them for things we resolve as something to download. > I couldn't find a test suite for makepkg (only pacman and > makepkg-template), so, here is the test case: > > mkdir dir > touch a dir/b /tmp/c > cat > PKGBUILD <<'EOF' > pkgname=dir-test > pkgver=1 > pkgrel=1 > arch=('any') > source=(a dir/b /tmp/c) > md5sums=(SKIP SKIP SKIP) > package() { cp "$srcdir"/* "$pkgdir/" ; } > EOF How would this handle the second case being, instead, local://dir/b ? The proto=local semantic is used both for files that don't have a proto, and files which use the local:// proto. The local:// proto is *actually* used, believe it or not. It's how we specify that the user is expected to provide the file themselves, for e.g. AUR packages which require proprietary sources (like games hidden behind paywalls). The AUR checks the proto, to make sure that no-proto files which *are* expected to be provided together with the PKGBUILD are not missing. ... Why don't people use, for the third case, DLAGENTS+=('file::/usr/bin/curl -C - -o %o %u'); source=('file:///tmp/c') ? And what sources do we expect to have a guaranteed filesystem location, but are not supplied by a package? I consider cp during build/package to be perfectly adequate for files provided by some other package, and in fact I use this here: https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/broadcom-wl&id=fba4c424ae9bc381f0babc61fca4643e1569de1a#n31
On 29/04/18 13:12, Eli Schwartz wrote: > On 04/28/2018 04:31 AM, Vladimir Panteleev wrote: >> Currently, makepkg doesn't support (actually, mishandles) sources in a >> subdirectory of the package directory. Though there aren't many uses >> for such a feature, it is useful for an otherwise source-less package >> of Arch- or site-specific scripts, to be installed on one or a few >> systems, for which it doesn't make sense to have a separate source >> repository. In this case, it would be nice to either move the source >> files in a subdirectory, or allow pulling them from somewhere else on >> the filesystem. >> >> Whether or not this is considered a valid usecase, the current >> behavior is broken (makepkg computes broken paths and attempts to >> operate on files that aren't there), so this probably ought to be >> fixed regardless. > Also https://bugs.archlinux.org/task/39718 > > I'm not sure we have a specified behavior, but I'm personally of the > opinion this should be prohibited. IMHO there is no reason to hide files > in subdirectories when the entire directory with the PKGBUILD is meant > to be the package sources etc. > > If we did support it officially, we should make sure --source supports > it too, and possibly decide whether we support this in the AUR as well. > > An argument could be made BTW, that these source paths are in fact > malformed and should possibly be file://$(pwd)/ urls which are > downloaded via curl according to the intent of the PKGBUILD author > (these would then be downloaded from a local URI). The logical extension > of that line of thought, is we should add a lint_pkgbuild check with a > more informative error when no protocol is specified, but a non-filename > is used. > > Yes, file:/// sources work fine. If it isn't specifically local:// then > download_file is used, which does DLAGENTS lookup. We don't provide a > "file" DLAGENTS protocol, which means we usually just silently exit that > function without actually downloading anything... but nothing is > stopping people from adding one in the PKGBUILD itself. curl supports > downloading from file:///absolute/path/to/file > > ... > > I *really* dislike the inconsistency of permitting subdirs for local > files (implying we think this is a reasonable use case) but not > permitting them for things we resolve as something to download. > >> I couldn't find a test suite for makepkg (only pacman and >> makepkg-template), so, here is the test case: >> >> mkdir dir >> touch a dir/b /tmp/c >> cat > PKGBUILD <<'EOF' >> pkgname=dir-test >> pkgver=1 >> pkgrel=1 >> arch=('any') >> source=(a dir/b /tmp/c) >> md5sums=(SKIP SKIP SKIP) >> package() { cp "$srcdir"/* "$pkgdir/" ; } >> EOF > > How would this handle the second case being, instead, local://dir/b ? > > The proto=local semantic is used both for files that don't have a proto, > and files which use the local:// proto. The local:// proto is *actually* > used, believe it or not. It's how we specify that the user is expected > to provide the file themselves, for e.g. AUR packages which require > proprietary sources (like games hidden behind paywalls). The AUR checks > the proto, to make sure that no-proto files which *are* expected to be > provided together with the PKGBUILD are not missing. > > ... > > Why don't people use, for the third case, > DLAGENTS+=('file::/usr/bin/curl -C - -o %o %u'); source=('file:///tmp/c') ? > > And what sources do we expect to have a guaranteed filesystem location, > but are not supplied by a package? I consider cp during build/package to > be perfectly adequate for files provided by some other package, and in > fact I use this here: > > https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/broadcom-wl&id=fba4c424ae9bc381f0babc61fca4643e1569de1a#n31 > Firstly, we do not consider the AUR when making development decisions. Also, I am thinking you are mixing up --source and --allsource. I am happy for a file:// protocol to be added. We already use it for dealing with local packages/repos in pacman, so it should be supported for pacman. In fact, I thought this had already been done, so was surprised when it did not work! As far as the justification for source=("foo/bar") type sources, I think reasonable arguments can be made for their inclusion. For VCS sources, I think a git submodule is a valid example. Also, consider gcc - if you download a source for (e.g.) gmp into a particular directory in its sources, it will compile it and build against that. And the simple example provided by OP of config files organised in different subdirectories relative to the PKGBUILD also seems reasonable. So, sources downloaded to subdirectories is a feature I would support implementing. However, I will not consider partial solutions - or more accurately, partial fixes that look like that will need partially reverted to implement the complete solution. On the top of my head, the complete solution covers: 1) files in subdirectories of the builddir. 2) standard sources 3) VCS sources 4) Sources with foo/bar::http:// type syntax 5) --allsource Allan
On 04/29/2018 06:17 AM, Allan McRae wrote: > Firstly, we do not consider the AUR when making development decisions. Well, yeah, I'll have to raise that separately on aur-dev based on the outcome of this discussion. (Also it is relevant at least in the sense that the nature of the AUR ensures the local:// protocol gets used in real-world cases.) > Also, I am thinking you are mixing up --source and --allsource. No, when neither file:// nor local:// are used --source should be adding it as well, which means this applies to that too AFAICT. > I am happy for a file:// protocol to be added. We already use it for > dealing with local packages/repos in pacman, so it should be supported > for pacman. In fact, I thought this had already been done, so was > surprised when it did not work! Makes sense... I do wonder how many AUR packages incorrectly use file:// (no default DLAGENT handler) instead of local:// (explicitly disables DLAGENT handling) and will therefore break as a result. We'll get more technical correctness out of this, which is neat! > As far as the justification for source=("foo/bar") type sources, I think > reasonable arguments can be made for their inclusion. For VCS sources, > I think a git submodule is a valid example. Also, consider gcc - if you > download a source for (e.g.) gmp into a particular directory in its > sources, it will compile it and build against that. And the simple > example provided by OP of config files organised in different > subdirectories relative to the PKGBUILD also seems reasonable. So, > sources downloaded to subdirectories is a feature I would support > implementing. Reasonable enough. I'd forgotten about VCS submodules, this could actually simplify handling a lot, there... But I'm not sure how we'd handle that properly. We use bare clones, so there are potential file/directory name conflicts that would not normally manifest in the git worktree (which relies on the developer layout that explicitly handles that). It's also quite ugly IMO to do bare git clones recursively into each other. BTW we've been asked for this before: https://bugs.archlinux.org/task/39718 > However, I will not consider partial solutions - or more accurately, > partial fixes that look like that will need partially reverted to > implement the complete solution. On the top of my head, the complete > solution covers: > > 1) files in subdirectories of the builddir. > 2) standard sources > 3) VCS sources > 4) Sources with foo/bar::http:// type syntax > 5) --allsource I think a solution would probably look like: 1) add file:// DLAGENTS. As you pointed out, it's useful, a common URI scheme, and should be there regardless. 2) add support for proto:// downloads to subdirectories, requires pre-creating directory layout I think, plus passing the correct paths in download_foo to the DLAGENT. All DLAGENTS should natively support writing to any relative specified path, we just don't use that. 3) ensuring extract_foo handles the requested $srcdir layout. This is where FS#39718 bites us. 4) ensure source packages correctly generate. This should be sufficient to modify: ln -s "$absfile" "$srclinks/$pkgbase" to create the right destination, and mkdir -p the ${destination/*} first.
On 04/29/2018 09:26 AM, Eli Schwartz wrote: This also helps solve https://bugs.archlinux.org/task/40118
Currently, makepkg doesn't support (actually, mishandles) sources in a subdirectory of the package directory. Though there aren't many uses for such a feature, it is useful for an otherwise source-less package of Arch- or site-specific scripts, to be installed on one or a few systems, for which it doesn't make sense to have a separate source repository. In this case, it would be nice to either move the source files in a subdirectory, or allow pulling them from somewhere else on the filesystem. Whether or not this is considered a valid usecase, the current behavior is broken (makepkg computes broken paths and attempts to operate on files that aren't there), so this probably ought to be fixed regardless. I couldn't find a test suite for makepkg (only pacman and makepkg-template), so, here is the test case: mkdir dir touch a dir/b /tmp/c cat > PKGBUILD <<'EOF' pkgname=dir-test pkgver=1 pkgrel=1 arch=('any') source=(a dir/b /tmp/c) md5sums=(SKIP SKIP SKIP) package() { cp "$srcdir"/* "$pkgdir/" ; } EOF One thing to note is that this doesn't protect against filename collisions: specifying dir1/file and dir2/file as sources will just make makepkg clobber the first file. Still, that's not new: source=(http://xxx/a http://yyy/a) are going to have the same problem. Vladimir Panteleev (2): libmakepkg: Fix handling of file paths in get_filepath libmakepkg: Use the correct path in extract_sources scripts/libmakepkg/source.sh.in | 2 +- scripts/libmakepkg/util/source.sh.in | 22 ++++++++++++++++++++-- 2 files changed, 21 insertions(+), 3 deletions(-)