From patchwork Thu Feb 15 19:49:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 392 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 8BB45218903B for ; Thu, 15 Feb 2018 19:50:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Thu, 15 Feb 2018 19:50:12 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 9FB788AC71C7D; Thu, 15 Feb 2018 19:50:04 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Thu, 15 Feb 2018 19:50:01 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 8A4782BBB9; Thu, 15 Feb 2018 19:49:57 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id B6AB92B85E for ; Thu, 15 Feb 2018 19:49:53 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Thu, 15 Feb 2018 19:49:53 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 20F098AC71C64 for ; Thu, 15 Feb 2018 19:49:48 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [IPv6:2001:19f0:5c00:8069:5400:ff:fe26:6a86]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Thu, 15 Feb 2018 19:49:48 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id D5CD880503 for ; Thu, 15 Feb 2018 14:49:46 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Thu, 15 Feb 2018 14:49:45 -0500 Message-Id: <20180215194946.20820-2-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180215194946.20820-1-lukeshu@lukeshu.com> References: <20180215194946.20820-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH 1/2] Don't quote $PKGEXT X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker This partially reverts commit b61a7148eaf546a31597b74d9dd8948e4a39dea1. In dbscripts, PKGEXT is a glob pattern--it needs to be "unquoted"; and The magic-quoting done by `[[ ... ]]` breaks that. The closing-quote coming before ${PKGEXT} was quite intentional. --- db-functions | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/db-functions b/db-functions index e8949d7..93a5af3 100644 --- a/db-functions +++ b/db-functions @@ -374,8 +374,8 @@ check_pkgrepos() { local pkgver="$(getpkgver ${pkgfile})" || return 1 local pkgarch="$(getpkgarch ${pkgfile})" || return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT} ]] && return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT}.sig ]] && return 1 + [ -f "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXT} ] && return 1 + [ -f "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXT}.sig ] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/} ]] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/}.sig ]] && return 1 From patchwork Thu Feb 15 19:49:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 391 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 37298218902C for ; Thu, 15 Feb 2018 19:50:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Thu, 15 Feb 2018 19:50:07 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id BB5888AC71C71; Thu, 15 Feb 2018 19:49:58 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Thu, 15 Feb 2018 19:49:58 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 0E0B72B8F8; Thu, 15 Feb 2018 19:49:56 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 229372B837 for ; Thu, 15 Feb 2018 19:49:53 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Thu, 15 Feb 2018 19:49:53 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 556F28AC71C65 for ; Thu, 15 Feb 2018 19:49:48 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [104.207.138.63]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Thu, 15 Feb 2018 19:49:48 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id 34A8E80504 for ; Thu, 15 Feb 2018 14:49:47 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Thu, 15 Feb 2018 14:49:46 -0500 Message-Id: <20180215194946.20820-3-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180215194946.20820-1-lukeshu@lukeshu.com> References: <20180215194946.20820-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH 2/2] config: Rename PKGEXT to PKGEXT_glob X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker Unlike the other *EXT variables, which are prescriptive, PKGEXT is descriptive, and is a blob. This is confusing because of the other variables, and because it is used prescriptively in makepkg.conf. Simply, the configuration variable name PKGEXT is overloaded. Now, in test/lib/common.bash, there *are* 2 places where it is used prescriptively. How does that work!? The value has a glob character in it! Well, because of sloppy quoting, it just kind of works out. So, in those places, *don't* rename it to PKGEXT_glob, but introduce a local PKGEXT variable with a prescriptive value. --- config | 4 +++- cron-jobs/ftpdir-cleanup | 6 +++--- db-functions | 4 ++-- db-move | 4 ++-- db-update | 8 ++++---- test/cases/db-repo-add.bats | 6 +++--- test/cases/db-update.bats | 2 +- test/cases/ftpdir-cleanup.bats | 4 ++-- test/lib/common.bash | 6 ++++-- 9 files changed, 24 insertions(+), 20 deletions(-) diff --git a/config b/config index d2c1942..d41850e 100644 --- a/config +++ b/config @@ -23,10 +23,12 @@ LOCK_TIMEOUT=300 STAGING="$HOME/staging" TMPDIR="/var/tmp" ARCHES=(x86_64) +# prescriptive DBEXT=".db.tar.gz" FILESEXT=".files.tar.gz" -PKGEXT=".pkg.tar.?z" SRCEXT=".src.tar.gz" +# descriptive +PKGEXT_glob=".pkg.tar.?z" # Allowed licenses: get sourceballs only for licenses in this array ALLOWED_LICENSES=('GPL' 'GPL1' 'GPL2' 'GPL3' 'LGPL' 'LGPL1' 'LGPL2' 'LGPL2.1' 'LGPL3') diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 2f3d5aa..4dc02a0 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -38,7 +38,7 @@ for repo in ${PKGREPOS[@]}; do continue fi # get a list of actual available package files - find "${FTP_BASE}/${repo}/os/${arch}" -xtype f -name "*${PKGEXT}" -printf '%f\n' | sort > "${WORKDIR}/repo-${repo}-${arch}" + find "${FTP_BASE}/${repo}/os/${arch}" -xtype f -name "*${PKGEXT_glob}" -printf '%f\n' | sort > "${WORKDIR}/repo-${repo}-${arch}" # get a list of package files defined in the repo db bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" | awk '/^%FILENAME%/{getline;print}' | sort > "${WORKDIR}/db-${repo}-${arch}" @@ -62,7 +62,7 @@ for repo in ${PKGREPOS[@]}; do done # get a list of all available packages in the pacakge pool -find "$FTP_BASE/${PKGPOOL}" -name "*${PKGEXT}" -printf '%f\n' | sort > "${WORKDIR}/pool" +find "$FTP_BASE/${PKGPOOL}" -name "*${PKGEXT_glob}" -printf '%f\n' | sort > "${WORKDIR}/pool" # create a list of packages in our db find "${WORKDIR}" -maxdepth 1 -type f -name 'db-*' -exec cat {} \; | sort -u > "${WORKDIR}/db" @@ -75,7 +75,7 @@ if [ ${#old_pkgs[@]} -ge 1 ]; then done fi -old_pkgs=($(find ${CLEANUP_DESTDIR} -type f -name "*${PKGEXT}" -mtime +${CLEANUP_KEEP} -printf '%f\n')) +old_pkgs=($(find ${CLEANUP_DESTDIR} -type f -name "*${PKGEXT_glob}" -mtime +${CLEANUP_KEEP} -printf '%f\n')) if [ ${#old_pkgs[@]} -ge 1 ]; then msg "Removing old packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do diff --git a/db-functions b/db-functions index 93a5af3..6f2d131 100644 --- a/db-functions +++ b/db-functions @@ -374,8 +374,8 @@ check_pkgrepos() { local pkgver="$(getpkgver ${pkgfile})" || return 1 local pkgarch="$(getpkgarch ${pkgfile})" || return 1 - [ -f "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXT} ] && return 1 - [ -f "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXT}.sig ] && return 1 + [ -f "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXT_glob} ] && return 1 + [ -f "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXT_glob}.sig ] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/} ]] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/}.sig ]] && return 1 diff --git a/db-move b/db-move index 37a9884..e3bc16e 100755 --- a/db-move +++ b/db-move @@ -49,7 +49,7 @@ for pkgbase in ${args[@]:2}; do for pkgname in ${pkgnames[@]}; do for tarch in ${tarches[@]}; do - getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT} >/dev/null + getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT_glob} >/dev/null done done continue 2 @@ -95,7 +95,7 @@ for pkgbase in ${args[@]:2}; do for pkgname in ${pkgnames[@]}; do for tarch in ${tarches[@]}; do - pkgpath=$(getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT}) + pkgpath=$(getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT_glob}) pkgfile="${pkgpath##*/}" ln -s "../../../${PKGPOOL}/${pkgfile}" ${ftppath_to}/${tarch}/ diff --git a/db-update b/db-update index 45755a4..4afeb6e 100755 --- a/db-update +++ b/db-update @@ -9,7 +9,7 @@ if (( $# >= 1 )); then fi # Find repos with packages to release -if ! staging_repos=($(find "${STAGING}" -mindepth 1 -type f -name "*${PKGEXT}" -printf '%h\n' | sort -u)); then +if ! staging_repos=($(find "${STAGING}" -mindepth 1 -type f -name "*${PKGEXT_glob}" -printf '%h\n' | sort -u)); then die "Could not read %s" "$STAGING" fi @@ -32,7 +32,7 @@ for repo in ${repos[@]}; do if ! check_repo_permission "${repo}"; then die "You don't have permission to update packages in %s" "$repo" fi - pkgs=($(getpkgfiles "${STAGING}/${repo}/"*${PKGEXT})) + pkgs=($(getpkgfiles "${STAGING}/${repo}/"*${PKGEXT_glob})) if (( $? == 0 )); then for pkg in ${pkgs[@]}; do if [[ -h ${pkg} ]]; then @@ -70,10 +70,10 @@ done for repo in ${repos[@]}; do msg "Updating [%s]..." "$repo" - any_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-any${PKGEXT} 2>/dev/null)) + any_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-any${PKGEXT_glob} 2>/dev/null)) for pkgarch in ${ARCHES[@]}; do add_pkgs=() - arch_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-${pkgarch}${PKGEXT} 2>/dev/null)) + arch_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-${pkgarch}${PKGEXT_glob} 2>/dev/null)) for pkg in ${arch_pkgs[@]} ${any_pkgs[@]}; do pkgfile="${pkg##*/}" msg2 "${pkgfile} (${pkgarch})" diff --git a/test/cases/db-repo-add.bats b/test/cases/db-repo-add.bats index 9fde381..ace7452 100644 --- a/test/cases/db-repo-add.bats +++ b/test/cases/db-repo-add.bats @@ -14,10 +14,10 @@ __movePackageToRepo() { fi # FIXME: pkgbase might not be part of the package filename - mv -v "${STAGING}"/${repo}/${pkgbase}-*-*-${arch}${PKGEXT}{,.sig} "${FTP_BASE}/${PKGPOOL}/" + mv -v "${STAGING}"/${repo}/${pkgbase}-*-*-${arch}${PKGEXT_glob}{,.sig} "${FTP_BASE}/${PKGPOOL}/" for tarch in ${tarches[@]}; do - ln -sv ${FTP_BASE}/${PKGPOOL}/${pkgbase}-*-*-${arch}${PKGEXT} "${FTP_BASE}/${repo}/os/${tarch}/" - ln -sv ${FTP_BASE}/${PKGPOOL}/${pkgbase}-*-*-${arch}${PKGEXT}.sig "${FTP_BASE}/${repo}/os/${tarch}/" + ln -sv ${FTP_BASE}/${PKGPOOL}/${pkgbase}-*-*-${arch}${PKGEXT_glob} "${FTP_BASE}/${repo}/os/${tarch}/" + ln -sv ${FTP_BASE}/${PKGPOOL}/${pkgbase}-*-*-${arch}${PKGEXT_glob}.sig "${FTP_BASE}/${repo}/os/${tarch}/" done } diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index 1da7eef..bc0a7b2 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -150,7 +150,7 @@ load ../lib/common @test "add invalid signed package fails" { local p releasePackage extra 'pkg-any-a' - for p in "${STAGING}"/extra/*${PKGEXT}; do + for p in "${STAGING}"/extra/*${PKGEXT_glob}; do unxz $p xz -0 ${p%%.xz} done diff --git a/test/cases/ftpdir-cleanup.bats b/test/cases/ftpdir-cleanup.bats index 6280ce0..7dfad4a 100644 --- a/test/cases/ftpdir-cleanup.bats +++ b/test/cases/ftpdir-cleanup.bats @@ -13,8 +13,8 @@ __checkRepoRemovedPackage() { local pkgname for pkgname in $(__getPackageNamesFromPackageBase ${pkgbase}); do - [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT} ]] - [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT} ]] + [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT_glob} ]] + [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT_glob} ]] done } diff --git a/test/lib/common.bash b/test/lib/common.bash index 540e403..abc88bd 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -20,11 +20,12 @@ __buildPackage() { local pkgarches local tarch local pkgnames + PKGEXT='.pkg.tar.xz' if [[ -n ${BUILDDIR} ]]; then cache=${BUILDDIR}/$(__getCheckSum PKGBUILD) if [[ -d ${cache} ]]; then - cp -Lv ${cache}/*${PKGEXT}{,.sig} ${pkgdest} + cp -Lv ${cache}/*${PKGEXT_glob}{,.sig} ${pkgdest} return 0 else mkdir -p ${cache} @@ -168,6 +169,7 @@ checkPackageDB() { local repoarches local pkgfile local pkgname + local PKGEXT='.pkg.tar.xz' # FIXME: We guess the location of the PKGBUILD used for this repo # We cannot read from trunk as __updatePKGBUILD() might have bumped the version @@ -215,7 +217,7 @@ checkPackageDB() { for db in ${DBEXT} ${FILESEXT}; do [ -r "${FTP_BASE}/${repo}/os/${repoarch}/${repo}${db%.tar.*}" ] - bsdtar -xf "${FTP_BASE}/${repo}/os/${repoarch}/${repo}${db%.tar.*}" -O | grep -q "${pkgfile%${PKGEXT}}" + bsdtar -xf "${FTP_BASE}/${repo}/os/${repoarch}/${repo}${db%.tar.*}" -O | grep -q "${pkgfile%${PKGEXT_glob}}" done done done