From patchwork Sun Feb 18 17:17:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 405 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id D7D0521DE33E for ; Sun, 18 Feb 2018 17:17:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:57 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 38B588C7A6B9B; Sun, 18 Feb 2018 17:17:53 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Sun, 18 Feb 2018 17:17:50 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 60A612B049; Sun, 18 Feb 2018 17:17:48 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 2EBE12B03F for ; Sun, 18 Feb 2018 17:17:46 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:46 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id BCB428C7A6B78 for ; Sun, 18 Feb 2018 17:17:40 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [IPv6:2001:19f0:5c00:8069:5400:ff:fe26:6a86]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:39 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id 2EDCA80503 for ; Sun, 18 Feb 2018 12:17:38 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Sun, 18 Feb 2018 12:17:29 -0500 Message-Id: <20180218171736.4473-2-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180218171736.4473-1-lukeshu@lukeshu.com> References: <20180218171736.4473-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH v2 1/8] test: common.bash:__getCheckSum: Don't rely on IFS X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker I managed to stumble across a bug in BATS where the run() function screwed with the global IFS. This breaks __getCheckSum(). The bug has been fixed in git, but isn't in a release yet; go ahead and work around it. The code getting more robust isn't a bad thing! https://github.com/sstephenson/bats/issues/89 --- test/lib/common.bash | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/test/lib/common.bash b/test/lib/common.bash index 540e403..cad4e13 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -9,8 +9,9 @@ __updatePKGBUILD() { } __getCheckSum() { - local result=($(sha1sum $1)) - echo ${result[0]} + local result + result="$(sha1sum "$1")" + echo "${result%% *}" } __buildPackage() { From patchwork Sun Feb 18 17:17:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 408 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 04A2121DE351 for ; Sun, 18 Feb 2018 17:18:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:18:14 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id D530D8C7A6BD7; Sun, 18 Feb 2018 17:18:06 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Sun, 18 Feb 2018 17:18:00 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 7DDA72B048; Sun, 18 Feb 2018 17:17:51 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id EA64F2B04C for ; Sun, 18 Feb 2018 17:17:48 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:48 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 777AD8C7A6B76 for ; Sun, 18 Feb 2018 17:17:40 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [104.207.138.63]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:39 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id 83BC780504 for ; Sun, 18 Feb 2018 12:17:38 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Sun, 18 Feb 2018 12:17:30 -0500 Message-Id: <20180218171736.4473-3-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180218171736.4473-1-lukeshu@lukeshu.com> References: <20180218171736.4473-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH v2 2/8] test: db-update: @test "update same any package to same repository fails": change PKGEXT X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker This has the test change PKGEXT the second time it tries to release the package. Currently, this causes the tests to fail. That's a good thing; it's checking for the regression where db-functions:check_pkgrepos isn't treating PKGEXT as a glob. Without this, that regression didn't cause test failure because the checks right after it were tripping anyway. https://lists.archlinux.org/pipermail/arch-projects/2018-February/004742.html --- test/cases/db-update.bats | 3 ++- test/lib/common.bash | 7 ++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index 1da7eef..36511c9 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -92,8 +92,9 @@ load ../lib/common db-update checkPackage extra pkg-any-a - releasePackage extra pkg-any-a + PKGEXT=.pkg.tar.gz releasePackage extra pkg-any-a run db-update + [[ -z $BUILDDIR ]] || rm -f "${BUILDDIR}/$(__getCheckSum "${TMP}/svn-packages-copy/pkg-any-a/trunk/PKGBUILD")"/*.pkg.tar.gz{,.sig} [ "$status" -ne 0 ] } diff --git a/test/lib/common.bash b/test/lib/common.bash index cad4e13..d34af8a 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -14,6 +14,11 @@ __getCheckSum() { echo "${result%% *}" } +# Check if a file exists, even if the file uses wildcards +__isGlobfile() { + [[ -f $1 ]] +} + __buildPackage() { local pkgdest=${1:-.} local p @@ -24,7 +29,7 @@ __buildPackage() { if [[ -n ${BUILDDIR} ]]; then cache=${BUILDDIR}/$(__getCheckSum PKGBUILD) - if [[ -d ${cache} ]]; then + if __isGlobfile "${cache}"/*${PKGEXT}; then cp -Lv ${cache}/*${PKGEXT}{,.sig} ${pkgdest} return 0 else From patchwork Sun Feb 18 17:17:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 407 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id CA59021DE34E for ; Sun, 18 Feb 2018 17:18:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:18:09 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 8081C8C7A6BCB; Sun, 18 Feb 2018 17:18:04 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Sun, 18 Feb 2018 17:17:56 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 211292B141; Sun, 18 Feb 2018 17:17:51 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 953DC2B04B for ; Sun, 18 Feb 2018 17:17:48 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:48 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id B9DDC8C7A6B7C for ; Sun, 18 Feb 2018 17:17:43 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [104.207.138.63]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:39 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id E073280505 for ; Sun, 18 Feb 2018 12:17:38 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Sun, 18 Feb 2018 12:17:31 -0500 Message-Id: <20180218171736.4473-4-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180218171736.4473-1-lukeshu@lukeshu.com> References: <20180218171736.4473-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH v2 3/8] config: Rename PKGEXT to PKGEXT_glob X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker Unlike the other *EXT variables, which are prescriptive, PKGEXT is descriptive, and is a glob (this has always been the case). This is confusing because of the other variables, and because it is used prescriptively in makepkg.conf. Simply put, the configuration variable name PKGEXT is overloaded. Let's rename the glob version, to make things clearer. Now, in test/lib/common.bash, there *are* 2 places where it is used prescriptively. How does that work!? The value has a glob character in it! Well, because of sloppy quoting, it just kind of works out. So, in those places, *don't* rename it to PKGEXT_glob, but introduce a local PKGEXT variable with a prescriptive value. In db-update.bats, there's a place where it was set to .pkg.tar.gz both descriptively and prescriptively; so set both PKGEXT and PKGEXT_glob in that case. --- config | 4 +++- cron-jobs/ftpdir-cleanup | 6 +++--- db-functions | 4 ++-- db-move | 4 ++-- db-update | 8 ++++---- test/cases/db-repo-add.bats | 6 +++--- test/cases/db-update.bats | 4 ++-- test/cases/ftpdir-cleanup.bats | 4 ++-- test/lib/common.bash | 8 +++++--- 9 files changed, 26 insertions(+), 22 deletions(-) diff --git a/config b/config index d2c1942..13fe202 100644 --- a/config +++ b/config @@ -23,10 +23,12 @@ LOCK_TIMEOUT=300 STAGING="$HOME/staging" TMPDIR="/var/tmp" ARCHES=(x86_64) +# prescriptive DBEXT=".db.tar.gz" FILESEXT=".files.tar.gz" -PKGEXT=".pkg.tar.?z" SRCEXT=".src.tar.gz" +# descriptive; bash glob listing allowed extensions. +PKGEXT_glob=".pkg.tar.?z" # Allowed licenses: get sourceballs only for licenses in this array ALLOWED_LICENSES=('GPL' 'GPL1' 'GPL2' 'GPL3' 'LGPL' 'LGPL1' 'LGPL2' 'LGPL2.1' 'LGPL3') diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 2f3d5aa..4dc02a0 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -38,7 +38,7 @@ for repo in ${PKGREPOS[@]}; do continue fi # get a list of actual available package files - find "${FTP_BASE}/${repo}/os/${arch}" -xtype f -name "*${PKGEXT}" -printf '%f\n' | sort > "${WORKDIR}/repo-${repo}-${arch}" + find "${FTP_BASE}/${repo}/os/${arch}" -xtype f -name "*${PKGEXT_glob}" -printf '%f\n' | sort > "${WORKDIR}/repo-${repo}-${arch}" # get a list of package files defined in the repo db bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" | awk '/^%FILENAME%/{getline;print}' | sort > "${WORKDIR}/db-${repo}-${arch}" @@ -62,7 +62,7 @@ for repo in ${PKGREPOS[@]}; do done # get a list of all available packages in the pacakge pool -find "$FTP_BASE/${PKGPOOL}" -name "*${PKGEXT}" -printf '%f\n' | sort > "${WORKDIR}/pool" +find "$FTP_BASE/${PKGPOOL}" -name "*${PKGEXT_glob}" -printf '%f\n' | sort > "${WORKDIR}/pool" # create a list of packages in our db find "${WORKDIR}" -maxdepth 1 -type f -name 'db-*' -exec cat {} \; | sort -u > "${WORKDIR}/db" @@ -75,7 +75,7 @@ if [ ${#old_pkgs[@]} -ge 1 ]; then done fi -old_pkgs=($(find ${CLEANUP_DESTDIR} -type f -name "*${PKGEXT}" -mtime +${CLEANUP_KEEP} -printf '%f\n')) +old_pkgs=($(find ${CLEANUP_DESTDIR} -type f -name "*${PKGEXT_glob}" -mtime +${CLEANUP_KEEP} -printf '%f\n')) if [ ${#old_pkgs[@]} -ge 1 ]; then msg "Removing old packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do diff --git a/db-functions b/db-functions index e8949d7..84be241 100644 --- a/db-functions +++ b/db-functions @@ -374,8 +374,8 @@ check_pkgrepos() { local pkgver="$(getpkgver ${pkgfile})" || return 1 local pkgarch="$(getpkgarch ${pkgfile})" || return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT} ]] && return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT}.sig ]] && return 1 + [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT_glob} ]] && return 1 + [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT_glob}.sig ]] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/} ]] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/}.sig ]] && return 1 diff --git a/db-move b/db-move index 37a9884..e3bc16e 100755 --- a/db-move +++ b/db-move @@ -49,7 +49,7 @@ for pkgbase in ${args[@]:2}; do for pkgname in ${pkgnames[@]}; do for tarch in ${tarches[@]}; do - getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT} >/dev/null + getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT_glob} >/dev/null done done continue 2 @@ -95,7 +95,7 @@ for pkgbase in ${args[@]:2}; do for pkgname in ${pkgnames[@]}; do for tarch in ${tarches[@]}; do - pkgpath=$(getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT}) + pkgpath=$(getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT_glob}) pkgfile="${pkgpath##*/}" ln -s "../../../${PKGPOOL}/${pkgfile}" ${ftppath_to}/${tarch}/ diff --git a/db-update b/db-update index 45755a4..4afeb6e 100755 --- a/db-update +++ b/db-update @@ -9,7 +9,7 @@ if (( $# >= 1 )); then fi # Find repos with packages to release -if ! staging_repos=($(find "${STAGING}" -mindepth 1 -type f -name "*${PKGEXT}" -printf '%h\n' | sort -u)); then +if ! staging_repos=($(find "${STAGING}" -mindepth 1 -type f -name "*${PKGEXT_glob}" -printf '%h\n' | sort -u)); then die "Could not read %s" "$STAGING" fi @@ -32,7 +32,7 @@ for repo in ${repos[@]}; do if ! check_repo_permission "${repo}"; then die "You don't have permission to update packages in %s" "$repo" fi - pkgs=($(getpkgfiles "${STAGING}/${repo}/"*${PKGEXT})) + pkgs=($(getpkgfiles "${STAGING}/${repo}/"*${PKGEXT_glob})) if (( $? == 0 )); then for pkg in ${pkgs[@]}; do if [[ -h ${pkg} ]]; then @@ -70,10 +70,10 @@ done for repo in ${repos[@]}; do msg "Updating [%s]..." "$repo" - any_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-any${PKGEXT} 2>/dev/null)) + any_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-any${PKGEXT_glob} 2>/dev/null)) for pkgarch in ${ARCHES[@]}; do add_pkgs=() - arch_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-${pkgarch}${PKGEXT} 2>/dev/null)) + arch_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-${pkgarch}${PKGEXT_glob} 2>/dev/null)) for pkg in ${arch_pkgs[@]} ${any_pkgs[@]}; do pkgfile="${pkg##*/}" msg2 "${pkgfile} (${pkgarch})" diff --git a/test/cases/db-repo-add.bats b/test/cases/db-repo-add.bats index 9fde381..ace7452 100644 --- a/test/cases/db-repo-add.bats +++ b/test/cases/db-repo-add.bats @@ -14,10 +14,10 @@ __movePackageToRepo() { fi # FIXME: pkgbase might not be part of the package filename - mv -v "${STAGING}"/${repo}/${pkgbase}-*-*-${arch}${PKGEXT}{,.sig} "${FTP_BASE}/${PKGPOOL}/" + mv -v "${STAGING}"/${repo}/${pkgbase}-*-*-${arch}${PKGEXT_glob}{,.sig} "${FTP_BASE}/${PKGPOOL}/" for tarch in ${tarches[@]}; do - ln -sv ${FTP_BASE}/${PKGPOOL}/${pkgbase}-*-*-${arch}${PKGEXT} "${FTP_BASE}/${repo}/os/${tarch}/" - ln -sv ${FTP_BASE}/${PKGPOOL}/${pkgbase}-*-*-${arch}${PKGEXT}.sig "${FTP_BASE}/${repo}/os/${tarch}/" + ln -sv ${FTP_BASE}/${PKGPOOL}/${pkgbase}-*-*-${arch}${PKGEXT_glob} "${FTP_BASE}/${repo}/os/${tarch}/" + ln -sv ${FTP_BASE}/${PKGPOOL}/${pkgbase}-*-*-${arch}${PKGEXT_glob}.sig "${FTP_BASE}/${repo}/os/${tarch}/" done } diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index 36511c9..c1b8eb4 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -92,7 +92,7 @@ load ../lib/common db-update checkPackage extra pkg-any-a - PKGEXT=.pkg.tar.gz releasePackage extra pkg-any-a + PKGEXT=.pkg.tar.gz PKGEXT_glob=.pkg.tar.gz releasePackage extra pkg-any-a run db-update [[ -z $BUILDDIR ]] || rm -f "${BUILDDIR}/$(__getCheckSum "${TMP}/svn-packages-copy/pkg-any-a/trunk/PKGBUILD")"/*.pkg.tar.gz{,.sig} [ "$status" -ne 0 ] @@ -151,7 +151,7 @@ load ../lib/common @test "add invalid signed package fails" { local p releasePackage extra 'pkg-any-a' - for p in "${STAGING}"/extra/*${PKGEXT}; do + for p in "${STAGING}"/extra/*${PKGEXT_glob}; do unxz $p xz -0 ${p%%.xz} done diff --git a/test/cases/ftpdir-cleanup.bats b/test/cases/ftpdir-cleanup.bats index 6280ce0..7dfad4a 100644 --- a/test/cases/ftpdir-cleanup.bats +++ b/test/cases/ftpdir-cleanup.bats @@ -13,8 +13,8 @@ __checkRepoRemovedPackage() { local pkgname for pkgname in $(__getPackageNamesFromPackageBase ${pkgbase}); do - [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT} ]] - [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT} ]] + [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT_glob} ]] + [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT_glob} ]] done } diff --git a/test/lib/common.bash b/test/lib/common.bash index d34af8a..94fedfe 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -26,11 +26,12 @@ __buildPackage() { local pkgarches local tarch local pkgnames + local PKGEXT="${PKGEXT:-.pkg.tar.xz}" if [[ -n ${BUILDDIR} ]]; then cache=${BUILDDIR}/$(__getCheckSum PKGBUILD) - if __isGlobfile "${cache}"/*${PKGEXT}; then - cp -Lv ${cache}/*${PKGEXT}{,.sig} ${pkgdest} + if __isGlobfile "${cache}"/*${PKGEXT_glob}; then + cp -Lv ${cache}/*${PKGEXT_glob}{,.sig} ${pkgdest} return 0 else mkdir -p ${cache} @@ -174,6 +175,7 @@ checkPackageDB() { local repoarches local pkgfile local pkgname + local PKGEXT="${PKGEXT:-.pkg.tar.xz}" # FIXME: We guess the location of the PKGBUILD used for this repo # We cannot read from trunk as __updatePKGBUILD() might have bumped the version @@ -221,7 +223,7 @@ checkPackageDB() { for db in ${DBEXT} ${FILESEXT}; do [ -r "${FTP_BASE}/${repo}/os/${repoarch}/${repo}${db%.tar.*}" ] - bsdtar -xf "${FTP_BASE}/${repo}/os/${repoarch}/${repo}${db%.tar.*}" -O | grep -q "${pkgfile%${PKGEXT}}" + bsdtar -xf "${FTP_BASE}/${repo}/os/${repoarch}/${repo}${db%.tar.*}" -O | grep -q "${pkgfile%${PKGEXT_glob}}" done done done From patchwork Sun Feb 18 17:17:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 406 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id D041B21DE346 for ; Sun, 18 Feb 2018 17:18:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:18:04 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id C76BD8C7A6BB2; Sun, 18 Feb 2018 17:17:57 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Sun, 18 Feb 2018 17:17:51 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 488502B04F; Sun, 18 Feb 2018 17:17:49 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 5DADB2B048 for ; Sun, 18 Feb 2018 17:17:47 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:47 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 5BD7D8C7A6B75 for ; Sun, 18 Feb 2018 17:17:40 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [IPv6:2001:19f0:5c00:8069:5400:ff:fe26:6a86]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:40 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id 4979B80506 for ; Sun, 18 Feb 2018 12:17:39 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Sun, 18 Feb 2018 12:17:32 -0500 Message-Id: <20180218171736.4473-5-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180218171736.4473-1-lukeshu@lukeshu.com> References: <20180218171736.4473-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH v2 4/8] Correctly treat PKGEXT_glob as a glob X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker Several places treated PKGEXT as a fixed string in [[ -f ]] existence checks. Fix that elegantly by introducing an is_globfile function. This fixes the failing db-update.bats "update same any package to same repository fails" test. This is based on a patch by Eli Schwartz --- db-functions | 9 +++++++-- test/cases/ftpdir-cleanup.bats | 4 ++-- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/db-functions b/db-functions index 84be241..769d7ef 100644 --- a/db-functions +++ b/db-functions @@ -16,6 +16,11 @@ restore_umask () { umask $UMASK >/dev/null } +# Check if a file exists, even if the file uses wildcards +is_globfile() { + [[ -f $1 ]] +} + # just like mv -f, but we touch the file and then copy the content so # default ACLs in the target dir will be applied mv_acl() { @@ -374,8 +379,8 @@ check_pkgrepos() { local pkgver="$(getpkgver ${pkgfile})" || return 1 local pkgarch="$(getpkgarch ${pkgfile})" || return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT_glob} ]] && return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT_glob}.sig ]] && return 1 + is_globfile "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXT_glob} && return 1 + is_globfile "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXT_glob}.sig && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/} ]] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/}.sig ]] && return 1 diff --git a/test/cases/ftpdir-cleanup.bats b/test/cases/ftpdir-cleanup.bats index 7dfad4a..efc18a8 100644 --- a/test/cases/ftpdir-cleanup.bats +++ b/test/cases/ftpdir-cleanup.bats @@ -13,8 +13,8 @@ __checkRepoRemovedPackage() { local pkgname for pkgname in $(__getPackageNamesFromPackageBase ${pkgbase}); do - [[ ! -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-*${PKGEXT_glob} ]] - [[ ! -f ${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}-*${PKGEXT_glob} ]] + ! is_globfile "${FTP_BASE}/${PKGPOOL}/${pkgname}"-*${PKGEXT_glob} + ! is_globfile "${FTP_BASE}/${repo}/os/${repoarch}/${pkgname}"-*${PKGEXT_glob} done } From patchwork Sun Feb 18 17:17:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 411 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 396BE21DE369 for ; Sun, 18 Feb 2018 17:18:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:18:29 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 60AB48C7A6C15; Sun, 18 Feb 2018 17:18:22 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Sun, 18 Feb 2018 17:18:09 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id A9994208D5; Sun, 18 Feb 2018 17:17:59 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id EB25D2B14B for ; Sun, 18 Feb 2018 17:17:55 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:55 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 228488C7A6B82 for ; Sun, 18 Feb 2018 17:17:46 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [104.207.138.63]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:45 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id A471580507 for ; Sun, 18 Feb 2018 12:17:39 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Sun, 18 Feb 2018 12:17:33 -0500 Message-Id: <20180218171736.4473-6-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180218171736.4473-1-lukeshu@lukeshu.com> References: <20180218171736.4473-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH v2 5/8] config: let PKGEXT_glob be an extglob; have its value match makepkg X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker 1. In order to support PKGEXT_glob being an extglob, we must switch several uses of `find` (in db-update and ftpdir-cleanup) to use bash globbing instead, as `find` can't be made to use extended globbing. In general, this technique requires us to also enable nullglob (which we do globally in db-functions). Of the 4 searches that we have to rewrite: - (ftpdir-cleanup) Two were just being used to strip leading directory paths, and can be replaced by ${filepath##*/} in a for loop. - (ftpdir-cleanup) One was checking the modification time of the files, and can be replaced with touch(1) and [[ -nt ]]. Although this introduces an additional temporary file, this is not such a big deal. - (db-update) One was getting leading directory paths, this can be replaced with globstar (which we also enable globally in db-functions) and ${filepath%/*} in a for loop. We can drop the error handling that was on that search, as the old code never aborted on errors anyway, as without `set -o pipefail` the sort command swallowed the return code. 2. In addition to enabling exglob in db-functions for the main code, we also need to do it in test/lib/common.bash for the tests. Go ahead and also enable globstar and nullglob there too, for consistency with db-functions. Turning on nullglob there in turn forces us to fix up a couple of [ -f ] glob checks in sourceballs.bats. 3. Use this new extended globbing capability to set the default value of PKGEXT_glob to reflect the behavior of makepkg (v5.0.2); the old value of PKGEXT_glob both accepted things that makepkg would reject, and rejected things that makepkg would accept. This is based on patches by Eli Schwartz --- config | 4 ++-- cron-jobs/ftpdir-cleanup | 18 +++++++++++++++--- db-functions | 3 +++ db-update | 10 +++++++--- test/cases/sourceballs.bats | 4 ++-- test/lib/common.bash | 3 +++ 6 files changed, 32 insertions(+), 10 deletions(-) diff --git a/config b/config index 13fe202..d2d92ba 100644 --- a/config +++ b/config @@ -27,8 +27,8 @@ ARCHES=(x86_64) DBEXT=".db.tar.gz" FILESEXT=".files.tar.gz" SRCEXT=".src.tar.gz" -# descriptive; bash glob listing allowed extensions. -PKGEXT_glob=".pkg.tar.?z" +# descriptive; bash glob listing allowed extensions. Note that db-functions turns on extglob. +PKGEXT_glob=".pkg.tar?(.gz|.bz2|.xz|.lrz|.lzo|.Z)" # Allowed licenses: get sourceballs only for licenses in this array ALLOWED_LICENSES=('GPL' 'GPL1' 'GPL2' 'GPL3' 'LGPL' 'LGPL1' 'LGPL2' 'LGPL2.1' 'LGPL3') diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 4dc02a0..20d579f 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -38,7 +38,11 @@ for repo in ${PKGREPOS[@]}; do continue fi # get a list of actual available package files - find "${FTP_BASE}/${repo}/os/${arch}" -xtype f -name "*${PKGEXT_glob}" -printf '%f\n' | sort > "${WORKDIR}/repo-${repo}-${arch}" + for f in "${FTP_BASE}"/${repo}/os/${arch}/*${PKGEXT_glob}; do + if [[ -f $f ]]; then + printf '%s\n' "${f##*/}" + fi + done | sort > "${WORKDIR}/repo-${repo}-${arch}" # get a list of package files defined in the repo db bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" | awk '/^%FILENAME%/{getline;print}' | sort > "${WORKDIR}/db-${repo}-${arch}" @@ -62,7 +66,9 @@ for repo in ${PKGREPOS[@]}; do done # get a list of all available packages in the pacakge pool -find "$FTP_BASE/${PKGPOOL}" -name "*${PKGEXT_glob}" -printf '%f\n' | sort > "${WORKDIR}/pool" +for f in "$FTP_BASE/${PKGPOOL}"/*${PKGEXT_glob}; do + printf '%s\n' "${f##*/}" +done | sort > "${WORKDIR}/pool" # create a list of packages in our db find "${WORKDIR}" -maxdepth 1 -type f -name 'db-*' -exec cat {} \; | sort -u > "${WORKDIR}/db" @@ -75,7 +81,13 @@ if [ ${#old_pkgs[@]} -ge 1 ]; then done fi -old_pkgs=($(find ${CLEANUP_DESTDIR} -type f -name "*${PKGEXT_glob}" -mtime +${CLEANUP_KEEP} -printf '%f\n')) +old_pkgs=() +touch -d "${CLEANUP_KEEP} days ago" "${WORKDIR}/cleanup_timestamp" +for f in "${CLEANUP_DESTDIR}"/**/*${PKGEXT_glob}; do + if [[ ${WORKDIR}/cleanup_timestamp -nt $f ]]; then + old_pkgs+=("${f##*/}") + fi +done if [ ${#old_pkgs[@]} -ge 1 ]; then msg "Removing old packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do diff --git a/db-functions b/db-functions index 769d7ef..8ca9205 100644 --- a/db-functions +++ b/db-functions @@ -2,6 +2,9 @@ . /usr/share/makepkg/util.sh +# global shell options for enhanced bash scripting +shopt -s extglob globstar nullglob + # Some PKGBUILDs need CARCH to be set CARCH="x86_64" diff --git a/db-update b/db-update index 4afeb6e..37cffbf 100755 --- a/db-update +++ b/db-update @@ -9,9 +9,13 @@ if (( $# >= 1 )); then fi # Find repos with packages to release -if ! staging_repos=($(find "${STAGING}" -mindepth 1 -type f -name "*${PKGEXT_glob}" -printf '%h\n' | sort -u)); then - die "Could not read %s" "$STAGING" -fi +readarray -t staging_repos < <( + for f in "${STAGING}"/**/*${PKGEXT}; do + if [[ -f $f ]]; then + printf '%s\n' "${f%/*}" + fi + done | sort -u +) repos=() for staging_repo in ${staging_repos[@]##*/}; do diff --git a/test/cases/sourceballs.bats b/test/cases/sourceballs.bats index a0a2999..df7ddd4 100644 --- a/test/cases/sourceballs.bats +++ b/test/cases/sourceballs.bats @@ -2,12 +2,12 @@ load ../lib/common __checkSourcePackage() { local pkgbase=$1 - [ -r ${FTP_BASE}/${SRCPOOL}/${pkgbase}-*${SRCEXT} ] + __isGlobfile "${FTP_BASE}/${SRCPOOL}/${pkgbase}"-*"${SRCEXT}" } __checkRemovedSourcePackage() { local pkgbase=$1 - [ ! -r ${FTP_BASE}/${SRCPOOL}/${pkgbase}-*${SRCEXT} ] + ! __isGlobfile "${FTP_BASE}/${SRCPOOL}/${pkgbase}"-*"${SRCEXT}" } @test "create simple package sourceballs" { diff --git a/test/lib/common.bash b/test/lib/common.bash index 94fedfe..9d6edeb 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -1,5 +1,8 @@ . /usr/share/makepkg/util.sh +# global shell options for enhanced bash scripting +shopt -s extglob globstar nullglob + __updatePKGBUILD() { local pkgrel From patchwork Sun Feb 18 17:17:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 409 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id D22FA21DE359 for ; Sun, 18 Feb 2018 17:18:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:18:16 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id C2D338C7A6BE7; Sun, 18 Feb 2018 17:18:09 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Sun, 18 Feb 2018 17:18:03 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id C9F672B04C; Sun, 18 Feb 2018 17:17:54 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 111AF2B04E for ; Sun, 18 Feb 2018 17:17:53 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:53 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id B945F8C7A6B80 for ; Sun, 18 Feb 2018 17:17:45 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [104.207.138.63]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:45 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id 0BC3E80508 for ; Sun, 18 Feb 2018 12:17:40 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Sun, 18 Feb 2018 12:17:34 -0500 Message-Id: <20180218171736.4473-7-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180218171736.4473-1-lukeshu@lukeshu.com> References: <20180218171736.4473-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH v2 6/8] ftpdir-cleanup: fix typo in a comment ("pacakge") X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker This is based on a patch by Eli Schwartz --- cron-jobs/ftpdir-cleanup | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 20d579f..64d50ef 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -65,7 +65,7 @@ for repo in ${PKGREPOS[@]}; do done done -# get a list of all available packages in the pacakge pool +# get a list of all available packages in the package pool for f in "$FTP_BASE/${PKGPOOL}"/*${PKGEXT_glob}; do printf '%s\n' "${f##*/}" done | sort > "${WORKDIR}/pool" From patchwork Sun Feb 18 17:17:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 412 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id F33FB21DE36C for ; Sun, 18 Feb 2018 17:18:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:18:31 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 309668C7A6C1E; Sun, 18 Feb 2018 17:18:25 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Sun, 18 Feb 2018 17:18:12 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id EDD772B050; Sun, 18 Feb 2018 17:18:02 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 8A8C02078A for ; Sun, 18 Feb 2018 17:18:01 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:18:01 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 84E398C7A6B93 for ; Sun, 18 Feb 2018 17:17:52 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [IPv6:2001:19f0:5c00:8069:5400:ff:fe26:6a86]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:47 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id 68A0A80509 for ; Sun, 18 Feb 2018 12:17:40 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Sun, 18 Feb 2018 12:17:35 -0500 Message-Id: <20180218171736.4473-8-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180218171736.4473-1-lukeshu@lukeshu.com> References: <20180218171736.4473-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH v2 7/8] Replace all instances of `find` command with bash globbing X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker This fully removes the use of find from the codebase, and leads to a micro-optimization in a couple cases. Of the 5 instances in these files: - (ftpdir-cleanup) One was unnecessary as `cat` can natively consume all files passed to it (from a glob) and no directory traversal was in use. We don't have to worry about the glob being empty, unless config:ARCHES or config:PKGREPOS is mis/non-configured and is empty. - (sourceballs) One was only being used to strip leading directory paths, and can be replaced by ${filepath##*/} in a for loop. - (sourceballs) Two were unnecessary as they were hardcoded to read a single file. - (sourceballs) One was checking the modification time of the files, and can be replaced with touch(1) and [[ -nt ]]. Although this introduces an additional temporary file, this is not such a big deal. This is based on patches by Eli Schwartz --- cron-jobs/ftpdir-cleanup | 2 +- cron-jobs/sourceballs | 18 ++++++++++++++---- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 64d50ef..6a6c82f 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -70,7 +70,7 @@ for f in "$FTP_BASE/${PKGPOOL}"/*${PKGEXT_glob}; do printf '%s\n' "${f##*/}" done | sort > "${WORKDIR}/pool" # create a list of packages in our db -find "${WORKDIR}" -maxdepth 1 -type f -name 'db-*' -exec cat {} \; | sort -u > "${WORKDIR}/db" +cat "${WORKDIR}"/db-* 2>/dev/null | sort -u > "${WORKDIR}/db" old_pkgs=($(comm -23 "${WORKDIR}/pool" "${WORKDIR}/db")) if [ ${#old_pkgs[@]} -ge 1 ]; then diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index 9ab4e98..c47d5ad 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -46,7 +46,11 @@ for repo in ${PKGREPOS[@]}; do done # Create a list of all available source package file names -find "${FTP_BASE}/${SRCPOOL}" -xtype f -name "*${SRCEXT}" -printf '%f\n' | sort -u > "${WORKDIR}/available-src-pkgs" +for f in "${FTP_BASE}"/${SRCPOOL}/*${SRCEXT}; do + if [[ -f $f ]]; then + printf '%s\n' "${f##*/}" + fi +done | sort -u > "${WORKDIR}/available-src-pkgs" # Check for all packages if we need to build a source package for repo in ${PKGREPOS[@]}; do @@ -117,8 +121,8 @@ for repo in ${PKGREPOS[@]}; do done # Cleanup old source packages -find "${WORKDIR}" -maxdepth 1 -type f -name 'expected-src-pkgs' -exec cat {} \; | sort -u > "${WORKDIR}/expected-src-pkgs.sort" -find "${WORKDIR}" -maxdepth 1 -type f -name 'available-src-pkgs' -exec cat {} \; | sort -u > "${WORKDIR}/available-src-pkgs.sort" +sort -u "${WORKDIR}/expected-src-pkgs" > "${WORKDIR}/expected-src-pkgs.sort" +sort -u "${WORKDIR}/available-src-pkgs" > "${WORKDIR}/available-src-pkgs.sort" old_pkgs=($(comm -23 "${WORKDIR}/available-src-pkgs.sort" "${WORKDIR}/expected-src-pkgs.sort")) if [ ${#old_pkgs[@]} -ge 1 ]; then @@ -133,7 +137,13 @@ if [ ${#old_pkgs[@]} -ge 1 ]; then done fi -old_pkgs=($(find ${SOURCE_CLEANUP_DESTDIR} -type f -name "*${SRCEXT}" -mtime +${SOURCE_CLEANUP_KEEP} -printf '%f\n')) +old_pkgs=() +touch -d "${SOURCE_CLEANUP_KEEP} days ago" "${WORKDIR}/cleanup_timestamp" +for f in "${SOURCE_CLEANUP_DESTDIR}"/*${SRCEXT}; do + if [[ ${WORKDIR}/cleanup_timestamp -nt $f ]]; then + old_pkgs+=("${f##*/}") + fi +done if [ ${#old_pkgs[@]} -ge 1 ]; then msg "Removing old source packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do From patchwork Sun Feb 18 17:17:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luke Shumaker X-Patchwork-Id: 410 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 70B7221DE35E for ; Sun, 18 Feb 2018 17:18:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:18:22 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 5FD598C7A6BFB; Sun, 18 Feb 2018 17:18:15 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Sun, 18 Feb 2018 17:18:06 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 5902A20792; Sun, 18 Feb 2018 17:17:59 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=none Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 3A8122B14E for ; Sun, 18 Feb 2018 17:17:56 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:56 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id DB8108C7A6B8D for ; Sun, 18 Feb 2018 17:17:48 +0000 (UTC) Received: from mav.lukeshu.com (mav.lukeshu.com [IPv6:2001:19f0:5c00:8069:5400:ff:fe26:6a86]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Sun, 18 Feb 2018 17:17:47 +0000 (UTC) Received: from build64-par (unknown [IPv6:2601:803:202:9275:da50:e6ff:fe00:4a5b]) by mav.lukeshu.com (Postfix) with ESMTPSA id C22368050A for ; Sun, 18 Feb 2018 12:17:40 -0500 (EST) From: Luke Shumaker To: arch-projects@archlinux.org Date: Sun, 18 Feb 2018 12:17:36 -0500 Message-Id: <20180218171736.4473-9-lukeshu@lukeshu.com> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180218171736.4473-1-lukeshu@lukeshu.com> References: <20180218171736.4473-1-lukeshu@lukeshu.com> Subject: [arch-projects] [dbscripts] [PATCH v2 8/8] ftpdir-cleanup, sourceballs: swap out [ -ge 1 ] for (( > 0 )) X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Arch Linux projects development discussion Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" From: Luke Shumaker This is based on a patch by Eli Schwartz --- cron-jobs/ftpdir-cleanup | 2 +- cron-jobs/sourceballs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 6a6c82f..fac05bd 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -88,7 +88,7 @@ for f in "${CLEANUP_DESTDIR}"/**/*${PKGEXT_glob}; do old_pkgs+=("${f##*/}") fi done -if [ ${#old_pkgs[@]} -ge 1 ]; then +if (( ${#old_pkgs[@]} > 0 )); then msg "Removing old packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do msg2 "${old_pkg}" diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index c47d5ad..d8c0a43 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -144,7 +144,7 @@ for f in "${SOURCE_CLEANUP_DESTDIR}"/*${SRCEXT}; do old_pkgs+=("${f##*/}") fi done -if [ ${#old_pkgs[@]} -ge 1 ]; then +if (( ${#old_pkgs[@]} > 0 )); then msg "Removing old source packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do msg2 "${old_pkg}"