From patchwork Mon Feb 19 20:11:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 413 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 7867421FD220 for ; Mon, 19 Feb 2018 20:11:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:28 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 95AE28D0ED39F; Mon, 19 Feb 2018 20:11:20 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Mon, 19 Feb 2018 20:11:20 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 4767F2D83D; Mon, 19 Feb 2018 20:11:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1519071079; bh=dSBypzZYj0Bwrze3/lRRZYk1G/w6ZMMxXboLA0/B+ng=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=ebzUGUSzAjTUPTSmumeNvT9ztI6YK2p0Sd0T156GglrxRR5e2q2axMWtwtHgYivQP gLql3/M4j64zl4rp3FuXSjpaZo3gsyc+mLBYCltzc5YTzrdCxuqKxndDwFsWJhGA28 fkjAFA3rgDBx7YnA4+/fg71g70706tpoWh72JiVG5aH1GO8PRoimFOHmLD42Ao+mdO aIaduqSn9Fjq9WGRMBQtUZfN0NAYrn6EkyU9wVDXLGATf69+Emc77cu+dSF9MOfm5s Qv73dX1dH3imFUMBnNi+8lE24FO1EOCA0M5eNopHBhWEkGQBJgX+16LhHgobN7f08m RQJUMzXBGFzgAfA17Ohsin3kpdY0cGJy3PTnQKIZEDG+a399cldmipmMq4CqlXDowR JH5iOEJpZyq8sNedESEx1rtyh+4hKLRDajBNORnp5/hPW3kjb4GkGTIkFRlEo/R0O5 Flghs4/Cj49u1No+EzMEYn+OWmJPGfAYJK5k0bjKgg2GDN8K4EIQH8tgONe14O0tpy z4jTps01kMgioG4WZgjbV0vYKGud/XtaYPrHpSicRqXffX1PhjumYh7HCI9b5O3sWz qtMmZhzwNde9Qgd3YVBu/kt+pne4ILeprgu0BPyvoTQ75eGOL+/9mwaFRV4FzITDsv VK2D/O86YN2SQP4cJxm1pJuA= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 183662D4C0 for ; Mon, 19 Feb 2018 20:11:16 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:16 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 6892A8D0ED386; Mon, 19 Feb 2018 20:11:06 +0000 (UTC) Received: from didactylos.localdomain (unknown [216.97.144.179]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id E816E8D0ED385; Mon, 19 Feb 2018 20:11:05 +0000 (UTC) To: arch-projects@archlinux.org Date: Mon, 19 Feb 2018 15:11:41 -0500 Message-Id: <20180219201145.14057-2-eschwartz@archlinux.org> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180219201145.14057-1-eschwartz@archlinux.org> References: <20180216034504.27610-1-eschwartz@archlinux.org> <20180219201145.14057-1-eschwartz@archlinux.org> Subject: [arch-projects] [dbscripts] [PATCH v2 1/5] Use even more bashisms. X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" Catch some cases that were missed in the previous run. Signed-off-by: Eli Schwartz --- This patch is new + refactor some changes from: ftpdir-cleanup,sourceballs: replace external find command with bash globbing cron-jobs/devlist-mailer | 6 +++--- cron-jobs/ftpdir-cleanup | 14 +++++++------- cron-jobs/integrity-check | 2 +- cron-jobs/sourceballs | 12 ++++++------ cron-jobs/update-web-db | 6 +++--- 5 files changed, 20 insertions(+), 20 deletions(-) diff --git a/cron-jobs/devlist-mailer b/cron-jobs/devlist-mailer index ca2e46b..65a5483 100755 --- a/cron-jobs/devlist-mailer +++ b/cron-jobs/devlist-mailer @@ -7,17 +7,17 @@ LIST="arch-dev-public@archlinux.org" FROM="repomaint@archlinux.org" SUBJECT="Repository Maintenance $(date +"%d-%m-%Y")" -if [ $# -ge 1 ]; then +if (( $# >= 1 )); then SUBJECT="$1 $(date +"%d-%m-%Y")" fi -if [ $# -ge 2 ]; then +if (( $# >= 2 )); then LIST="$2" fi stdin="$(cat)" #echo used to strip whitespace for checking for actual data -if [ -n "$(echo $stdin)" ]; then +if [[ -n "$(echo $stdin)" ]]; then echo "Subject: $SUBJECT To: $LIST diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 2f3d5aa..c771950 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -9,11 +9,11 @@ clean_pkg() { if ! ${CLEANUP_DRYRUN}; then for pkg in "$@"; do - if [ -h "$pkg" ]; then + if [[ -h $pkg ]]; then rm -f "$pkg" "$pkg.sig" else mv_acl "$pkg" "$CLEANUP_DESTDIR/${pkg##*/}" - if [ -e "$pkg.sig" ]; then + if [[ -e $pkg.sig ]]; then mv_acl "$pkg.sig" "$CLEANUP_DESTDIR/${pkg##*/}.sig" fi touch "${CLEANUP_DESTDIR}/${pkg##*/}" @@ -34,7 +34,7 @@ ${CLEANUP_DRYRUN} && warning 'dry run mode is active' for repo in ${PKGREPOS[@]}; do for arch in ${ARCHES[@]}; do - if [ ! -f "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" ]; then + if [[ ! -f ${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT} ]]; then continue fi # get a list of actual available package files @@ -43,7 +43,7 @@ for repo in ${PKGREPOS[@]}; do bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" | awk '/^%FILENAME%/{getline;print}' | sort > "${WORKDIR}/db-${repo}-${arch}" missing_pkgs=($(comm -13 "${WORKDIR}/repo-${repo}-${arch}" "${WORKDIR}/db-${repo}-${arch}")) - if [ ${#missing_pkgs[@]} -ge 1 ]; then + if (( ${#missing_pkgs[@]} >= 1 )); then error "Missing packages in [%s] (%s)..." "$repo" "$arch" for missing_pkg in ${missing_pkgs[@]}; do msg2 "${missing_pkg}" @@ -51,7 +51,7 @@ for repo in ${PKGREPOS[@]}; do fi old_pkgs=($(comm -23 "${WORKDIR}/repo-${repo}-${arch}" "${WORKDIR}/db-${repo}-${arch}")) - if [ ${#old_pkgs[@]} -ge 1 ]; then + if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from [%s] (%s)..." "$repo" "$arch" for old_pkg in ${old_pkgs[@]}; do msg2 "${old_pkg}" @@ -67,7 +67,7 @@ find "$FTP_BASE/${PKGPOOL}" -name "*${PKGEXT}" -printf '%f\n' | sort > "${WORKDI find "${WORKDIR}" -maxdepth 1 -type f -name 'db-*' -exec cat {} \; | sort -u > "${WORKDIR}/db" old_pkgs=($(comm -23 "${WORKDIR}/pool" "${WORKDIR}/db")) -if [ ${#old_pkgs[@]} -ge 1 ]; then +if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from package pool..." for old_pkg in ${old_pkgs[@]}; do msg2 "${old_pkg}" @@ -76,7 +76,7 @@ if [ ${#old_pkgs[@]} -ge 1 ]; then fi old_pkgs=($(find ${CLEANUP_DESTDIR} -type f -name "*${PKGEXT}" -mtime +${CLEANUP_KEEP} -printf '%f\n')) -if [ ${#old_pkgs[@]} -ge 1 ]; then +if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do msg2 "${old_pkg}" diff --git a/cron-jobs/integrity-check b/cron-jobs/integrity-check index 211a24b..f1e75ab 100755 --- a/cron-jobs/integrity-check +++ b/cron-jobs/integrity-check @@ -7,7 +7,7 @@ dirname="$(dirname $0)" script_lock -if [ $# -ne 1 ]; then +if (( $# != 1 )); then die "usage: ${0##*/} " fi mailto=$1 diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index 9ab4e98..25a8abb 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -21,7 +21,7 @@ renice +10 -p $$ > /dev/null for repo in ${PKGREPOS[@]}; do for arch in ${ARCHES[@]}; do # Repo does not exist; skip it - if [ ! -f "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" ]; then + if [[ ! -f ${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT} ]]; then continue fi bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" \ @@ -81,7 +81,7 @@ for repo in ${PKGREPOS[@]}; do mkdir -p -m0770 "${WORKDIR}/pkgbuilds/${repo}-${pkgarch}" arch_svn export -q "${SVNREPO}/${pkgbase}/repos/${repo}-${pkgarch}" \ "${WORKDIR}/pkgbuilds/${repo}-${pkgarch}/${pkgbase}" >/dev/null 2>&1 - if [ $? -ge 1 ]; then + if (( $? >= 1 )); then failedpkgs+=("${pkgbase}-${pkgver}${SRCEXT}") continue fi @@ -89,7 +89,7 @@ for repo in ${PKGREPOS[@]}; do # Build the actual source package pushd "${WORKDIR}/pkgbuilds/${repo}-${pkgarch}/${pkgbase}" >/dev/null makepkg --nocolor --allsource --ignorearch --skippgpcheck --config "${dirname}/makepkg.conf" >"${WORKDIR}/${pkgbase}.log" 2>&1 - if [ $? -eq 0 ] && [ -f "${pkgbase}-${pkgver}${SRCEXT}" ]; then + if (( $? == 0 )) && [[ -f ${pkgbase}-${pkgver}${SRCEXT} ]]; then mv_acl "${pkgbase}-${pkgver}${SRCEXT}" "${FTP_BASE}/${SRCPOOL}/${pkgbase}-${pkgver}${SRCEXT}" # Avoid creating the same source package for every arch echo "${pkgbase}-${pkgver}${SRCEXT}" >> "${WORKDIR}/available-src-pkgs" @@ -121,7 +121,7 @@ find "${WORKDIR}" -maxdepth 1 -type f -name 'expected-src-pkgs' -exec cat {} \; find "${WORKDIR}" -maxdepth 1 -type f -name 'available-src-pkgs' -exec cat {} \; | sort -u > "${WORKDIR}/available-src-pkgs.sort" old_pkgs=($(comm -23 "${WORKDIR}/available-src-pkgs.sort" "${WORKDIR}/expected-src-pkgs.sort")) -if [ ${#old_pkgs[@]} -ge 1 ]; then +if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old source packages..." ${SOURCE_CLEANUP_DRYRUN} && warning 'dry run mode is active' for old_pkg in ${old_pkgs[@]}; do @@ -134,7 +134,7 @@ if [ ${#old_pkgs[@]} -ge 1 ]; then fi old_pkgs=($(find ${SOURCE_CLEANUP_DESTDIR} -type f -name "*${SRCEXT}" -mtime +${SOURCE_CLEANUP_KEEP} -printf '%f\n')) -if [ ${#old_pkgs[@]} -ge 1 ]; then +if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old source packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do msg2 "${old_pkg}" @@ -142,7 +142,7 @@ if [ ${#old_pkgs[@]} -ge 1 ]; then done fi -if [ -f "${WORKDIR}/makepkg-fail.log" ]; then +if [[ -f ${WORKDIR}/makepkg-fail.log ]]; then msg "Log of failed packages" cat "${WORKDIR}/makepkg-fail.log" fi diff --git a/cron-jobs/update-web-db b/cron-jobs/update-web-db index 23af067..c859ce0 100755 --- a/cron-jobs/update-web-db +++ b/cron-jobs/update-web-db @@ -27,7 +27,7 @@ renice +5 -p $$ > /dev/null echo "%s: Updating DB at %s" "$cmd" "$(date)" >> "${LOGOUT}" # source our virtualenv if it exists -if [ -f "$ENVPATH" ]; then +if [[ -f "$ENVPATH" ]]; then . "$ENVPATH" fi @@ -47,7 +47,7 @@ for repo in ${REPOS[@]}; do for arch in ${ARCHES[@]}; do repo_lock ${repo} ${arch} || exit 1 dbfile="/srv/ftp/${repo}/os/${arch}/${repo}${dbfileext}" - if [ -f "${dbfile}" ]; then + if [[ -f ${dbfile} ]]; then mkdir -p "${WORKDIR}/${repo}/${arch}" cp "${dbfile}" "${WORKDIR}/${repo}/${arch}/${repo}${dbfileext}" fi @@ -60,7 +60,7 @@ pushd $SPATH >/dev/null for repo in ${REPOS[@]}; do for arch in ${ARCHES[@]}; do dbcopy="${WORKDIR}/${repo}/${arch}/${repo}${dbfileext}" - if [ -f "${dbcopy}" ]; then + if [[ -f ${dbcopy} ]]; then echo "Updating ${repo}-${arch}" >> "${LOGOUT}" ./manage.py reporead ${flags} ${arch} "${dbcopy}" >> "${LOGOUT}" 2>&1 echo "" >> "${LOGOUT}" From patchwork Mon Feb 19 20:11:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 415 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 67C7921FD22A for ; Mon, 19 Feb 2018 20:11:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:36 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 02FFC8D0ED3C4; Mon, 19 Feb 2018 20:11:28 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Mon, 19 Feb 2018 20:11:27 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id C1A4A2D91B; Mon, 19 Feb 2018 20:11:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1519071080; bh=RrNw0oMuWVZ8gf4p0BAMq/eD1htbwCetcA8DOmtDkGQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=B0K8xcpQOBT9iobRHXzaDbjZsVcCn7lI0LioqFrBoFZmxXXSqqh1e7N/4PQX7X6al jWNbrmaQBL83wqdrcLjBuDYGb20sSYd5HUbSsyTk7FiIU5h3zTG5iPfP2Gyp5tLmMb dIIlulwYAoR63gzTh7NUQD0Efp3tgUPaXYCx9kM0dg8mjNqLn69QqwUEYlQ/V4eUEa M4kZMexBmki8F/qHq1QLzkf4XNYIl7+cZj+ePsmVm/o1Ctlu7EuC7K8vgnclnjw1rm jG/YmkFLgOFiluFhOxtfVfxuZWaYzVSExpG6W4ag3sQKd126Zc2+OB2JFj5eDAhrE8 TcKzbCJxwNscratdcA47rtrAS6CIAsIXnJRL0ivkQlHV/9S7xBMZ/FQoq+NGdyyWW5 QS5rhkd5meWTuWG4KWytP8julehctJVrFAj5/YuDioxqGA97jOR06WaIsS8Y46NE9b v2MKYMYAxDofybt0HilgTg4FZGqNRV2IqQCNky61ZepD3ufXYaS+X9yJvWJWHot+fI o6ivzPdU3PmaXX4zMYLjTuiCxSwiKL+BPA9lPlXkgKQWmWlQXGcvVRNN0xTGAcohR6 xV1MKi+ly1eLNqmGDJ0xl5HmGZiCjPARdR4lEKtnwgTAQY1RmXfG47WYiUcks+N0WL RKws4Aoyb9nBZlp48gfxZfM0= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id BCB9A2D82A for ; Mon, 19 Feb 2018 20:11:18 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:18 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 1B6938D0ED388; Mon, 19 Feb 2018 20:11:07 +0000 (UTC) Received: from didactylos.localdomain (unknown [216.97.144.179]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id 946E78D0ED387; Mon, 19 Feb 2018 20:11:06 +0000 (UTC) To: arch-projects@archlinux.org Date: Mon, 19 Feb 2018 15:11:42 -0500 Message-Id: <20180219201145.14057-3-eschwartz@archlinux.org> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180219201145.14057-1-eschwartz@archlinux.org> References: <20180216034504.27610-1-eschwartz@archlinux.org> <20180219201145.14057-1-eschwartz@archlinux.org> Subject: [arch-projects] [dbscripts] [PATCH v2 2/5] Fix overloading PKGEXT to mean two things. X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" PKGEXT is a makepkg variable referring to a fixed filename suffix, but we were also using it to mean a bash glob referring to candidate filenames. This is wrong, so rename it to PKGEXTS which is more descriptive of its purpose. Exclude the testsuite from this change, as the testsuite actually uses PKGEXT for its intended purpose. Fix the testsuite to consistently use PKGEXT, as it hardcoded the file extension in several cases, and extract its value from the makepkg.conf we ship. Signed-off-by: Eli Schwartz --- This is new, and renders some old things obsolete. I see no need to jump through more hoops than strictly necessary, plus, lots of this was technically broken anyway -- but it worked because again, the testsuite is not using PKGEXT in the dbscripts sense, rather in the makepkg sense. I'm hoping some of the logic there can be replaced with my --packagelist fixes for makepkg 5.1 as well. config | 2 +- cron-jobs/ftpdir-cleanup | 6 +++--- db-functions | 4 ++-- db-move | 4 ++-- db-update | 8 ++++---- test/cases/db-repo-add.bats | 6 +++--- test/cases/db-update.bats | 3 +-- test/cases/ftpdir-cleanup.bats | 6 +++--- test/lib/common.bash | 6 ++++-- 9 files changed, 23 insertions(+), 22 deletions(-) diff --git a/config b/config index d2c1942..5bb3b16 100644 --- a/config +++ b/config @@ -25,7 +25,7 @@ TMPDIR="/var/tmp" ARCHES=(x86_64) DBEXT=".db.tar.gz" FILESEXT=".files.tar.gz" -PKGEXT=".pkg.tar.?z" +PKGEXTS=".pkg.tar.?z" SRCEXT=".src.tar.gz" # Allowed licenses: get sourceballs only for licenses in this array diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index c771950..630efa8 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -38,7 +38,7 @@ for repo in ${PKGREPOS[@]}; do continue fi # get a list of actual available package files - find "${FTP_BASE}/${repo}/os/${arch}" -xtype f -name "*${PKGEXT}" -printf '%f\n' | sort > "${WORKDIR}/repo-${repo}-${arch}" + find "${FTP_BASE}/${repo}/os/${arch}" -xtype f -name "*${PKGEXTS}" -printf '%f\n' | sort > "${WORKDIR}/repo-${repo}-${arch}" # get a list of package files defined in the repo db bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" | awk '/^%FILENAME%/{getline;print}' | sort > "${WORKDIR}/db-${repo}-${arch}" @@ -62,7 +62,7 @@ for repo in ${PKGREPOS[@]}; do done # get a list of all available packages in the pacakge pool -find "$FTP_BASE/${PKGPOOL}" -name "*${PKGEXT}" -printf '%f\n' | sort > "${WORKDIR}/pool" +find "$FTP_BASE/${PKGPOOL}" -name "*${PKGEXTS}" -printf '%f\n' | sort > "${WORKDIR}/pool" # create a list of packages in our db find "${WORKDIR}" -maxdepth 1 -type f -name 'db-*' -exec cat {} \; | sort -u > "${WORKDIR}/db" @@ -75,7 +75,7 @@ if (( ${#old_pkgs[@]} >= 1 )); then done fi -old_pkgs=($(find ${CLEANUP_DESTDIR} -type f -name "*${PKGEXT}" -mtime +${CLEANUP_KEEP} -printf '%f\n')) +old_pkgs=($(find ${CLEANUP_DESTDIR} -type f -name "*${PKGEXTS}" -mtime +${CLEANUP_KEEP} -printf '%f\n')) if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do diff --git a/db-functions b/db-functions index e8949d7..e8eb2bc 100644 --- a/db-functions +++ b/db-functions @@ -374,8 +374,8 @@ check_pkgrepos() { local pkgver="$(getpkgver ${pkgfile})" || return 1 local pkgarch="$(getpkgarch ${pkgfile})" || return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT} ]] && return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXT}.sig ]] && return 1 + [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXTS} ]] && return 1 + [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXTS}.sig ]] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/} ]] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/}.sig ]] && return 1 diff --git a/db-move b/db-move index 37a9884..fb7ebac 100755 --- a/db-move +++ b/db-move @@ -49,7 +49,7 @@ for pkgbase in ${args[@]:2}; do for pkgname in ${pkgnames[@]}; do for tarch in ${tarches[@]}; do - getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT} >/dev/null + getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXTS} >/dev/null done done continue 2 @@ -95,7 +95,7 @@ for pkgbase in ${args[@]:2}; do for pkgname in ${pkgnames[@]}; do for tarch in ${tarches[@]}; do - pkgpath=$(getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXT}) + pkgpath=$(getpkgfile "${ftppath_from}/${tarch}/"${pkgname}-${pkgver}-${pkgarch}${PKGEXTS}) pkgfile="${pkgpath##*/}" ln -s "../../../${PKGPOOL}/${pkgfile}" ${ftppath_to}/${tarch}/ diff --git a/db-update b/db-update index 45755a4..a8d885a 100755 --- a/db-update +++ b/db-update @@ -9,7 +9,7 @@ if (( $# >= 1 )); then fi # Find repos with packages to release -if ! staging_repos=($(find "${STAGING}" -mindepth 1 -type f -name "*${PKGEXT}" -printf '%h\n' | sort -u)); then +if ! staging_repos=($(find "${STAGING}" -mindepth 1 -type f -name "*${PKGEXTS}" -printf '%h\n' | sort -u)); then die "Could not read %s" "$STAGING" fi @@ -32,7 +32,7 @@ for repo in ${repos[@]}; do if ! check_repo_permission "${repo}"; then die "You don't have permission to update packages in %s" "$repo" fi - pkgs=($(getpkgfiles "${STAGING}/${repo}/"*${PKGEXT})) + pkgs=($(getpkgfiles "${STAGING}/${repo}/"*${PKGEXTS})) if (( $? == 0 )); then for pkg in ${pkgs[@]}; do if [[ -h ${pkg} ]]; then @@ -70,10 +70,10 @@ done for repo in ${repos[@]}; do msg "Updating [%s]..." "$repo" - any_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-any${PKGEXT} 2>/dev/null)) + any_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-any${PKGEXTS} 2>/dev/null)) for pkgarch in ${ARCHES[@]}; do add_pkgs=() - arch_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-${pkgarch}${PKGEXT} 2>/dev/null)) + arch_pkgs=($(getpkgfiles "${STAGING}/${repo}/"*-${pkgarch}${PKGEXTS} 2>/dev/null)) for pkg in ${arch_pkgs[@]} ${any_pkgs[@]}; do pkgfile="${pkg##*/}" msg2 "${pkgfile} (${pkgarch})" diff --git a/test/cases/db-repo-add.bats b/test/cases/db-repo-add.bats index 9fde381..ac91058 100644 --- a/test/cases/db-repo-add.bats +++ b/test/cases/db-repo-add.bats @@ -31,7 +31,7 @@ __movePackageToRepo() { releasePackage extra ${pkgbase} for arch in ${arches[@]}; do __movePackageToRepo extra ${pkgbase} ${arch} - db-repo-add extra ${arch} ${pkgbase}-1-1-${arch}.pkg.tar.xz + db-repo-add extra ${arch} ${pkgbase}-1-1-${arch}${PKGEXT} done done @@ -54,7 +54,7 @@ __movePackageToRepo() { add_pkgs=() for pkgbase in ${pkgs[@]}; do __movePackageToRepo extra ${pkgbase} ${arch} - add_pkgs+=("${pkgbase}-1-1-${arch}.pkg.tar.xz") + add_pkgs+=("${pkgbase}-1-1-${arch}${PKGEXT}") done db-repo-add extra ${arch} ${add_pkgs[@]} done @@ -73,7 +73,7 @@ __movePackageToRepo() { for pkgbase in ${pkgs[@]}; do releasePackage extra ${pkgbase} __movePackageToRepo extra ${pkgbase} any - db-repo-add extra any ${pkgbase}-1-1-any.pkg.tar.xz + db-repo-add extra any ${pkgbase}-1-1-any${PKGEXT} done for pkgbase in ${pkgs[@]}; do diff --git a/test/cases/db-update.bats b/test/cases/db-update.bats index 1da7eef..349b195 100644 --- a/test/cases/db-update.bats +++ b/test/cases/db-update.bats @@ -151,8 +151,7 @@ load ../lib/common local p releasePackage extra 'pkg-any-a' for p in "${STAGING}"/extra/*${PKGEXT}; do - unxz $p - xz -0 ${p%%.xz} + printf '%s\n' "Not a real package" | gpg -v --detach-sign --no-armor --use-agent - > "${p}.sig" done run db-update [ "$status" -ne 0 ] diff --git a/test/cases/ftpdir-cleanup.bats b/test/cases/ftpdir-cleanup.bats index 6280ce0..fd485f3 100644 --- a/test/cases/ftpdir-cleanup.bats +++ b/test/cases/ftpdir-cleanup.bats @@ -82,7 +82,7 @@ __checkRepoRemovedPackage() { db-remove extra any pkg-any-a ftpdir-cleanup - local pkg1='pkg-any-a-1-1-any.pkg.tar.xz' + local pkg1="pkg-any-a-1-1-any${PKGEXT}" checkRemovedPackage extra 'pkg-any-a' for arch in ${arches[@]}; do __checkRepoRemovedPackage extra 'pkg-any-a' ${arch} @@ -138,8 +138,8 @@ __checkRepoRemovedPackage() { ftpdir-cleanup - local pkgfilea="pkg-simple-a-1-1-${arch}.pkg.tar.xz" - local pkgfileb="pkg-simple-b-1-1-${arch}.pkg.tar.xz" + local pkgfilea="pkg-simple-a-1-1-${arch}${PKGEXT}" + local pkgfileb="pkg-simple-b-1-1-${arch}${PKGEXT}" for arch in ${arches[@]}; do touch -d "-$(expr ${CLEANUP_KEEP} + 1)days" ${CLEANUP_DESTDIR}/${pkgfilea}{,.sig} done diff --git a/test/lib/common.bash b/test/lib/common.bash index 540e403..1b257ab 100644 --- a/test/lib/common.bash +++ b/test/lib/common.bash @@ -34,9 +34,9 @@ __buildPackage() { pkgarches=($(. PKGBUILD; echo ${arch[@]})) for tarch in ${pkgarches[@]}; do if [ "${tarch}" == 'any' ]; then - PKGDEST=${pkgdest} makepkg -c + PKGDEST=${pkgdest} makepkg --config "$MAKEPKG_CONF" -c else - PKGDEST=${pkgdest} CARCH=${tarch} makepkg -c + PKGDEST=${pkgdest} CARCH=${tarch} makepkg --config "$MAKEPKG_CONF" -c fi done @@ -82,6 +82,8 @@ setup() { local pkg local r local a + MAKEPKG_CONF="$PWD/../cron-jobs/makepkg.conf" + PKGEXT="$(. "$MAKEPKG_CONF"; printf '%s' "$PKGEXT")" TMP="$(mktemp -d)" From patchwork Mon Feb 19 20:11:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 416 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 667FC21FD229 for ; Mon, 19 Feb 2018 20:11:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:36 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id C42038D0ED3B6; Mon, 19 Feb 2018 20:11:27 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Mon, 19 Feb 2018 20:11:24 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 8D0A82D90A; Mon, 19 Feb 2018 20:11:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1519071080; bh=JYMb7Oy+IMiKDdvvdTB23DRUk5QHTgYujqaF9Z47prk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=eRI1lj4eXrOFxGkB/mnJXHO++RLcI29H3my7VrVnjxtvnHKzXPQ9R9O+Jiuwdqw+M iXwKLbLcugAf98j4qCbd5ffq/V7SNLkTwA9L5sU3IHwYOYUnKnXQp2ILwfNjR1lqru DI5DoSigSW5mZ7c7d5csfJANBBlwVYMP0+h/pFlvg6plwK+X13rHoO7HZUdMDc78td uW9s1YjRFI52tQtVNAjWPk3ME+6wyhwTGbxhwuvZVsXiFUdNl6x+zGz4mtJ13NsErd /n2Ybmpf4Ea4uJ8F1jVPM0ab4asl36mmus6hCZdxkws14Surm4JxZggPZRCObDZXxf 4uHT3QiRYrh8j7b54xOBdlaVAxZuR2wISuqFtwghuJkDIqmGrykZfVb40CvbjL/JsD lScSc+1AyiCUykrbQl2vy256YlpTWwjBZc6ADlif/83Nla6uz5JScfmjJ/ILCkKtfl cCKeuhj5alE8p3eO9SpLBuBckJNkz8deUP0XmjC3sQTfxBNIQmZLoYCqZQtSNFn//1 oHHCmewJxkK7ZGeOUU5qbNbigokM56SYPlK3honVjIda4C2atjMliy3ymCsiipcCHU TqGq97nViMP3dYfO5N/PhCplFNWklusB7xas2lSp3fegO4GBHUL0o+BatvY5xoY312 9cw7c4LKSwKhzzmdyLlbvjfg= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 6A5EA2D4C0 for ; Mon, 19 Feb 2018 20:11:18 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:18 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id B9FC88D0ED38A; Mon, 19 Feb 2018 20:11:07 +0000 (UTC) Received: from didactylos.localdomain (unknown [216.97.144.179]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id 442268D0ED389; Mon, 19 Feb 2018 20:11:07 +0000 (UTC) To: arch-projects@archlinux.org Date: Mon, 19 Feb 2018 15:11:43 -0500 Message-Id: <20180219201145.14057-4-eschwartz@archlinux.org> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180219201145.14057-1-eschwartz@archlinux.org> References: <20180216034504.27610-1-eschwartz@archlinux.org> <20180219201145.14057-1-eschwartz@archlinux.org> Subject: [arch-projects] [dbscripts] [PATCH v2 3/5] db-update: replace external find command with bash globbing X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" Don't bother emitting errors. bash doesn't show globbing errors if it cannot read a directory to try globbing there. And the former code never aborted on errors anyway, as without `set -o pipefail` the sort command swallowed the return code. Signed-off-by: Eli Schwartz --- v2: use mapfile as suggested by Luke, rather than running in_array in a loop. db-functions | 4 ++++ db-update | 11 ++++++++--- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/db-functions b/db-functions index e8eb2bc..394c7a2 100644 --- a/db-functions +++ b/db-functions @@ -2,6 +2,10 @@ . /usr/share/makepkg/util.sh +# global shell options for enhanced bash scripting +shopt -s globstar nullglob + + # Some PKGBUILDs need CARCH to be set CARCH="x86_64" diff --git a/db-update b/db-update index a8d885a..db12df8 100755 --- a/db-update +++ b/db-update @@ -9,9 +9,14 @@ if (( $# >= 1 )); then fi # Find repos with packages to release -if ! staging_repos=($(find "${STAGING}" -mindepth 1 -type f -name "*${PKGEXTS}" -printf '%h\n' | sort -u)); then - die "Could not read %s" "$STAGING" -fi +mapfile -t -d '' staging_repos < <( + for f in "${STAGING}"/**/*${PKGEXTS}; do + f="${f%/*}" + if [[ -d $f ]]; then + printf '%s\0' "$f" + fi + done | sort -uz +) repos=() for staging_repo in ${staging_repos[@]##*/}; do From patchwork Mon Feb 19 20:11:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 417 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id BB2B021FD237 for ; Mon, 19 Feb 2018 20:11:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:43 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 56C1C8D0ED3DA; Mon, 19 Feb 2018 20:11:36 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Mon, 19 Feb 2018 20:11:30 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 147D92D97D; Mon, 19 Feb 2018 20:11:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1519071081; bh=xZQ0P5UBAj67MoENr6mOuPYAo3WJgtm3TEzLplfodlE=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=A9MgBbEwZ7UH+vhK0ayW99raMl/nW5uccMUO8pTIOOg2OpvsGTaMOy5ma+aIr5YBh k4U2rMGiN+BXBDLvdsKF6P9jSY8CDnnA5uK9wmWo688RT4J6M/0d81nZ2pwZefAhVq wLgvpvcyHiGts2vN06Frn7yQJgEeEGt9gwa5yeKFNHc0+rp0klz2mfbLfumhOjhz34 0aQw8m23RVjmkgHTK8H03w81W+ilEbAkLjRY5GK5CZKKtzyn3IbhdPWltUd/FRWXc2 Gk9/DLGOE9MmQlkoP4klykxiBRC6Bl7bX4S9qyENroKlx0jq7bizA1vQSFFzk91x0e n8KS1E8rZrZcZUc+V9sQ9UA4QB4WS4zpSCT6IZAWSvzwH+NXqk1B/StQSMyhKSfNpC xGwd8koFhCzU2UaZZQ2KDKg9NuyUzlp0TeluQrBa0ElN3N223dXEPSpV/jN9OXnP2m uZZ076RxDmbV+h+UL6/QXJlb3ZV2lGu16rmLo3RwLn8iJE12QpmlGnKxRBRXAxyt2l Hu8hFYetSaEHkVdSuVvgKcOtykWhCv4dNl7f33XAjPv9GqhovtHYWRm5v+uNXj2QoM IL4/AgTQObobJyRjXMkCQCxa4pQSJN5ZHaku3jJMqy/EqeDufjUKmaNfQXnmW0DaGd rpY8+DFnAAWyDNA6/dr5cUX0= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 4C6472D894 for ; Mon, 19 Feb 2018 20:11:19 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:19 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 6C41C8D0ED38C; Mon, 19 Feb 2018 20:11:08 +0000 (UTC) Received: from didactylos.localdomain (unknown [216.97.144.179]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id E52778D0ED38B; Mon, 19 Feb 2018 20:11:07 +0000 (UTC) To: arch-projects@archlinux.org Date: Mon, 19 Feb 2018 15:11:44 -0500 Message-Id: <20180219201145.14057-5-eschwartz@archlinux.org> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180219201145.14057-1-eschwartz@archlinux.org> References: <20180216034504.27610-1-eschwartz@archlinux.org> <20180219201145.14057-1-eschwartz@archlinux.org> Subject: [arch-projects] [dbscripts] [PATCH v2 4/5] ftpdir-cleanup, sourceballs: replace external find command with bash globbing X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" This fully removes the use of find from the codebase, leads to a micro-optimization in a couple cases, and ensures that $PKGEXT is consistently treated as a shell globbing character (which is important because it is used as one). Of the eight instances in these files: - One was unnecessary as `cat` can natively consume all files passed to it and no directory traversal was in use. - Two were unnecessary as they were hardcoded to read a single file.... - Another four were only being used to strip leading directory paths, and can be replaced by globstar and ${filepath##*/} - The final two were checking the modification time of the files, and can be replaced with touch(1) and [[ -nt ]]. Although this introduces an additional temporary file, this is not such a big deal. Signed-off-by: Eli Schwartz --- v2: split out unrelated bash cleanup changes into patch 1/5 cron-jobs/ftpdir-cleanup | 22 +++++++++++++++++----- cron-jobs/sourceballs | 19 +++++++++++++++---- 2 files changed, 32 insertions(+), 9 deletions(-) diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 630efa8..ff65d46 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -38,7 +38,11 @@ for repo in ${PKGREPOS[@]}; do continue fi # get a list of actual available package files - find "${FTP_BASE}/${repo}/os/${arch}" -xtype f -name "*${PKGEXTS}" -printf '%f\n' | sort > "${WORKDIR}/repo-${repo}-${arch}" + for f in "${FTP_BASE}"/${repo}/os/${arch}/*${PKGEXTS}; do + if [[ -f $f ]]; then + printf '%s\n' "${f##*/}" + fi + done | sort > "${WORKDIR}/repo-${repo}-${arch}" # get a list of package files defined in the repo db bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" | awk '/^%FILENAME%/{getline;print}' | sort > "${WORKDIR}/db-${repo}-${arch}" @@ -61,10 +65,12 @@ for repo in ${PKGREPOS[@]}; do done done -# get a list of all available packages in the pacakge pool -find "$FTP_BASE/${PKGPOOL}" -name "*${PKGEXTS}" -printf '%f\n' | sort > "${WORKDIR}/pool" +# get a list of all available packages in the package pool +for f in "$FTP_BASE/${PKGPOOL}"/*${PKGEXTS}; do + printf '%s\n' "${f##*/}" +done | sort > "${WORKDIR}/pool" # create a list of packages in our db -find "${WORKDIR}" -maxdepth 1 -type f -name 'db-*' -exec cat {} \; | sort -u > "${WORKDIR}/db" +cat "${WORKDIR}"/db-* 2>/dev/null | sort -u > "${WORKDIR}/db" old_pkgs=($(comm -23 "${WORKDIR}/pool" "${WORKDIR}/db")) if (( ${#old_pkgs[@]} >= 1 )); then @@ -75,7 +81,13 @@ if (( ${#old_pkgs[@]} >= 1 )); then done fi -old_pkgs=($(find ${CLEANUP_DESTDIR} -type f -name "*${PKGEXTS}" -mtime +${CLEANUP_KEEP} -printf '%f\n')) +unset old_pkgs +touch -d "${CLEANUP_KEEP} days ago" "${WORKDIR}/cleanup_timestamp" +for f in "${CLEANUP_DESTDIR}"/**/*${PKGEXTS}; do + if [[ ${WORKDIR}/cleanup_timestamp -nt $f ]]; then + old_pkgs+=("${f##*/}") + fi +done if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index 25a8abb..8f089b3 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -46,7 +46,11 @@ for repo in ${PKGREPOS[@]}; do done # Create a list of all available source package file names -find "${FTP_BASE}/${SRCPOOL}" -xtype f -name "*${SRCEXT}" -printf '%f\n' | sort -u > "${WORKDIR}/available-src-pkgs" +for f in "${FTP_BASE}"/${SRCPOOL}/*${SRCEXT}; do + if [[ -f $f ]]; then + printf '%s\n' "${f##*/}" + fi +done | sort -u > "${WORKDIR}/available-src-pkgs" # Check for all packages if we need to build a source package for repo in ${PKGREPOS[@]}; do @@ -117,8 +121,8 @@ for repo in ${PKGREPOS[@]}; do done # Cleanup old source packages -find "${WORKDIR}" -maxdepth 1 -type f -name 'expected-src-pkgs' -exec cat {} \; | sort -u > "${WORKDIR}/expected-src-pkgs.sort" -find "${WORKDIR}" -maxdepth 1 -type f -name 'available-src-pkgs' -exec cat {} \; | sort -u > "${WORKDIR}/available-src-pkgs.sort" +sort -u "${WORKDIR}/expected-src-pkgs" > "${WORKDIR}/expected-src-pkgs.sort" +sort -u "${WORKDIR}/available-src-pkgs" > "${WORKDIR}/available-src-pkgs.sort" old_pkgs=($(comm -23 "${WORKDIR}/available-src-pkgs.sort" "${WORKDIR}/expected-src-pkgs.sort")) if (( ${#old_pkgs[@]} >= 1 )); then @@ -133,7 +137,14 @@ if (( ${#old_pkgs[@]} >= 1 )); then done fi -old_pkgs=($(find ${SOURCE_CLEANUP_DESTDIR} -type f -name "*${SRCEXT}" -mtime +${SOURCE_CLEANUP_KEEP} -printf '%f\n')) +unset old_pkgs +touch -d "${SOURCE_CLEANUP_KEEP} days ago" "${WORKDIR}/cleanup_timestamp" +for f in "${SOURCE_CLEANUP_DESTDIR}"/*${SRCEXT}; do + if [[ ${WORKDIR}/cleanup_timestamp -nt $f ]]; then + old_pkgs+=("${f##*/}") + fi +done + if (( ${#old_pkgs[@]} >= 1 )); then msg "Removing old source packages from the cleanup directory..." for old_pkg in ${old_pkgs[@]}; do From patchwork Mon Feb 19 20:11:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 418 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 3A7CB21FD240 for ; Mon, 19 Feb 2018 20:11:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:49 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 771A08D0ED3EC; Mon, 19 Feb 2018 20:11:41 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Mon, 19 Feb 2018 20:11:33 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 5BEED2DB49; Mon, 19 Feb 2018 20:11:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1519071086; bh=T9G/MzqweEzaeChLdBjIzg3Cpxpnmcd9IzP/XPbOSW0=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=oGFlrIVSmks7xOKjucDLd6MOrKLK6J2JwytNoD/m9aVLlkfkk4vfWc+625rgG7Iq7 lUfWgIVvewjg+SXh6K80nStGvmqSEs8jdLSQxFioVBSaSVa+6c2Ckftm/LNFGPJXbH GmhgCReAEw/U1aSshRxe3pv/AUwU5km0egMxPrcZ+BkkloZrUkRMOLLHOug73W56p3 1SOmv0x84v63IHxaTF/DC4JuZCu3SLhZz0eYzbmXumoVSrCf3nASFuXL+Bvih8iLRs SRn4TNZJU9d6QJZ1DdUi6b5VRj9J+ivXBqQAcjVMRlChEPW+AP77BwRyLdv5zVYjkp 3pzPiRhdEWemtrPO+1dzlXeDOObNrIoXjaLrRP0+kMsmMSussMQpD7582IqAcyJHxX LpKvTfuGXKzdzwQZp0Q0hY7KTCWoeeQ0v8nUypuxy/H13Kow9/9bzy126msGQts9to Re88XSWIM0l7sDyXRlAH2cognrl784SoHmntBybvKG3yUAdQQaahiROzymTnT1D4dq 9CPITsQwZJCy8U/GnWk4x/7sLxs/fpTfVKBlXTXDhnjyRJF1mYSGp6dSqRueqIn5WV ruEYy91Zhah50axQG1tcSg5Vkjvi9SCore95V953kxM2ELjqT+LjaX2arQzNNPpGgm mT7JWnh5Rz9PG+DIYmnXfP/M= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 8E49C2DA16 for ; Mon, 19 Feb 2018 20:11:24 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Mon, 19 Feb 2018 20:11:24 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 0CE838D0ED390; Mon, 19 Feb 2018 20:11:15 +0000 (UTC) Received: from didactylos.localdomain (unknown [216.97.144.179]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id 958988D0ED38D; Mon, 19 Feb 2018 20:11:08 +0000 (UTC) To: arch-projects@archlinux.org Date: Mon, 19 Feb 2018 15:11:45 -0500 Message-Id: <20180219201145.14057-6-eschwartz@archlinux.org> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180219201145.14057-1-eschwartz@archlinux.org> References: <20180216034504.27610-1-eschwartz@archlinux.org> <20180219201145.14057-1-eschwartz@archlinux.org> MIME-Version: 1.0 Subject: [arch-projects] [dbscripts] [PATCH v2 5/5] Globally set $PKGEXT to a bash extended glob representing valid choices. X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" The current glob `*.pkg.tar.?z` is both less restrictive and more restrictive than makepkg, as it accepts any valid unicode character. To be more exact, it's almost completely orthogonal to the one in makepkg. makepkg only accepts .tar.gz, .tar.bz2, .tar.xz, .tar.lzo, .tar.lrz, and .tar.Z and most of those fail to match against a two-char compression type. dbscripts accepts .pkg.tar.💩z which incidentally is what I think of cherry-picking xz and gz as supported methods. Since this can be anything makepkg.conf accepts, it needs to be able to match all that, unless we decide to perform additional restrictions in which case we should still explicitly list each allowed extension. Using bash extended globbing allows us to do this relatively painlessly. Document the fact that this has *always* been some sort of glob, and update the two cases where this was (not!) being evaluated by bash [[ ... ]], to use a not-elegant-at-all proxy function is_globfile() to evaluate globs *before* testing if they exist. Signed-off-by: Eli Schwartz --- No changes, just a reworded commit message. config | 3 ++- db-functions | 11 ++++++++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/config b/config index 5bb3b16..0d33de0 100644 --- a/config +++ b/config @@ -25,7 +25,8 @@ TMPDIR="/var/tmp" ARCHES=(x86_64) DBEXT=".db.tar.gz" FILESEXT=".files.tar.gz" -PKGEXTS=".pkg.tar.?z" +# bash glob listing allowed extensions. Note that db-functions turns on extglob. +PKGEXTS=".pkg.tar.@(gz|bz2|xz|lzo|lrz|Z)" SRCEXT=".src.tar.gz" # Allowed licenses: get sourceballs only for licenses in this array diff --git a/db-functions b/db-functions index 394c7a2..e36d43b 100644 --- a/db-functions +++ b/db-functions @@ -3,7 +3,7 @@ . /usr/share/makepkg/util.sh # global shell options for enhanced bash scripting -shopt -s globstar nullglob +shopt -s extglob globstar nullglob # Some PKGBUILDs need CARCH to be set @@ -20,6 +20,11 @@ restore_umask () { umask $UMASK >/dev/null } +# Check if a file exists, even if the file uses wildcards +is_globfile() { + [[ -f $1 ]] +} + # just like mv -f, but we touch the file and then copy the content so # default ACLs in the target dir will be applied mv_acl() { @@ -378,8 +383,8 @@ check_pkgrepos() { local pkgver="$(getpkgver ${pkgfile})" || return 1 local pkgarch="$(getpkgarch ${pkgfile})" || return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXTS} ]] && return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXTS}.sig ]] && return 1 + is_globfile "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXTS} && return 1 + is_globfile "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXTS}.sig && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/} ]] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/}.sig ]] && return 1