From patchwork Tue Feb 20 01:57:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 420 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 954FF220385E for ; Tue, 20 Feb 2018 01:57:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.0.10] [127.0.0.10] [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Tue, 20 Feb 2018 01:57:18 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id CD9AE8D2B3A6C; Tue, 20 Feb 2018 01:57:09 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Tue, 20 Feb 2018 01:57:09 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 9ADCD2C36F; Tue, 20 Feb 2018 01:57:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1519091829; bh=gPdMQNmq2ufjeJWPeZhJv/2Bl3SGIlvIXr6gfpZXpzk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=FqpwQHG381GEMuH6YTzfP+g27rpFqMgtexERRvp8uHQIWKe121Xc45NDCJFmtgVot ZyxQwzRg037eJsjIDopk8Np7p2Oly22aLBIxI8WROwr+5zXSgtcSAgLvKEjQbT84Zq WVLpVt2ov8ZipJph+2hNiPE9AC6gs+tSpZ95rm/6LQPKY/b038bTY8S/N4kqd0L26v 63wYSYLIola/exrfF4680ZXTRHr8O3Ewf7FZR24mCiqjlJ7kDkTjfJXZgjwxJ0T87d E50IlOKGeQ0Rd1U645rmimz4rIHhHLoRCWiluIW9vGBVnhgLFVyD9+cU2N4KyE2YuV mg9HPcLrxBqi2T6HtTiVZEfrEGJ9uvLGFdE+nAo7WUmox4d17c2rb3EKcDo4SjhpaS ln+Xvpnopsb4AliY3dFY7/95Q/+idtOeR+LHHPz4MioV5yC5oGJUD6bgaQPa6p2/R4 04S5TsDw5VPOLH+7UPrNuKVzHF+hOvqLNchXpOr4cOXe6+9YaFpHJVxc8N2KHSQ2kh hz3gwMWZIiELc1N8jLkofvAo5fA3XSYVTuiahOJfemCStqtj3BnplX2EZQhc4aehWM YEGUxKlKU25KFFh7qmXTly9GWn3TtAiRfzI3YMkRzIxIGnCzKOlTIhV+81olZjiHpF pgmNBL9eR/HQcnCTI9GZYkWs= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 5F87F2BDFE for ; Tue, 20 Feb 2018 01:57:07 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Tue, 20 Feb 2018 01:57:07 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 482AF8D2B3A68; Tue, 20 Feb 2018 01:56:56 +0000 (UTC) Received: from localhost.localdomain (ool-18bdf232.dyn.optonline.net [24.189.242.50]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id B42C78D2B3A67; Tue, 20 Feb 2018 01:56:55 +0000 (UTC) To: arch-projects@archlinux.org Date: Mon, 19 Feb 2018 20:57:34 -0500 Message-Id: <20180220015734.29867-1-eschwartz@archlinux.org> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180219201145.14057-4-eschwartz@archlinux.org> References: <20180219201145.14057-4-eschwartz@archlinux.org> Subject: [arch-projects] [dbscripts] [PATCH v3 3/5] db-update: replace external find command with bash globbing X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" Don't bother emitting errors. bash doesn't show globbing errors if it cannot read a directory to try globbing there. And the former code never aborted on errors anyway, as without `set -o pipefail` the sort command swallowed the return code. Signed-off-by: Eli Schwartz --- v3: remove extraneous use of [[ -d ... ]] db-functions | 4 ++++ db-update | 8 +++++--- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/db-functions b/db-functions index e8eb2bc..394c7a2 100644 --- a/db-functions +++ b/db-functions @@ -2,6 +2,10 @@ . /usr/share/makepkg/util.sh +# global shell options for enhanced bash scripting +shopt -s globstar nullglob + + # Some PKGBUILDs need CARCH to be set CARCH="x86_64" diff --git a/db-update b/db-update index a8d885a..4e17184 100755 --- a/db-update +++ b/db-update @@ -9,9 +9,11 @@ if (( $# >= 1 )); then fi # Find repos with packages to release -if ! staging_repos=($(find "${STAGING}" -mindepth 1 -type f -name "*${PKGEXTS}" -printf '%h\n' | sort -u)); then - die "Could not read %s" "$STAGING" -fi +mapfile -t -d '' staging_repos < <( + for f in "${STAGING}"/**/*${PKGEXTS}; do + printf '%s\0' "${f%/*}" + done | sort -uz +) repos=() for staging_repo in ${staging_repos[@]##*/}; do From patchwork Tue Feb 20 01:58:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 421 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 50C4522038BA for ; Tue, 20 Feb 2018 01:58:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,RCVD_IN_DNSWL_MED=-2.3 autolearn=ham autolearn_force=no version=3.4.1 X-Spam-BL-Results: [127.0.9.2] [127.0.0.10] [127.0.0.10] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Tue, 20 Feb 2018 01:58:33 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 0975F8D2B3A8D; Tue, 20 Feb 2018 01:58:24 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Tue, 20 Feb 2018 01:58:24 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id E35402BF41; Tue, 20 Feb 2018 01:58:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1519091903; bh=6ZZm7DG2nSMw/pCnkmWzSmsVqASgw7/H3EisVYDhhL4=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=QjatBr42A+JJKXcuRSLfLODMLlRfA5DhUz5PSDKK4qkL3QJTb9SH1iBVk7H6MXe4c RmjQtHMAw8ORNmhu7US/Vre62YyjhuDgwVLU/AsuvJtuKyTbz24P1rechJv0fgqlz1 PtyGrD+RAtozM+45wkHtTT7QXLoqmf2oarkjc/e/DexcIrJRttxCBef3V0ZbwmXj9E 6CTGScTYX/jk1Oug0TdP/8kG+q61VAOiPaxb8rnYyy2n/qMJZnauoe1YhNDgiOt0Fh /P55vnebxzcUjbJKsYK10DLhiTnnMTG1ZCXFoys1I53RtS1Ex6ildee0sShJFarDov iOS3/oeN+UcaAFjVXwSIUq6o1bro24BvPrNQOMHS35Pbf2Bb0+KMtqCeu5DSxUur7z A+MwTAQWqh9ExHNAD6ioPBFDKTXnJxdGeW3jCFFDLeeiz4agM3fxal4AGB/f7d8//k wN8mk29FKkbWRI8gia4CRY5nfJ+JW+UCVS81MrtQzndLkio826x6oFJMk40jBplGUL JA2nUO62fvO6PWtBi0DekuNv+84piJviOvQrADLiifueKCi0IG+ud8s9AMSR80A7JS 2HUljWgDx6oHiYc8phSvdpZxOCNVgj2mS+1Snii1XEaYk5BlOlOW15J4T4rhlIuImr urWJQxJanN7ownIp1WCGKBGM= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 69FF22C829 for ; Tue, 20 Feb 2018 01:58:21 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Tue, 20 Feb 2018 01:58:21 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 66D078D2B3A86; Tue, 20 Feb 2018 01:58:10 +0000 (UTC) Received: from localhost.localdomain (ool-18bdf232.dyn.optonline.net [24.189.242.50]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id C23238D2B3A85; Tue, 20 Feb 2018 01:58:09 +0000 (UTC) To: arch-projects@archlinux.org Date: Mon, 19 Feb 2018 20:58:50 -0500 Message-Id: <20180220015850.29991-1-eschwartz@archlinux.org> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180219201145.14057-6-eschwartz@archlinux.org> References: <20180219201145.14057-6-eschwartz@archlinux.org> MIME-Version: 1.0 Subject: [arch-projects] [dbscripts] [PATCH v3 5/5] Globally set $PKGEXT to a bash extended glob representing valid choices. X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" The current glob `*.pkg.tar.?z` is both less restrictive and more restrictive than makepkg, as it accepts any valid unicode character. To be more exact, it's almost completely orthogonal to the one in makepkg. makepkg only accepts .tar.gz, .tar.bz2, .tar.xz, .tar.lzo, .tar.lrz, and .tar.Z and most of those fail to match against a two-char compression type. dbscripts accepts .pkg.tar.💩z which incidentally is what I think of cherry-picking xz and gz as supported methods. Since this can be anything makepkg.conf accepts, it needs to be able to match all that, unless we decide to perform additional restrictions in which case we should still explicitly list each allowed extension. Using bash extended globbing allows us to do this relatively painlessly. Document the fact that this has *always* been some sort of glob, and update the two cases where this was (not!) being evaluated by bash [[ ... ]], to use a not-elegant-at-all proxy function is_globfile() to evaluate globs *before* testing if they exist. Signed-off-by: Eli Schwartz --- v3: add comment describing the purpose of is_globfile() config | 3 ++- db-functions | 14 +++++++++++--- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/config b/config index 5bb3b16..0d33de0 100644 --- a/config +++ b/config @@ -25,7 +25,8 @@ TMPDIR="/var/tmp" ARCHES=(x86_64) DBEXT=".db.tar.gz" FILESEXT=".files.tar.gz" -PKGEXTS=".pkg.tar.?z" +# bash glob listing allowed extensions. Note that db-functions turns on extglob. +PKGEXTS=".pkg.tar.@(gz|bz2|xz|lzo|lrz|Z)" SRCEXT=".src.tar.gz" # Allowed licenses: get sourceballs only for licenses in this array diff --git a/db-functions b/db-functions index 394c7a2..8b71cae 100644 --- a/db-functions +++ b/db-functions @@ -3,7 +3,7 @@ . /usr/share/makepkg/util.sh # global shell options for enhanced bash scripting -shopt -s globstar nullglob +shopt -s extglob globstar nullglob # Some PKGBUILDs need CARCH to be set @@ -20,6 +20,14 @@ restore_umask () { umask $UMASK >/dev/null } +# Proxy function to check if a file exists. Using [[ -f ... ]] directly is not +# always wanted because we might want to expand bash globs first. This way we +# can pass unquoted globs to is_globfile() and have them expanded as function +# arguments before being checked. +is_globfile() { + [[ -f $1 ]] +} + # just like mv -f, but we touch the file and then copy the content so # default ACLs in the target dir will be applied mv_acl() { @@ -378,8 +386,8 @@ check_pkgrepos() { local pkgver="$(getpkgver ${pkgfile})" || return 1 local pkgarch="$(getpkgarch ${pkgfile})" || return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXTS} ]] && return 1 - [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}${PKGEXTS}.sig ]] && return 1 + is_globfile "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXTS} && return 1 + is_globfile "${FTP_BASE}/${PKGPOOL}/${pkgname}-${pkgver}-${pkgarch}"${PKGEXTS}.sig && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/} ]] && return 1 [[ -f ${FTP_BASE}/${PKGPOOL}/${pkgfile##*/}.sig ]] && return 1