From patchwork Tue Mar 12 17:28:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eli Schwartz X-Patchwork-Id: 1043 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 22CC9C70FCB5 for ; Tue, 12 Mar 2019 17:28:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on apollo X-Spam-Level: X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00=-1, DKIMWL_WL_HIGH=-0.001,DKIM_SIGNED=0.1,DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,MAILING_LIST_MULTI=-1,RCVD_IN_DNSWL_MED=-2.3, T_DMARC_POLICY_NONE=0.01 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Tue, 12 Mar 2019 17:28:50 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 33EC511331AD6C; Tue, 12 Mar 2019 17:28:49 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Tue, 12 Mar 2019 17:28:49 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 18A412BF74; Tue, 12 Mar 2019 17:28:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1552411729; bh=dvm8B7LXT+gxgqPHxLuKRlJ+4qXnMis4GPu2DJF1hks=; h=From:To:Date:Subject:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:Reply-To; b=lboavrL3hzdEyUdql+1nY2DnivaWhmsom+l2iVjl6zqdDuENmNQt62AnZq1arFnJa DSv0KSaYC3d9Em6gR7nAhszwbJ4ViX6dYDpA0HcH7EvurqTJXczBeRQmCcav8LoDvw 3qEyIA5QSUS+JBN4YDjN6TVEreWtBIDRRTF0YSfy7dTD5c2SeFJcDVwC7jGBh61suR wv7W5tjjgP7rpQGEFWTXdNt7ywhUKYEWcXpcTlNI/xRZllqPs5QLp2QMplbsgxWECX irXmxRndbhYnYGAiyOYgmfX6bgjXnMEHCr4NRQQVIaa7tGgaIbUUwRCyKGqGi6mv9T OkGpo4FJjg+gs/nnS6FYMbXj5gHfIwpqemjE0x8Wv3dscrTYMdgMkWoMVPH6oA/6N2 qnCGEUD193JnoB1wf7Kk46kHeofeSpHzAPwLKaCs9W56+PwkkTnuAtEcXfbPaY0oBr 2bgoSxApn1y8k5UFD48T56H2xhJDqSwWJ//Go/seALen48+Jbo/z5B0tal+ebmY0Eh v7d78kwZJMWeLoKyUXwS0b5GpYmikGEYWbUpCc5bl2HN9QRzBTPtyiBvft50vl/dLA kF9sUhCfWt0HxX+lEAtcGt5BiIkBhTFKqUocT21jbHqG6Wd0t68ffnXYo+kJe3dRU4 qmNQJMgsLqsZdVf9T+R7w9bg= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 224A52BF64 for ; Tue, 12 Mar 2019 17:28:48 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Tue, 12 Mar 2019 17:28:48 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 0928211331AD51; Tue, 12 Mar 2019 17:28:41 +0000 (UTC) Received: from didactylos.localdomain (unknown [216.97.144.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id 8C26611331AD50; Tue, 12 Mar 2019 17:28:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=orion; t=1552411720; bh=dvm8B7LXT+gxgqPHxLuKRlJ+4qXnMis4GPu2DJF1hks=; h=From:To:Subject:Date; b=IYA5QHEgPMxQWlxG6lnL8tULNtGcZRFPlDcovEWEEiWkHR9rjfqa5EoEKK21e456t AVNbD5n6XKvjVdid20tmI0F5tWt29BbODr8Rs5766Y66d1meVQgDauJExgJG2Ne5dC xywvM1J7rxsX5TLNE5g934SZbJT574YlhnZIktm9Kx5AX6e/LQ3ACMCKaTbviWhCJ2 ZP2RbGGUU6PgjWT+++CTcMIPwbUQ5vcP04MWi1u1paSz4mSorJUE4fllnS+UBjurgG FzsTms7ABA1n0LH5T7ztdJfFX5oqxYX4t9nMMIgX1AWixOl60fuEhSd53FsL3V/qpT KYKxsb0NmPudAHR+rpUvj/PJpFVYuD6NEUacDEJjJwzk0Zzqs95+SjD+D6QlsWtQYd naOnUOqbfMRslBCjZ3jaZgKy5Mg3Swgzc7F8ORCGZgKyPiwq/lYqJzUpUskNbq5aAz BV33KuF3qJhMdvkNzpbCe6vCUWqx3x7cYJQfB7LDT95qu9bktmwWlRljaoFmvgrgu/ ZPhsOcpR0aWqP/n6++XW1PWvHI4XQzY5o8mtOOwREEF/6rfbJP3bHLdezGE5HKl+6L ePlUVDTRgJEppIgR8gdJk71W968wwsDXulcayS0W5nv/QYNt1qBatM9Rr58yaLI25v xlNJZoc0zjzOCacwC7e1edvE= From: Eli Schwartz To: pacman-dev@archlinux.org Date: Tue, 12 Mar 2019 13:28:27 -0400 Message-Id: <20190312172828.13249-1-eschwartz@archlinux.org> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Subject: [pacman-dev] [PATCH v2 1/2] libmakepkg: fix reporting of invalid archive extensions in compress.sh X-BeenThere: pacman-dev@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for pacman development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Discussion list for pacman development Errors-To: pacman-dev-bounces@archlinux.org Sender: "pacman-dev" In commit 1825bd6716c2a51c92642e8b96beac0101e83805 this was split out from makepkg, but the warning was not properly migrated; $ext did not ever exist. As a result, no matter what you did, the only possible warning was: ==> WARNING: '' is not a valid archive extension. Fix to filter based on the presence of .tar in the argument, and building the $ext variable for all checking and messaging purposes within the function. Signed-off-by: Eli Schwartz --- scripts/libmakepkg/util/compress.sh.in | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/scripts/libmakepkg/util/compress.sh.in b/scripts/libmakepkg/util/compress.sh.in index 3d4d88fd..9e3150f7 100644 --- a/scripts/libmakepkg/util/compress.sh.in +++ b/scripts/libmakepkg/util/compress.sh.in @@ -29,21 +29,21 @@ source "$LIBRARY/util/message.sh" # Wrapper around many stream compression formats, for use in the middle of a # pipeline. A tar archive is passed on stdin and compressed to stdout. compress_as() { - # $1: final archive filename extension for compression type detection + # $1: final archive filename extension for compression type detection - local filename="$1" + local ext=".tar${1##*.tar}" - case "$filename" in - *tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; - *tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; - *tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; - *tar.zst) ${COMPRESSZST[@]:-zstd -c -z -q -} ;; - *tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; - *tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; - *tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; - *tar.lz4) ${COMPRESSLZ4[@]:-lz4 -q} ;; - *tar.lz) ${COMPRESSLZ[@]:-lzip -c -f} ;; - *tar) cat ;; + case "$ext" in + *.tar.gz) ${COMPRESSGZ[@]:-gzip -c -f -n} ;; + *.tar.bz2) ${COMPRESSBZ2[@]:-bzip2 -c -f} ;; + *.tar.xz) ${COMPRESSXZ[@]:-xz -c -z -} ;; + *.tar.zst) ${COMPRESSZST[@]:-zstd -c -z -q -} ;; + *.tar.lrz) ${COMPRESSLRZ[@]:-lrzip -q} ;; + *.tar.lzo) ${COMPRESSLZO[@]:-lzop -q} ;; + *.tar.Z) ${COMPRESSZ[@]:-compress -c -f} ;; + *.tar.lz4) ${COMPRESSLZ4[@]:-lz4 -q} ;; + *.tar.lz) ${COMPRESSLZ[@]:-lzip -c -f} ;; + *.tar) cat ;; *) warning "$(gettext "'%s' is not a valid archive extension.")" \ "$ext"; cat ;; esac From patchwork Tue Mar 12 17:28:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eli Schwartz X-Patchwork-Id: 1042 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id BC725C70FCA4 for ; Tue, 12 Mar 2019 17:28:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on apollo X-Spam-Level: X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00=-1, DKIMWL_WL_HIGH=-0.001,DKIM_SIGNED=0.1,DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,MAILING_LIST_MULTI=-1,RCVD_IN_DNSWL_MED=-2.3, T_DMARC_POLICY_NONE=0.01 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Tue, 12 Mar 2019 17:28:46 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 10B9A11331AD57; Tue, 12 Mar 2019 17:28:45 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Tue, 12 Mar 2019 17:28:44 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id BC5812BF26; Tue, 12 Mar 2019 17:28:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1552411724; bh=mH1dgLzaZMpXbf2NE4khsMP/822WT/9eoSXyhuIcv+w=; h=From:To:Date:In-Reply-To:References:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Reply-To; b=X0q63mjSVA/QcoyAJjJoCXeOb8e2T3OB/QC8nv83c8uClW6JLdNk3keqXeSxbHxGp dMdJTlmWtt3pJDa8Bcx5M7tOsASvWVjs5zEg3ugt62lg9I0yWvrWn6SqkmKBeDl/Hq 181mpx3kPaPWidqDGPPQMmg/7OlG8rUlFE2UXSv+4FZV4zpasP95oA6saQ35UsMCOV r7Cp2JCwm430mj6Me8ss4ERX0hK7t0lwaOs+i6dZ5kEjbCxPPX0OCl47lSJduZVcQj qdbpwgA1hsUP/3jONviXdA/1cHOrQp2NaXrUGRtwck2N7dQ4Ku41GU3g1kzVUwkypp NYDYOT2B+tVlfIveEEHBgakrpcxJ6LCcys2XF89jc2UQ4GxnlUiiCFBf5l6hgR7hRS ++AFoKwMJAT4tjPLGxI8nr/dtCBg8DQLemBE3nugmdc1cEkIRTGQ1A5CIYoYXNFN9l qxXaTB6+VTlOJnfp5TAHEc4In+8/WgZvv+F4fxlXQFvGwvqLOxyk7ZNNt0TRmOGpR8 RAI3WzC3DNADdOsVdpyln4PiatdPtWWM49eTWF+UY0E7bG65GAEfgmZeCG22roe85L 8WhwUZ64ANG/14GYLuVERx64hBl00yFEPvTy6pi46vV0UhJ36COcmhDitOfxXletSF g4gaGbApI975jhZARbNmxkd4= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id EA2EE2BF22 for ; Tue, 12 Mar 2019 17:28:42 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Tue, 12 Mar 2019 17:28:42 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 9710111331AD53; Tue, 12 Mar 2019 17:28:41 +0000 (UTC) Received: from didactylos.localdomain (unknown [216.97.144.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id 2304711331AD52; Tue, 12 Mar 2019 17:28:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=orion; t=1552411721; bh=mH1dgLzaZMpXbf2NE4khsMP/822WT/9eoSXyhuIcv+w=; h=From:To:Subject:Date:In-Reply-To:References; b=pAnBp2joKX8zM5VUYcs3fq47AlYkH+ZuN1EVxRS9YjtCq8MWbvEQJ9MOs57jLbtZz 3L4pxs7hL5FTLNU5p7S/oiV42VSFMl7GyxBi/Ir1Un+1+dVXS8XFR7fh0/I45xcwUZ 9H8BgqKKdN/1junpFPvZr5C/XRDR3pO28U63HKPk1EW9WmoD1T9WDd4ksZ70MDLksJ F3bGzIzZG4jTQnZ2bAK8C6xyOqfl8NMm/Z9EZ6HIRX/40Lja51prfzVBp9W96h8+WR Tc0QjkXiEfXIZWl6cE4XNLOVm986Gn4korkkxK75W7KSsaoQ7pnRz2SD6dojsR7+E4 XvLDU5IjZBxCkIFxW4UhhCtIIlyZgd8wRULE8Z7Lq9moo5w1DnQnvT7JSzxMvju/Y4 DT7CViYMQIywnsQPUWi+hne7Dpby3CNTHTxpo5+vGr7JJJgzQoYI7BCTUSO9QVGXL9 7wuRTd17Vm8UvSujDG2Z+xHWDqCgv0TOw7pKIUhGDHHBGik6xAg4IKmQqiZpoCHtVx NgekB0M2sVz9tNKewGZgzuPa1dYpoLr741CZmJ3Unr8NAbB2swoi2bekbTwcN4zmfB U67YnToi1zcXruvskv9okKaxK/SPflw9/wiF57/w4OnEMzqcPcwYQSVzXOUhgF7vWx KUlzQLefgtzROen+0lM/8QTE= From: Eli Schwartz To: pacman-dev@archlinux.org Date: Tue, 12 Mar 2019 13:28:28 -0400 Message-Id: <20190312172828.13249-2-eschwartz@archlinux.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190312172828.13249-1-eschwartz@archlinux.org> References: <20190312172828.13249-1-eschwartz@archlinux.org> MIME-Version: 1.0 Subject: [pacman-dev] [PATCH v2 2/2] repo-add: use more libmakepkg to handle common compression routines X-BeenThere: pacman-dev@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for pacman development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Discussion list for pacman development Errors-To: pacman-dev-bounces@archlinux.org Sender: "pacman-dev" Currently the list of supported formats for an archive, is maintained in two places. And repo-add does not actually get updated. :( In the process, remove some of the logical duplication when calling bsdtar/compress_as. Also, move to emulating makepkg by not returning a hard error if given an invalid archive extension; compress_as will accept it, throw a warning, and proceed using `cat` as the compression filter, which pacman consumes just as well. Signed-off-by: Eli Schwartz --- v2: use a better argument to compress_as, expand the commit message a bit to clarify the practical effects of the change. scripts/repo-add.sh.in | 27 +++++++++------------------ 1 file changed, 9 insertions(+), 18 deletions(-) diff --git a/scripts/repo-add.sh.in b/scripts/repo-add.sh.in index 57413df5..ec2838a7 100644 --- a/scripts/repo-add.sh.in +++ b/scripts/repo-add.sh.in @@ -44,6 +44,7 @@ CLEAN_LOCK=0 USE_COLOR='y' # Import libmakepkg +source "$LIBRARY"/util/compress.sh source "$LIBRARY"/util/message.sh # ensure we have a sane umask set @@ -188,21 +189,12 @@ verify_signature() { } verify_repo_extension() { - local repofile=$1 - - case $repofile in - *.db.tar.gz) TAR_OPT="z" ;; - *.db.tar.bz2) TAR_OPT="j" ;; - *.db.tar.xz) TAR_OPT="J" ;; - *.db.tar.zst) TAR_OPT="--zstd" ;; - *.db.tar.Z) TAR_OPT="Z" ;; - *.db.tar) TAR_OPT="" ;; + case $1 in + *.db.tar.*|*.db.tar) ;; *) error "$(gettext "'%s' does not have a valid database archive extension.")" \ - "$repofile" + "$1" exit 1 ;; esac - - printf '%s' "$TAR_OPT" } # write an entry to the pacman database @@ -513,7 +505,6 @@ rotate_db() { } create_db() { - TAR_OPT=$(verify_repo_extension "$REPO_DB_FILE") # $LOCKFILE is already guaranteed to be absolute so this is safe dirname=${LOCKFILE%/*} @@ -523,13 +514,13 @@ create_db() { tempname=$dirname/.tmp.$filename pushd "$tmpdir/$repo" >/dev/null - if ( shopt -s nullglob; files=(*); (( ${#files[*]} )) ); then - bsdtar -c${TAR_OPT}f "$tempname" * - else + local files=(*) + if [[ ${files[*]} = '*' ]]; then # we have no packages remaining? zip up some emptyness warning "$(gettext "No packages remain, creating empty database.")" - bsdtar -c${TAR_OPT}f "$tempname" -T /dev/null + files=(-T /dev/null) fi + bsdtar -cf - "${files[@]}" | compress_as "$filename" > "$tempname" popd >/dev/null create_signature "$tempname" @@ -644,7 +635,7 @@ else LOCKFILE=$PWD/$REPO_DB_FILE.lck fi -verify_repo_extension "$REPO_DB_FILE" >/dev/null +verify_repo_extension "$REPO_DB_FILE" REPO_DB_PREFIX=${REPO_DB_FILE##*/} REPO_DB_PREFIX=${REPO_DB_PREFIX%.db.*}