From patchwork Thu Mar 10 15:38:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonas Witschel X-Patchwork-Id: 2046 Return-Path: Delivered-To: patchwork@archlinux.org Received: from mail.archlinux.org [95.216.189.61] by patchwork.archlinux.org with IMAP (fetchmail-6.4.28) for (single-drop); Thu, 10 Mar 2022 15:39:07 +0000 (UTC) Received: from mail.archlinux.org by mail.archlinux.org with LMTP id TgCOAZsbKmJTUAEAK+/4rw (envelope-from ) for ; Thu, 10 Mar 2022 15:39:07 +0000 Received: from lists.archlinux.org (lists.archlinux.org [IPv6:2a01:4f9:c010:9eb4::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.archlinux.org (Postfix) with ESMTPS id 87719DD0EDA; Thu, 10 Mar 2022 15:39:05 +0000 (UTC) Received: from lists.archlinux.org (localhost [IPv6:::1]) by lists.archlinux.org (Postfix) with ESMTP id 5FA16DC4874; Thu, 10 Mar 2022 15:39:05 +0000 (UTC) X-Original-To: pacman-dev@lists.archlinux.org Delivered-To: pacman-dev@lists.archlinux.org Received: from mail.archlinux.org (mail.archlinux.org [95.216.189.61]) by lists.archlinux.org (Postfix) with ESMTPS id 3AF61DC4864 for ; Thu, 10 Mar 2022 15:39:02 +0000 (UTC) From: Jonas Witschel DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=archlinux.org; s=dkim-rsa; t=1646926741; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=5dxg/PqogQ2UHCdfKrig1AE21oNCCO9E+b12D7xLqF4=; b=vwZn2KChXCKftZI3eNNrgG1qN8N+WU14hDB4ZsoVFe0DKvYQ9KHKGPJHq7VrNhh/GBrn5x SO4YRZq1UOTqzwmaHLXFMLSffbZj6s9cVAj86K9skziCoJprg4onzhHHS9L2LE3y19Rs30 4MbRhLVnu6GVnKJjnj3+fe7NFWYk7VY9biyW4w8JU1AFMQEPbRH8rI///Mx3f1sHHJxmrI 0b7o+y1pvqM9x+3yhqgo7xydTFxgMzKebzAZ/7ckYHrWoVODaTasSm/IAIG7ERio2GZkcw A0t3485hzDGW+SpIhhxAKYrBhpdWrH9i+DLkrYu1NmqsyolwId5vEXy3bxQ4hfkyra5KxL 8Q0TMFk1BB19rkPxwHIfRRyneG4i/ny6Ct1QvHAbO9BQSAgP8n6LD5hbZoj7jH74FUvE8I 1bEDv1xih7kxzp1FXRq15elSB6O22VCzTF+sk/iASCu7FIp4yX5a1qQcj8u/LKt9r+09i+ iJlOBRh3B52HDyUf/wZTiAkbH3RqrCrvkCsKIzWf/XJOa+C3RRJ+lWami4tnF68a4mRGad 7+OkbphjJeQtpOw0OFSBtOgMN0Veo6eHYTsTNHfF4e24Y8wcpj0TWkxSTQ2ROZjnAcDQmr X5VJJHhh0iAoKOU+G7wArwT7p0fv00PBjfEIYOwFOWDtpFe8gH+Vw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=archlinux.org; s=dkim-ed25519; t=1646926741; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=5dxg/PqogQ2UHCdfKrig1AE21oNCCO9E+b12D7xLqF4=; b=VynupaueuuXE4a0gM6b0wRDVdh+HJ4EgDPj1ZNxhqXuT2+5oYbUBGWoTPeQeJsC+55W9Mc Ts4qt6allBc5nRDA== To: pacman-dev@lists.archlinux.org Cc: Jonas Witschel Subject: [PATCH] makepkg: use bsdtar --no-read-sparse for archive creation if available Date: Thu, 10 Mar 2022 16:38:42 +0100 Message-Id: <20220310153842.133677-1-diabonas@archlinux.org> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 X-BeenThere: pacman-dev@lists.archlinux.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: Discussion list for pacman development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pacman-dev-bounces@lists.archlinux.org Sender: "pacman-dev" X-Spamd-Result: default: False [-2.61 / 15.00]; DWL_DNSWL_MED(-2.00)[archlinux.org:dkim]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DMARC_POLICY_ALLOW(-0.50)[archlinux.org,none]; RCVD_IN_DNSWL_MED(-0.40)[95.216.189.61:received,2a01:4f9:c010:9eb4::1:from]; MAILLIST(-0.20)[mailman]; R_DKIM_ALLOW(-0.20)[archlinux.org:s=dkim-rsa,archlinux.org:s=dkim-ed25519]; R_SPF_ALLOW(-0.20)[+ip6:2a01:4f9:c010:9eb4::1]; MIME_GOOD(-0.10)[text/plain]; HAS_LIST_UNSUB(-0.01)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f9::/32, country:DE]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[archlinux.org:+]; FROM_HAS_DN(0.00)[]; NEURAL_HAM(-0.00)[-0.993]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[diabonas@archlinux.org,pacman-dev-bounces@lists.archlinux.org]; PREVIOUSLY_DELIVERED(0.00)[pacman-dev@lists.archlinux.org]; FORGED_SENDER_MAILLIST(0.00)[] Authentication-Results: mail.archlinux.org; dkim=pass header.d=archlinux.org header.s=dkim-rsa header.b=vwZn2KCh; dkim=pass header.d=archlinux.org header.s=dkim-ed25519 header.b=Vynupaue; dmarc=pass (policy=none) header.from=archlinux.org; spf=pass (mail.archlinux.org: domain of pacman-dev-bounces@lists.archlinux.org designates 2a01:4f9:c010:9eb4::1 as permitted sender) smtp.mailfrom=pacman-dev-bounces@lists.archlinux.org X-Rspamd-Queue-Id: 87719DD0EDA X-Rspamd-Server: mail.archlinux.org bsdtar uses the "pax" TAR archive format by default, which has support for storing sparse file information in the archive. Unfortunately this is a source of unreproducibility because the sparse encoding is taken from the file system and different file systems handle sparse files differently: some file systems have no support for sparsely encoded files at all, and even file systems with sparse file support can report different file information for identical files due to differing implementations. As a real world example where this happens, consider the Arch Linux package "brotli-testdata 1.0.9-7", which contains a sparsely encoded all-zeros file "usr/share/brotli/testdata/zeros". Building this package on a btrfs file system yields a different package than building it on tmpfs or ext4 solely due to different sparse file information that gets recorded in the package tarball. To improve the reproducibility of archives containing sparsely encoded files, libarchive version 3.6.0 introduces a new --no-read-sparse option. This skips reading sparse file information from disk entirely and therefore stores files "expanded" in the archive, which is the only way to make them reliably reproducible across file systems. makepkg will use this option if libarchive is recent enough to support it, which is detected at build time. --- build-aux/edit-script.sh.in | 1 + meson.build | 6 ++++++ scripts/makepkg.sh.in | 4 ++-- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/build-aux/edit-script.sh.in b/build-aux/edit-script.sh.in index 85c56cfe..992033b2 100644 --- a/build-aux/edit-script.sh.in +++ b/build-aux/edit-script.sh.in @@ -20,6 +20,7 @@ sed \ -e "s|@DEBUGSUFFIX[@]|@DEBUGSUFFIX@|g" \ -e "s|@INODECMD[@]|@INODECMD@|g" \ -e "s|@FILECMD[@]|@FILECMD@|g" \ + -e "s|@BSDTAR_NO_READ_SPARSE[@]|@BSDTAR_NO_READ_SPARSE@|g" \ "$input" >"$output" if [[ $mode ]]; then diff --git a/meson.build b/meson.build index b7cca865..1519a2bb 100644 --- a/meson.build +++ b/meson.build @@ -91,6 +91,11 @@ endif libarchive = dependency('libarchive', version : '>=3.0.0', static : get_option('buildstatic')) +if libarchive.version().version_compare('>=3.6.0') + bsdtar_no_read_sparse = '--no-read-sparse' +else + bsdtar_no_read_sparse = '' +endif libcurl = dependency('libcurl', version : '>=7.55.0', @@ -274,6 +279,7 @@ substs.set('LIBMAKEPKGDIR', LIBMAKEPKGDIR) substs.set('STRIP_BINARIES', strip_binaries) substs.set('STRIP_SHARED', strip_shared) substs.set('STRIP_STATIC', strip_static) +substs.set('BSDTAR_NO_READ_SPARSE', bsdtar_no_read_sparse) subdir('lib/libalpm') subdir('src/common') diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 5aaabf63..69757d03 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -616,7 +616,7 @@ create_package() { msg2 "$(gettext "Compressing package...")" # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail - list_package_files | LANG=C bsdtar --no-fflags -cnf - --null --files-from - | + list_package_files | LANG=C bsdtar --no-fflags @BSDTAR_NO_READ_SPARSE@ -cnf - --null --files-from - | compress_as "$PKGEXT" > "${pkg_file}" || ret=$? shopt -u -o pipefail @@ -714,7 +714,7 @@ create_srcpackage() { # TODO: Maybe this can be set globally for robustness shopt -s -o pipefail - LANG=C bsdtar --no-fflags -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$? + LANG=C bsdtar --no-fflags @BSDTAR_NO_READ_SPARSE@ -cLf - ${pkgbase} | compress_as "$SRCEXT" > "${pkg_file}" || ret=$? shopt -u -o pipefail