From patchwork Thu Nov 21 07:34:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 1383 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 8DE981531271A for ; Thu, 21 Nov 2019 07:35:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1,DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1, MAILING_LIST_MULTI=-1,RCVD_IN_DNSWL_MED=-2.3,SPF_HELO_NONE=0.001, T_DMARC_POLICY_NONE=0.01 autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Thu, 21 Nov 2019 07:35:27 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id D0DB916C4010CF; Thu, 21 Nov 2019 07:35:25 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: luna) by orion.archlinux.org (Postfix) with ESMTPSA id 755F816C4010C8; Thu, 21 Nov 2019 07:35:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=orion; t=1574321725; bh=S5NrUiZFbbAx2zOQFPbyp9HvVHSzD5v1A8F+obdEnRQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=qscyQteaEJU4FMfchQhReROzBepLbkH0cBXpVUrHD2KmUt3rqYdjDumXcJL2FIJO/ ifRAWiS6GcWeiVojxJbKWO+iKyTSUl0873zBfIEva7VpPSFb1TtQLnBgiGYWxWv6/i tGPlzhoJjZHGRd4li/Mew9UnMbJsbj2nBU/LRKvAzndesFsATSNrMbHAi42yTqbaxS K7Id5aTb6rWwUljFFFIT5IQnL/nTACDfJtkizFHSCkGXLTaDoFS+j1UjBLOjcA022h +Syika8MCQxs2vOEELZoPzTB79I43ZKY0v8CIU9gPl0w/S3kXLUufJeGnlLMyptYRy gj19459T67XUfPWJ3HhlfHPKMB9a0HYQuJjlwNbkSbFaMJ6PQ1rZM+fLn1qqof8wBV PnyZ5nJ6LVt4oqsilZpDjyYKtotBw+TErVhfLEm5HmMsp/4EFsqhrjhv9nVL7ci4aY lHds+ge0h4M0IVlx8uVWzWcx0QbA2r193lYc+kSP0usN2ApWpVKuxqNfnzWB4ri2jM nLOeHR5xwWML+40tJbZc3kB2OI2/pzslSkch5y58P42bs3L6NSIoEWdaF4r5f2jN+X mZ/MlRskPd6Vbr7UXViVeU+twO8lvChLXxCYMJ0uFEPABfY8VboiPQSSvzxy5TCyZF avJr9eAoFSw/Ug1K2f8/mq+M= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 352812CAC9; Thu, 21 Nov 2019 07:35:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1574321725; bh=S5NrUiZFbbAx2zOQFPbyp9HvVHSzD5v1A8F+obdEnRQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=EzwW+AVXx2AdVX4ZeHqjXhpzzQyMC6ancm8tyixxHVCK3hsY3CX8r+F6bNTzq7eyt 4LyGhsOf+WVeJB9G/GCcH4BEqcd8ZD+suUFgknMx8/dDMVHBbdKUZJVrpVHGcyP2n4 Tcp51xcM2z2RnxX/JH7KvVDx7m+JMlSvI3ap7dsAuBZHItTyy2dNSPjxZaEEDQJsxm dU4SWH1JcPeA1sJdodOmbLLB0+N5mQy2bNHyUJCkptcFb57O28X3RBshUO2KIJY1FJ Sw5UgFt5LN+HA+lkGdcFhsxr5dW+BRIKNJ6vHVmkk+tiMBCZfGMPTq/9S0UUfSXiFO YtJMfVgSW4ubxf9REyTr+HQPxssIfzkrr6y0jkpB2k50ZWcx/5dvtMBd21nJ9R8d/S oqVI9IyIm+BEBbHS7fg2mXRhlxc6KbRUGqQTGYhxVdYGnCyBS5xv8nVmFoxnbiNMuf L5B4Tx5/ZYKKFvqz2SxJMUN0AxyG8ILIPZks+TNE6dVgtL5v6Ss8QNGztBLdbYJR2B Q+B5CEddkYY4ugtuTPivm/OQUvXGMR7/hMLj6lEVkYS2AICKMm57IA+d2E6uF7B7O0 PaRzQ52qVVf/CWvgdLc9kp+w2cpDAb5TXjUgyWkU8OIC0IxPEXuDJgy/x22LoMgv9Y o4f4jtaI4jYzgJ25ayaFl1v0= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 955462CAC8 for ; Thu, 21 Nov 2019 07:35:20 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Thu, 21 Nov 2019 07:35:20 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id D655A16C4010C4; Thu, 21 Nov 2019 07:35:15 +0000 (UTC) Received: from didactylos.attlocal.net (unknown [IPv6:2600:1700:57f0:ca20:763a:c795:fcf6:91ea]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id 486DF16C4010C3; Thu, 21 Nov 2019 07:35:15 +0000 (UTC) To: arch-projects@archlinux.org Date: Thu, 21 Nov 2019 02:34:53 -0500 Message-Id: <20191121073453.1499914-1-eschwartz@archlinux.org> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191121053849.1475771-1-eschwartz@archlinux.org> References: <20191121053849.1475771-1-eschwartz@archlinux.org> MIME-Version: 1.0 Subject: [arch-projects] [devtools] [PATCH v3 2/3] makerepropkg: add new program to try to reproducibly build a package X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" This attempts to recreate a package that was probably created using makechrootpkg, and see if it conforms to the https://reproducible-builds.org/ specification. Signed-off-by: Eli Schwartz --- v3: make sure the pkgfile is actually a valid pkgfile .gitignore | 1 + Makefile | 1 + makerepropkg.in | 190 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 192 insertions(+) create mode 100755 makerepropkg.in diff --git a/.gitignore b/.gitignore index 6a1d1e4..7844219 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,7 @@ commitpkg finddeps lddd makechrootpkg +makerepropkg mkarchroot rebuildpkgs zsh_completion diff --git a/Makefile b/Makefile index 0eb7a88..090063d 100644 --- a/Makefile +++ b/Makefile @@ -14,6 +14,7 @@ IN_PROGS = \ finddeps \ find-libdeps \ lddd \ + makerepropkg \ mkarchroot \ makechrootpkg \ rebuildpkgs \ diff --git a/makerepropkg.in b/makerepropkg.in new file mode 100755 index 0000000..710f3ca --- /dev/null +++ b/makerepropkg.in @@ -0,0 +1,190 @@ +#!/bin/bash +# makerepropkg - rebuild a package to see if it is reproducible +# +# Copyright (c) 2019 by Eli Schwartz +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +m4_include(lib/common.sh) +m4_include(lib/archroot.sh) + +source /usr/share/makepkg/util/config.sh +source /usr/share/makepkg/util/message.sh + +declare -A buildinfo +declare -a buildenv buildopts installed installpkgs + +archiveurl='https://archive.archlinux.org/packages' +buildroot=/var/lib/archbuild/reproducible +chroot=testenv + +parse_buildinfo() { + local line var val + + while read -r line; do + var="${line%% = *}" + val="${line#* = }" + case ${var} in + buildenv) + buildenv+=("${val}") + ;; + options) + buildopts+=("${val}") + ;; + installed) + installed+=("${val}") + ;; + *) + buildinfo["${var}"]="${val}" + ;; + esac + done +} + +get_pkgfile() { + local cdir=${cache_dirs[0]} + local pkgfilebase=${1} + local pkgname=${pkgfilebase%-*-*-*} + local pkgfile ext + + for ext in .xz .zstd ''; do + pkgfile=${pkgfilebase}.pkg.tar${ext} + + for c in "${cache_dirs[@]}"; do + if [[ -f ${c}/${pkgfile} ]]; then + cdir=${c} + break + fi + done + + for f in "${pkgfile}" "${pkgfile}.sig"; do + if [[ ! -f "${cdir}/${f}" ]]; then + msg2 "retrieving '%s'..." "${f}" >&2 + curl -Llf -# -o "${cdir}/${f}" "${archiveurl}/${pkgname:0:1}/${pkgname}/${f}" || continue 2 + fi + done + printf '%s\n' "file://${cdir}/${pkgfile}" + return 0 + done + + return 1 +} + +usage() { + cat << __EOF__ +usage: ${BASH_SOURCE[0]##*/} [options] + +Run this script in a PKGBUILD dir to build a package inside a +clean chroot while attempting to reproduce it. The package file +will be used to derive metadata needed for reproducing the +package, including the .PKGINFO as well as the buildinfo. + +For more details see https://reproducible-builds.org/ + +OPTIONS + -c Set pacman cache + -M Location of a makepkg config file + -h Show this usage message +__EOF__ +} + +while getopts 'M:c:h' arg; do + case "$arg" in + M) archroot_args+=(-M "$OPTARG") ;; + c) cache_dirs+=("$OPTARG") ;; + h) usage; exit 0 ;; + *|?) usage; exit 1 ;; + esac +done +shift $((OPTIND - 1)) + +check_root + +if [[ -n $1 ]]; then + pkgfile="$1" + if ! bsdtar -tqf "${pkgfile}" .BUILDINFO >/dev/null 2>&1; then + error "file is not a valid pacman package: '%s'" "${pkgfile}" + exit 1 + fi +else + error "no package file specified. Try '${BASH_SOURCE[0]##*/} -h' for more information. " + exit 1 +fi + +if (( ${#cache_dirs[@]} == 0 )); then + mapfile -t cache_dirs < <(pacman-conf CacheDir) +fi + +ORIG_HOME=${HOME} +IFS=: read -r _ _ _ _ _ HOME _ < <(getent passwd "${SUDO_USER:-$USER}") +load_makepkg_config +HOME=${ORIG_HOME} +[[ -d ${SRCDEST} ]] || SRCDEST=${PWD} + +parse_buildinfo < <(bsdtar -xOqf "${pkgfile}" .BUILDINFO) +export SOURCE_DATE_EPOCH="${buildinfo[builddate]}" +PACKAGER="${buildinfo[packager]}" +BUILDDIR="${buildinfo[builddir]}" + +# nuke and restore reproducible testenv +for copy in "${buildroot}"/*/; do + [[ -d ${copy} ]] || continue + subvolume_delete_recursive "${copy}" +done +rm -rf --one-file-system "${buildroot}" +(umask 0022; mkdir -p "${buildroot}") + +for fname in "${installed[@]}"; do + if ! allpkgfiles+=("$(get_pkgfile "${fname}")"); then + error "failed to retrieve ${fname}" + exit 1 + fi +done +printf '%s\n' "${allpkgfiles[@]}" | mkarchroot -U "${archroot_args[@]}" "${buildroot}"/root - || exit 1 + + +# use makechrootpkg to prep the build directory +makechrootpkg -r "${buildroot}" -l "${chroot}" -- --packagelist || exit 1 + +# set detected makepkg.conf options +{ + for var in PACKAGER BUILDDIR; do + printf '%s=%s\n' "${var}" "${!var@Q}" + done + printf 'OPTIONS=(%s)\n' "${buildopts[*]@Q}" + printf 'BUILDENV=(%s)\n' "${buildenv[*]@Q}" +} >> "${buildroot}/${chroot}"/etc/makepkg.conf >> "${buildroot}/${chroot}"/etc/makepkg.conf +install -d -o "${SUDO_UID:-$UID}" -g "$(id -g "${SUDO_UID:-$UID}")" "${buildroot}/${chroot}/${BUILDDIR}" + +# kick off the build +arch-nspawn "${buildroot}/${chroot}" \ + --bind="${PWD}:/startdir" \ + --bind="${SRCDEST}:/srcdest" \ + /chrootbuild -C --noconfirm --log --holdver --skipinteg + +if (( $? == 0 )); then + msg2 "built succeeded! built packages can be found in ${buildroot}/${chroot}/pkgdest" + msg "comparing artifacts..." + if cmp -s "${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}"; then + msg2 "Package successfully reproduced!" + exit 0 + else + warning "Package is not reproducible. :(" + sha256sum "${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}" + fi +fi + +# the package either failed to build, or was unreproducible +exit 1