From patchwork Mon Nov 18 02:32:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 1377 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 8034E152612E9 for ; Mon, 18 Nov 2019 02:33:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on apollo X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1,DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1, MAILING_LIST_MULTI=-1,RCVD_IN_DNSWL_MED=-2.3,SPF_HELO_NONE=0.001, T_DMARC_POLICY_NONE=0.01 autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Mon, 18 Nov 2019 02:33:35 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 1735716B40D260; Mon, 18 Nov 2019 02:33:15 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) (Authenticated sender: luna) by orion.archlinux.org (Postfix) with ESMTPSA id 5CB3116B40D25C; Mon, 18 Nov 2019 02:33:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=orion; t=1574044393; bh=djxeRot3uHwr2qSmJoY+cTDGOsaxle0Ofs5te2d0V6M=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=W0Kg6Ku+9rFCFLLABD8vqzCovGxARxxaXFdvTnwy4filqMLFsUpxr3N+F468cSNgJ 3XDLedlLjeEzqlmK8vscEkoul6jXLNQArJ4Lyg5E8ksUQEJ8Dn/TzLOAMwY2epiYOm ZXsgLjuN+4+KnCRnd+/+FX6MNACDyj+FIYpS1DrHYOxEm3YGhzJGxEw2c9c/EVKp+v MXVri6QQ+yyf8nCi0DpbQUIy8mILSKsdPAFPlHMWVMbZeBZWCUtFgGhZrvU8Oyc66H vbALYpzScddQ6MMxTwRXnVYM7y3+Wjl3NgFymSNnrvCjBEqjccvlb2JhbtwHV03aK+ UU0xw1LRILjBM6GDp1+4q9/awN90vitKkgjQJ9IkE5rA63Po+fxFWrUOjYGnaUzwK0 u+MqHups0Dk86QGx+67GHZPGeVruGoZ0ZkgztT4qFhxQLe0VXemT8cFjn44OCh45fx Yu73WZNLLEVXybbmrThgKFr5ev9/CKFqSYLioVaTMflIcSi1i4cLHmAkfYqHF1TIN0 90rqR7hfi2M/x6YubH1zeROKu6z3aiANdbXlV9A1/cFWfNmSP/gmwoxPEF7q+0Mnfr sx9adUE8VsqcYdRDT/tMeCXRcFezThL7ak0XP23MIK8GGtxIbhMzemwqJAv/1DoX3k HK5kq4q3dC4RipK0Nco85s68= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 3653C20F0C; Mon, 18 Nov 2019 02:33:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1574044382; bh=djxeRot3uHwr2qSmJoY+cTDGOsaxle0Ofs5te2d0V6M=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=TEmKLqXLzRAESoDXSpdM4CR43SdLzte2G8YAy4z7q1MAD7rAA9RvOqL59NUeuMUsq c8AMBsqCOo9gyDWv64zZIpZdS36RSrqWWgQ7w/xsuWHFbZQ4a2fijE4VltpUVUR4jf CW13r+glYD5ruoEI5VL6sQkilL0TAu13y9qXGo6lGobAmhmrVD7BE5JLQKwzLHLBuf S/ghStrpjqhoVuvtcB/E9hqdw036t58fPH3F9Aota6bjrkGD36C7v9UpJntBeBh2BD 2JB4DZNOnNv3racXBP60LMRXxAG2RcyfPNRkT3H5C0bHbU3RYzQqzmTktzZ6QXyGCF f9uCNKN3KshvnOaiukwyLsuS3QcdqV42J7CzE4X9fUCCdlhWxYJH7l0ky3p1b8+h+E 7ZIiJhWxxXHYubUmBuMWBLZNSS4shSrzdwJqIq8tz6AOld+m1c/kPDCJOgG5WJ+gRq oSdY3CxquFFUJU9+AwaCD99Znp5AysYAV7r/8u3L2L6dXRK7wdxjE5D9esU+9/w+QU RTPslFpqkBl4AoHQyofALJsZhE4lXdLJ0ynNZxVkx783s558coIZvS0r6FuskgVFQb F+NNy0IcskB3JHF9xXtZDY6zDQG3nw7oGfrHQVjMxnlgED+TvxHotG96iflcGAir2a HGwZs/BZPNO1sKgSuwPPRa9s= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id EC83520788 for ; Mon, 18 Nov 2019 02:32:58 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Mon, 18 Nov 2019 02:32:58 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 7472316B40D1CD; Mon, 18 Nov 2019 02:32:57 +0000 (UTC) Received: from didactylos.attlocal.net (unknown [IPv6:2600:1700:57f0:ca20:763a:c795:fcf6:91ea]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id DAB3916B40D1C9; Mon, 18 Nov 2019 02:32:56 +0000 (UTC) To: arch-projects@archlinux.org Date: Sun, 17 Nov 2019 21:32:42 -0500 Message-Id: <20191118023243.339374-3-eschwartz@archlinux.org> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191118023243.339374-1-eschwartz@archlinux.org> References: <20191118023243.339374-1-eschwartz@archlinux.org> MIME-Version: 1.0 Subject: [arch-projects] [devtools] [PATCH 2/3] makerepropkg: add new program to try to reproducibly build a package X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" This attempts to recreate a package that was probably created using makechrootpkg, and see if it conforms to the https://reproducible-builds.org/ specification. Signed-off-by: Eli Schwartz --- Makefile | 1 + makerepropkg.in | 186 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 187 insertions(+) create mode 100755 makerepropkg.in diff --git a/Makefile b/Makefile index 0eb7a88..090063d 100644 --- a/Makefile +++ b/Makefile @@ -14,6 +14,7 @@ IN_PROGS = \ finddeps \ find-libdeps \ lddd \ + makerepropkg \ mkarchroot \ makechrootpkg \ rebuildpkgs \ diff --git a/makerepropkg.in b/makerepropkg.in new file mode 100755 index 0000000..d1414d8 --- /dev/null +++ b/makerepropkg.in @@ -0,0 +1,186 @@ +#!/bin/bash +# makerepropkg - rebuild a package to see if it is reproducible +# +# Copyright (c) 2019 by Eli Schwartz +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +m4_include(lib/common.sh) +m4_include(lib/archroot.sh) + +source /usr/share/makepkg/util/config.sh +source /usr/share/makepkg/util/message.sh + +declare -A buildinfo +declare -a buildenv buildopts installed installpkgs + +archiveurl='https://archive.archlinux.org/packages' +buildroot=/var/lib/archbuild/reproducible +chroot=testenv + +parse_buildinfo() { + local line var val + + while read -r line; do + var="${line%% = *}" + val="${line#* = }" + case ${var} in + buildenv) + buildenv+=("${val}") + ;; + options) + buildopts+=("${val}") + ;; + installed) + installed+=("${val}") + ;; + *) + buildinfo["${var}"]="${val}" + ;; + esac + done +} + +get_pkgfile() { + local cdir=${cache_dirs[0]} + local pkgfilebase=${1} + local pkgname=${pkgfilebase%-*-*-*} + local pkgfile ext + + for ext in .xz .zstd ''; do + pkgfile=${pkgfilebase}.pkg.tar${ext} + + for c in "${cache_dirs[@]}"; do + if [[ -f ${c}/${pkgfile} ]]; then + cdir=${c} + break + fi + done + + for f in "${pkgfile}" "${pkgfile}.sig"; do + if [[ ! -f "${cdir}/${f}" ]]; then + msg2 "retrieving '%s'..." "${f}" >&2 + curl -Llf -# -o "${cdir}/${f}" "${archiveurl}/${pkgname:0:1}/${pkgname}/${f}" || continue 2 + fi + done + printf '%s\n' "file://${cdir}/${pkgfile}" + return 0 + done + + return 1 +} + +usage() { + cat << __EOF__ +usage: ${BASH_SOURCE[0]##*/} [options] + +Run this script in a PKGBUILD dir to build a package inside a +clean chroot while attempting to reproduce it. The package file +will be used to derive metadata needed for reproducing the +package, including the .PKGINFO as well as the buildinfo. + +For more details see https://reproducible-builds.org/ + +OPTIONS + -c Set pacman cache + -M Location of a makepkg config file + -h Show this usage message +__EOF__ +} + +while getopts 'M:c:h' arg; do + case "$arg" in + M) archroot_args+=(-M "$OPTARG") ;; + c) cache_dirs+=("$OPTARG") ;; + h) usage; exit 0 ;; + *|?) usage; exit 1 ;; + esac +done +shift $((OPTIND - 1)) + +check_root + +if [[ -n $1 ]]; then + pkgfile="$1" +else + error "no package file specified. Try '${BASH_SOURCE[0]##*/} -h' for more information. " + exit 1 +fi + +if (( ${#cache_dirs[@]} == 0 )); then + mapfile -t cache_dirs < <(pacman-conf CacheDir) +fi + +ORIG_HOME=${HOME} +IFS=: read -r _ _ _ _ _ HOME _ < <(getent passwd "${SUDO_USER:-$USER}") +load_makepkg_config +HOME=${ORIG_HOME} +[[ -d ${SRCDEST} ]] || SRCDEST=${PWD} + +parse_buildinfo < <(bsdtar -xOqf "${pkgfile}" .BUILDINFO) +export SOURCE_DATE_EPOCH="${buildinfo[builddate]}" +PACKAGER="${buildinfo[packager]}" +BUILDDIR="${buildinfo[builddir]}" + +# nuke and restore reproducible testenv +for copy in "${buildroot}"/*/; do + [[ -d ${copy} ]] || continue + subvolume_delete_recursive "${copy}" +done +rm -rf --one-file-system "${buildroot}" +(umask 0022; mkdir -p "${buildroot}") + +for fname in "${installed[@]}"; do + if ! allpkgfiles+=("$(get_pkgfile "${fname}")"); then + error "failed to retrieve ${fname}" + exit 1 + fi +done +printf '%s\n' "${allpkgfiles[@]}" | mkarchroot -U "${archroot_args[@]}" "${buildroot}"/root - || exit 1 + + +# use makechrootpkg to prep the build directory +makechrootpkg -r "${buildroot}" -l "${chroot}" -- --packagelist || exit 1 + +# set detected makepkg.conf options +{ + for var in PACKAGER BUILDDIR; do + printf '%s=%s\n' "${var}" "${!var@Q}" + done + printf 'OPTIONS=(%s)\n' "${buildopts[*]@Q}" + printf 'BUILDENV=(%s)\n' "${buildenv[*]@Q}" +} >> "${buildroot}/${chroot}"/etc/makepkg.conf >> "${buildroot}/${chroot}"/etc/makepkg.conf +install -d -o "${SUDO_UID:-$UID}" -g "$(id -g "${SUDO_UID:-$UID}")" "${buildroot}/${chroot}/${BUILDDIR}" + +# kick off the build +arch-nspawn "${buildroot}/${chroot}" \ + --bind="${PWD}:/startdir" \ + --bind="${SRCDEST}:/srcdest" \ + /chrootbuild -C --noconfirm --log --holdver --skipinteg + +if (( $? == 0 )); then + msg2 "built succeeded! built packages can be found in ${buildroot}/${chroot}/pkgdest" + msg "comparing artifacts..." + if cmp -s "${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}"; then + msg2 "Package successfully reproduced!" + exit 0 + else + warning "Package is not reproducible. :(" + sha256sum "${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}" + fi +fi + +# the package either failed to build, or was unreproducible +exit 1