From patchwork Thu Apr 18 01:15:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 1086 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id E5367D657F4F for ; Thu, 18 Apr 2019 01:15:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on apollo X-Spam-Level: X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00=-1, DKIMWL_WL_HIGH=-0.055,DKIM_SIGNED=0.1,DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,MAILING_LIST_MULTI=-1,RCVD_IN_DNSWL_MED=-2.3, T_DMARC_POLICY_NONE=0.01 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Thu, 18 Apr 2019 01:15:46 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 061D811FF564D8; Thu, 18 Apr 2019 01:15:43 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Thu, 18 Apr 2019 01:15:42 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id D289E207AC; Thu, 18 Apr 2019 01:15:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1555550142; bh=EfAS1wD4TFgLp9NLRYRwUaKFCFUZmmfNvIS2/wIS01o=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=Ty0WRKqjwkUoTi/s3himvO8krjDuXFhHs+5lXHLvayq4ef4H25VkROs4WRa42GcHD /Ia83nfz1xoYSQcJ6AekmXcTOlcAVhoA+ZGxaJOvAhZs4ase7GyFc5Zm4ufTQWyP2d KNqSDQOSvVt3fEWSbM/yAunkLofFhTfUYJt7Ad5JwVu6akan0fgtqdIQHP3jTzP61L rFKUtj+qYCLJLHJZfDWFjQ7uqM6HXZA3fdhrLOt/RkNtIa3yy8f3AotRcnbZWOH2Vz FzR485L2YTrm05U8DuUOUB23b1d98DDRCz1YKk1NZ9DAbnRM9usUYrohQC9YzMQza7 Yvv+g7FHN6HvwEAly0SixgoOn+H+jwgIlUJyt2SlAEgSK+qrDhH1vVRHoA8/mw7Oha OVq8Jzj+gF6I4vo6k4ZxlieoT2lwbRMJJdUmYYEjwQZlHa8fwp9IuBi0j0UJSFSgWS IAN+gaUmBPxruNJVRzKtviH63dm2OLXKNeTzjr4ZhfspUbSlkwq1qjBNHm88u2pvLW p+0lKalpfQ+nYfYu68OyE/mBrFkDf4cYdTkm+idPClBCmjZoi7oo/1EqYyJ6FBp1LG mwQfEfBfI8v8OxzAuzCT2ska/t1dCBhhwXGshxIzMLW/+rL2n6H/Wb1gsccmoJ11Kk k+WXRyIjWVhmQ8lBmjMZ+jrs= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 21298207AA for ; Thu, 18 Apr 2019 01:15:38 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Thu, 18 Apr 2019 01:15:38 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 5136A11FF564D4; Thu, 18 Apr 2019 01:15:31 +0000 (UTC) Received: from didactylos.attlocal.net (unknown [IPv6:2600:1700:57f0:ca20:763a:c795:fcf6:91ea]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id 614CB11FF564D3; Thu, 18 Apr 2019 01:15:30 +0000 (UTC) To: arch-projects@archlinux.org Date: Wed, 17 Apr 2019 21:15:06 -0400 Message-Id: <20190418011506.4350-1-eschwartz@archlinux.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190414221953.26455-1-eschwartz@archlinux.org> References: <20190414221953.26455-1-eschwartz@archlinux.org> MIME-Version: 1.0 Subject: [arch-projects] [devtools] [PATCH v2] Add new tool: offload-build X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz , svenstaro@gmail.com Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" This tool is useful for running makechrootpkg on a remote build server, and is by default hooked up to send a PKGBUILD and initiate a build on our shiny new build server "dragon". Signed-off-by: Eli Schwartz --- v2: script -qfc does not preserve the exit status of the wrapped command. Add -e so that we can properly abort if makechrootpkg failed. Makefile | 1 + offload-build | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100755 offload-build diff --git a/Makefile b/Makefile index ba2d3e4..2306a17 100644 --- a/Makefile +++ b/Makefile @@ -20,6 +20,7 @@ IN_PROGS = \ BINPROGS = \ $(IN_PROGS) \ + offload-build \ sogrep CONFIGFILES = \ diff --git a/offload-build b/offload-build new file mode 100755 index 0000000..c991754 --- /dev/null +++ b/offload-build @@ -0,0 +1,100 @@ +#!/bin/bash +# +# offload-build - build a PKGBUILD on a remote server using makechrootpkg. +# +# Copyright (c) 2019 by Eli Schwartz +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + + +# global defaults suitable for use by Arch staff +repo=extra +arch=x86_64 +server=dragon.archlinux.org + +die() { printf "error: $1\n" "${@:2}"; exit 1; } + +usage() { + cat <<- _EOF_ + Usage: ${BASH_SOURCE[0]##*/} [--repo REPO] [--arch ARCHITECTURE] [--server SERVER] + + Build a PKGBUILD on a remote server using makechrootpkg. Requires a remote user + that can run archbuild without password auth. + + OPTIONS + -r, --repo Build against a specific repository (current: $repo) + -a, --arch Build against a specific architecture (current: $arch) + -s, --server Offload to a specific build server (current: $server) + -h, --help Show this help text +_EOF_ +} + +# option checking +while (( $# )); do + case $1 in + -h|--help) + usage + exit 0 + ;; + -r|--repo) + repo=$2 + shift 2 + ;; + -a|--arch) + arch=$2 + shift 2 + ;; + -s|--server) + server=$2 + shift 2 + ;; + *) + die "invalid argument: %s" "$1" + ;; + esac +done + +# multilib must be handled specially +if [[ $repo = multilib* ]]; then + arch= +fi + +trap 'rm -rf $SRCPKGDEST' EXIT + +# Use a source-only tarball as an intermediate to transfer files. This +# guarantees the checksums are okay, and guarantees that all needed files are +# transferred, including local sources, install scripts, and changelogs. +export SRCPKGDEST=$(mktemp -d) +makepkg --source || die "unable to make source package" + +mapfile -t files < <( + # This is sort of bash golfing but it allows running a mildly complex + # command over ssh with a single connection. + cat "$SRCPKGDEST"/*.src.tar.gz | + ssh $server ' + temp="${XDG_CACHE_HOME:-$HOME/.cache}/offload-build" && + mkdir -p "$temp" && + temp=$(mktemp -d -p "$temp") && + cd "$temp" && + { + bsdtar --strip-components 1 -xvf - && + script -qefc '"${repo}${arch:+-$arch}-build"' /dev/null && + printf "%s\n" "" "-> build complete" && + printf "\t%s\n" "$temp"/* + } >&2 && + makepkg --packagelist +') + +(( ${#files[@]} )) && printf '%s\n' '' '-> copying files...' && scp "${files[@]/#/$server:}" . From patchwork Tue Jun 4 16:11:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 1137 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 54DF7F4121DC for ; Tue, 4 Jun 2019 16:11:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on apollo X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,MAILING_LIST_MULTI=-1, RCVD_IN_DNSWL_MED=-2.3,SPF_HELO_NONE=0.001,T_DKIMWL_WL_HIGH=-0.01, T_DMARC_POLICY_NONE=0.01 autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Tue, 4 Jun 2019 16:11:41 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id E86FF1314808FA; Tue, 4 Jun 2019 16:11:36 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Tue, 4 Jun 2019 16:11:36 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id CB0B92C556; Tue, 4 Jun 2019 16:11:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1559664696; bh=IU71+mytbVA6O5CVVOyXwgZM6ZLEt1CJPOZtbaQ4MpA=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc; b=loh9RQ0K1naDeZMSPZ7FsTcyvjFCdn4GwRAprRiwifqTQ9EaVWtPY04BT67X7ZssF qcO/9iLdpvTTl5vqTz5/kcBFnIomm9pmYSRSPLYyT5xmp7xsXOnn4yGjoyIpXjLzVg ymCKpVBq1W40nLtiRxCJxyU3ZGdKb2iR08+VvX1noLydLGQK238Vdz1nho8vbulXjM zHWU/9EF/1cRcB9H8iTIw2cKcqqb7x7/mqTs0n9KJ2WpS8soP29h/+qabhUKX5px6I sZjWZODd/w8IZUvc8EWZ/l2x1qysi87m1awlpEXpqtHyytAwuGyoqJNmZbPQVS3vUL nNCi9fngq0EXEkuzPIWEvMMq7fOCJB6h0BEEXSMwzwrwDEIk/VftHvSjQZuZtQWAJn JiYkwkcXKoFf625fo1cOJ1D+lh0p929x5L6zatAhIUkW0WP4badmSCPDI4AVgRbPnM otrDwghPPY4NxEGEiM3h6x0Pr7X+4LCL92OJBEekDf2RiBFnCTKzuFR5iags2ZAiRu 328PG9BQWvTYY7VozsHMUV0KJTCSEfRh+UVmJHuFvOR4fEcvjV9NubpdLGctqe3ktS NV4/cJqTxqFNN/Li9gVEAkqLaGTSsEZlEDPt3lakTNXkbVPZISQ4HWuLF+qGbD40mz zacW6UWGfv77QWagxveuch0Q= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id B6BC02C530 for ; Tue, 4 Jun 2019 16:11:33 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Tue, 4 Jun 2019 16:11:33 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 638431314808F3; Tue, 4 Jun 2019 16:11:32 +0000 (UTC) Received: from didactylos.localdomain (unknown [216.97.144.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: eschwartz) by orion.archlinux.org (Postfix) with ESMTPSA id 8B6621314808F2; Tue, 4 Jun 2019 16:11:31 +0000 (UTC) To: arch-projects@archlinux.org Date: Tue, 4 Jun 2019 12:11:20 -0400 Message-Id: <20190604161120.4180-2-eschwartz@archlinux.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190604161120.4180-1-eschwartz@archlinux.org> References: <20190418011506.4350-1-eschwartz@archlinux.org> <20190604161120.4180-1-eschwartz@archlinux.org> MIME-Version: 1.0 Subject: [arch-projects] [devtools] [PATCH v3 2/2] doc: add manpage for the new offload-build tool X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eli Schwartz via arch-projects Reply-To: Arch Linux projects development discussion Cc: Eli Schwartz , svenstaro@gmail.com Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" Signed-off-by: Eli Schwartz --- v3: update manpage for new v3 parameters. Makefile | 1 + doc/offload-build.1.asciidoc | 52 ++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 doc/offload-build.1.asciidoc diff --git a/Makefile b/Makefile index 2306a17..27cd90f 100644 --- a/Makefile +++ b/Makefile @@ -69,6 +69,7 @@ BASHCOMPLETION_LINKS = \ MANS = \ doc/lddd.1 \ doc/checkpkg.1 \ + doc/offload-build.1 \ doc/sogrep.1 \ doc/mkarchroot.1 \ doc/find-libdeps.1 \ diff --git a/doc/offload-build.1.asciidoc b/doc/offload-build.1.asciidoc new file mode 100644 index 0000000..3633faa --- /dev/null +++ b/doc/offload-build.1.asciidoc @@ -0,0 +1,52 @@ +offload-build(1) +================ + +Name +---- +offload-build - Build a PKGBUILD on a remote server using makechrootpkg + +Synopsis +-------- +offload-build [OPTIONS] -- [ARCHBUILD_OPTIONS] + +Description +----------- + +Build a PKGBUILD on a remote server using makechrootpkg. Requires a remote user +that can run archbuild in a non-interactive manner, e.g. must be able to +elevate permissions using passwordless sudo. + +Options +------- + +*-r, --repo* :: + Build against a specific repository. The default is `extra`, to build packages using + the stable repositories via extra-x86_64-build. + +*-a, --arch* :: + Build against a specific architecture. The default is `x86_64`, the only + architecture officially supported by Arch Linux. + +*-s, --server* :: + Offload to a specific build server. The default is dragon.archlinux.org + which is used as part of the build toolchain for the official Arch Linux + repos. + +*-h, --help*:: + Show a help text. + +Passing options to archbuild +---------------------------- + +Options after a delimiting -- are passed on to archbuild on the remote. +archbuild in turn supports passing arguments on to makechrootpkg, which in turn +supports passing options to makepkg. Since each uses -- to delimit options that +are forwarded, make sure to escape them properly: + + `offload-build offload-args -- archbuild-args -- makechrootpkg-args -- makepkg-args` + +Example: To use a second `testing-x86_64-build` instance with another copydir: + + `offload-build -r testing -- -- -l ` + +include::footer.asciidoc[]