From patchwork Fri Nov 1 20:45:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Fleischer X-Patchwork-Id: 1324 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id DBCD614D43657 for ; Fri, 1 Nov 2019 20:45:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on apollo X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1,DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1, LOCAL_FAKEBUSINESS=0.5,MAILING_LIST_MULTI=-1,RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001,T_DMARC_POLICY_NONE=0.01 autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-BL-Results: [127.0.9.2] [127.0.0.11] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Fri, 1 Nov 2019 20:45:03 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id B5C311639AD1BA; Fri, 1 Nov 2019 20:44:58 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Fri, 1 Nov 2019 20:44:58 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 8984A2CB08; Fri, 1 Nov 2019 20:44:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1572641098; bh=PtAbfWX9Qgl5BgrffgHdALrVfIlJ3IDyLbFZFdcIET4=; h=From:To:Subject:Date:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe; b=sIQX6qH+Za7ibQT3K+BOFZpJemua1Bg1r+Pf4/7EOxJAB68zSkFTeDfy7SHkGycIW uWPyDRiURT6BoTVTOM2sbbeYakev2Z0OtrD5m3jtnEVFh860eGJ0w+XpGj6T/1O7NO XcidN06WnrID2qaReubpzPKtmErWyuDoUnylimiW+ryNtV8Nxzx/OiTcam1ytc85tp t6Lrm53e5GTMcyNao2jZVK/4jNgO3wwjvMBZgTpgo7dqSh6Ohvl+h6iFtcm+USZ3k/ 4K5zsHr8O7CHWhzbmBNcwp89AseJRgyE0Sle95QZc4+FOt1EQKhaYck5KbqtFPprbX pBqy2j3NdVwPTQHXThdI7TM2p6SdJsojr9rjVqiDn42fy9eu5u5D4loxFaS2UqIod7 nLeKcTWY07evlnFnnYeYMyh6UotHjSZ67OdbkrdkLN7/myQ2HROu/Ot0dbrX2ey/Is x/pV9B+GiXjjYpBs0J3TI89lhriwS2y/LcYkQvW0pIzjAmxB3RZYuDtgk5vOAMFL1r rzow8B6AbMR7Nhc12+09636tyrM0vOdq2G7ZSKvHseeSPEB9ZUEjrQK703HbNAniC9 OaU3kXva1LEYuHAMxAczwmaHU/5ydrGqhEwYUn1qRDGl6YHEvlTHBNImfLEIQ+/5Q/ AFqrJinuT24uDp7ozmEVfGrc= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 6A2AF2CB05 for ; Fri, 1 Nov 2019 20:44:55 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Fri, 1 Nov 2019 20:44:55 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 287CB1639AD1B8 for ; Fri, 1 Nov 2019 20:44:50 +0000 (UTC) Received: from localhost (wn-campus-nat-129-97-124-103.dynamic.uwaterloo.ca [129.97.124.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: lfleischer) by orion.archlinux.org (Postfix) with ESMTPSA id C96061639AD1B7 for ; Fri, 1 Nov 2019 20:44:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=orion; t=1572641090; bh=PtAbfWX9Qgl5BgrffgHdALrVfIlJ3IDyLbFZFdcIET4=; h=From:To:Subject:Date; b=XduGtOCY5IA1cW/ZoEyUCD9I+o3kf4C1BnCc8U/LfQVtsLNY1qYForKlZ6PTol+G7 UPHzZyxn4Sq0eKZAjjjizAOvOg7+WMuoMM69QpjBBSmmX3hwYYSTUJBNxBQJfAhy4w OrR4VEzeqF1iN71MPwfC52AHie2XYEj0XV41mPlF9YPIsd6IA2pxHObF4u9fxS2jCJ rfeUIoqWbEhsiylFsZlRAWqgJ5OOrIioBDJWGAfo7V7uZDecREeLdV8zpSM1EiFmeC YQnYTH5uDJvON0u2GUOcUZw75yC86i+N2fC5BONAGtbLv2bzBMuuW6Oilp8O8k9QRt Z6qhFWQXsIvWItWhkxPnzTyjyrhpXjyYl/AIPD9aTEUTzh/6CgC3Rx46+biIpAXpQ1 dU08LHmhUoHzIj4gHykk5vyfumlUkgtoxtU5qRuD52mhmR/9SyKCQfXnmE8XTIk2EU TUb+w0c+GIMc2boUvQMoBtIldb09HPnmI1Wcn0FSvs3xKGVXkoemtzSsdprblcAL8+ aidrrX0FH5RL4WAk9p0xOrkmQfXH8scgWjYHf08VbCJGrxYT9Zhuq+Lb345Ud9eGor FnMzJVuB50R/gKnp2iNjdKaNkmd1CBFjFi36ebAZq9ZAfV74fI+rrNaQL0R4l+bcz8 +UCXkRXjr8NVRz1EcJTFCWuA= From: Lukas Fleischer To: aur-dev@archlinux.org Subject: [PATCH] Document maintenance tasks and internals Date: Fri, 1 Nov 2019 16:45:45 -0400 Message-Id: <20191101204545.21147-1-lfleischer@archlinux.org> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 X-BeenThere: aur-dev@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Arch User Repository \(AUR\) Development" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: aur-dev-bounces@archlinux.org Sender: "aur-dev" Signed-off-by: Lukas Fleischer --- doc/maintenance.txt | 108 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 doc/maintenance.txt diff --git a/doc/maintenance.txt b/doc/maintenance.txt new file mode 100644 index 0000000..d609454 --- /dev/null +++ b/doc/maintenance.txt @@ -0,0 +1,108 @@ +aurweb Maintenance +================== + +Components +---------- + +aurweb has three user-facing components. + +The Git/SSH interface can be used to update package repositories and for basic +package maintenance from the command-line. More details can be found in +`doc/git-interface.txt`. + +The web interface can be used to browse packages, view package details, manage +aurweb accounts, add comments, vote for packages, flag packages, and submit +requests. Trusted Users can update package maintainers and delete/merge +packages. The web interface also includes an area for Trusted Users to post +AUR-related proposals and vote on them. + +The RPC interface can be used to query package information via HTTP. + +Installation +------------ + +The web backend requires a web server with PHP and an SQL database. The Git/SSH +interface requires Python, several Python modules and an up-to-date version of +Git. APCu or memcached can be used to reduce load on the database server. + +All dependencies and the full installation process are described in `INSTALL`. + +Updates +------- + +The `enable-maintenance` option (in the configuration file, usually located at +`/etc/aurweb/config`) can be used to switch aurweb into maintenance mode. This +disables both the Git/SSH interface and the web interface. The +`maintenance-exceptions` variable can be used to reactivate access for certain +IP addresses. Since changes to the database schema might temporarily break +parts of the backend, it is recommended to always enable maintenance mode +before performing an upgrade. + +To simplify the upgrade process, changes in the database schema (and other +changes that require manual interaction) are documented in `upgrading/`. An +exception are additions to the configuration file. It is recommended to always +compare `/etc/aurweb/config` to `conf/config.defaults` when upgrading to a new +release. + +Moreover, the aurweb Python modules and translations need to be reinstalled +with every upgrade. To this end, run `python3 setup.py install` from the aurweb +source tree and run `make install` in the `po/` subdirectory. + +Don't forget to always test all basic features first, then disable maintenance +mode after performing an upgrade. + +Maintenance Scripts +------------------- + +aurweb includes scheduled maintenance routines to perform expensive +computations and clean up the database: + +* aurweb-aurblup parses binary repositories and updates the `OfficialProviders` + table. This table is used to identify AUR packages that depend on packages in + the official repositories. It is also used to prevent users from uploading + packages that are in the official repositories already. + +* aurweb-tuvotereminder sends out reminders to TUs if the voting period for a + TU proposal ends soon. + +* aurweb-popupdate is used to recompute the popularity score of packages. + +* aurweb-pkgmaint automatically removes empty repositories that were created + within the last 24 hours but never populated. + +* aurweb-mkpkglists generates the package list files. + +* aurweb-usermaint removes the last login IP address of all users that did not + login within the past seven days. + +These scripts can be installed by running `python3 setup.py install` and are +usually scheduled using Cron. The current setup is: + +---- +*/5 * * * * aurweb-mkpkglists +1 */2 * * * aurweb-popupdate +2 */2 * * * aurweb-aurblup +3 */2 * * * aurweb-pkgmaint +4 */2 * * * aurweb-usermaint +5 */12 * * * aurweb-tuvotereminder +---- + +Advanced Administrative Features +-------------------------------- + +Trusted Users can set the AUR_OVERWRITE environment variable to enable +non-fast-forward pushes to the Git repositories. This feature is documented in +`doc/git-interface.txt`. + +Rate limiting is used to prevent users from hammering the RPC interface. The +`request_limit` and `window_length` options in the `ratelimit` section of the +configuration file can be used to configure this feature. Recent accesses are +stored in the `ApiRateLimit` table in the database. See commit 27654af (Add +rate limit support to API, 2018-02-01) for details. + +The database contains a `PackageBlacklist` table. Package names added to this +table will be rejected by the SSH/Git interface. This table can only be edited +by a database administrator. + +The `Bans` table can be used to ban certain IP addresses from both the web and +Git/SSH interface. This table can only be accessed by a database administrator.