From patchwork Thu Mar 26 20:19:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anatol Pomozov X-Patchwork-Id: 1566 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 6C94317D3022D for ; Thu, 26 Mar 2020 20:20:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_ADSP_CUSTOM_MED=0.001, DKIM_INVALID=1,DKIM_SIGNED=0.1,FREEMAIL_FROM=0.5,MAILING_LIST_MULTI=-1, RCVD_IN_DNSWL_MED=-2.3,SPF_HELO_NONE=0.001,SUBJ_OBFU_PUNCT_FEW=0.749, T_DMARC_POLICY_NONE=0.01,T_DMARC_SIMPLE_DKIM=0.01 autolearn=ham autolearn_force=no version=3.4.4 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Thu, 26 Mar 2020 20:20:19 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id BEB5E1A707A29A; Thu, 26 Mar 2020 20:20:17 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) (Authenticated sender: luna) by orion.archlinux.org (Postfix) with ESMTPSA id D3B031A707A286; Thu, 26 Mar 2020 20:20:15 +0000 (UTC) Authentication-Results: orion.archlinux.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=i+Mw34Fh Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id B2AD02A953; Thu, 26 Mar 2020 20:20:15 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=i+Mw34Fh Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 84D6029CE6 for ; Thu, 26 Mar 2020 20:20:12 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Thu, 26 Mar 2020 20:20:12 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 69D421A707A280 for ; Thu, 26 Mar 2020 20:20:11 +0000 (UTC) Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Thu, 26 Mar 2020 20:20:11 +0000 (UTC) Received: by mail-pf1-x443.google.com with SMTP id f206so3346451pfa.10 for ; Thu, 26 Mar 2020 13:20:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XzQ0kifOyCh4jBWX9BqvWojajwkJ/5y/DueFKGZusvs=; b=i+Mw34Fh+08wWE7vvRihdEGRbkCWyKHcDsN/XQlf7d/ZEHB+qCeSRfIP3XmFAGzO3Z cF+CIaeUxDvL5W2naQyYti0t7Gxl2+i8gPN2PxQpMN6cSo5B2Tg9TB8H9CMQlRjpd8v0 AsSdQO4ZXk7Urk+exQfY4JsIUwKT1PkW/l26ojcwO1ZqM20MDqk4q3f0wq1Nf2VH8G33 F283fwN6JVj42Y5rPFZAwy7ky+ury9Fs2eOB817/V3HrvmnmM1zZPOxzn0L+EHdeIM6s Fh74g44gXWpC/fW9hPXDxtgCfEr59rsQ2LEqdnW3lQ/c4WLNcE/3awa56A5OIKC1b5+I K4ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XzQ0kifOyCh4jBWX9BqvWojajwkJ/5y/DueFKGZusvs=; b=rBTPHe+qpOdpUdejv+BeF3o5kDvtinYqdfvTpnfvKB4CJngKUTMQWk0x5Z1I80fIEX 0shIMnIscNZHwJ1HL4CVeFSy3piVIshcG3d6Btc+3RmHKg65zRKulh2Gy4CgeKEMsYwu peIDSWLIMw2CNPd9V1FmC4rm51XJFFv6/8FY6+nadOvqfop0qULRadS4ElTH860jGbHK 5P+KzoJNDkXH3NPoAVIo5BO3Tso7X3Ti6LW/NOUQCRImmVz4u//slp60mq+7KiE+Y1vG CJFE3acL/jS0ExAF6/naE1ILBDBy70dlNZaBgAHRzxqesGkWL4W6BqOSXuC0/1OyW+VR HXRg== X-Gm-Message-State: ANhLgQ0Mph/MDx1zHIR00JU94Io053lzhpH5cRmQsXm0vWSdiZs+YZXd bnOb82YwmBcHC0dhd6dY1s/i4fXZnc8= X-Google-Smtp-Source: ADFU+vsYbO+AMpXs9Azk9QJHGMeQaYbTFHOfcU4g/oPv0jSEF95Xw5nh7KA0ndOOE5gtFcVyMcr3kg== X-Received: by 2002:a63:925d:: with SMTP id s29mr10137461pgn.176.1585254008756; Thu, 26 Mar 2020 13:20:08 -0700 (PDT) Received: from wolf.hsd1.ca.comcast.net ([2601:646:8500:a860:2efd:a1ff:febf:a363]) by smtp.gmail.com with ESMTPSA id d26sm2359670pfo.37.2020.03.26.13.20.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Mar 2020 13:20:07 -0700 (PDT) From: Anatol Pomozov To: pacman-dev@archlinux.org Date: Thu, 26 Mar 2020 13:19:59 -0700 Message-Id: <20200326201959.498251-1-anatol.pomozov@gmail.com> X-Mailer: git-send-email 2.26.0 MIME-Version: 1.0 Subject: [pacman-dev] [PATCH v3] Introduce alpm_dbs_update() function for parallel db updates X-BeenThere: pacman-dev@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for pacman development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Discussion list for pacman development Errors-To: pacman-dev-bounces@archlinux.org Sender: "pacman-dev" This is an equivalent of alpm_db_update but for multiplexed (parallel) download. The difference is that this function accepts list of databases to update. And then ALPM internals download it in parallel if possible. Add a stub for _alpm_multi_download the function that will do parallel payloads downloads in the future. Introduce dload_payload->filepath field that contains url path to the file we download. It is like fileurl field but does not contain protocol/server part. The rationale for having this field is that with the curl multidownload the server retry logic is going to move to a curl callback. And the callback needs to be able to reconstruct the 'next' fileurl. One will be able to do it by getting the next server url from 'servers' list and then concat with filepath. Once the 'parallel download' refactoring is over 'fileurl' field will go away. Signed-off-by: Anatol Pomozov --- lib/libalpm/alpm.h | 29 ++++++++++ lib/libalpm/be_sync.c | 128 ++++++++++++++++++++++++++++++++++++++++++ lib/libalpm/dload.c | 12 ++++ lib/libalpm/dload.h | 5 ++ 4 files changed, 174 insertions(+) diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 5d559db1..2cf20343 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -1049,6 +1049,35 @@ int alpm_db_remove_server(alpm_db_t *db, const char *url); */ int alpm_db_update(int force, alpm_db_t *db); +/** Update package databases + * + * An update of the package databases in the list \a dbs will be attempted. + * Unless \a force is true, the update will only be performed if the remote + * databases were modified since the last update. + * + * This operation requires a database lock, and will return an applicable error + * if the lock could not be obtained. + * + * Example: + * @code + * alpm_list_t *dbs = alpm_get_syncdbs(); + * ret = alpm_dbs_update(config->handle, dbs, force); + * if(ret < 0) { + * pm_printf(ALPM_LOG_ERROR, _("failed to synchronize all databases (%s)\n"), + * alpm_strerror(alpm_errno(config->handle))); + * } + * @endcode + * + * @note After a successful update, the \link alpm_db_get_pkgcache() + * package cache \endlink will be invalidated + * @param handle the context handle + * @param dbs list of package databases to update + * @param force if true, then forces the update, otherwise update only in case + * the databases aren't up to date + * @return 0 on success, -1 on error (pm_errno is set accordingly) + */ +int alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force); + /** Get a package entry from a package database. * @param db pointer to the package database to get the package from * @param name of the package diff --git a/lib/libalpm/be_sync.c b/lib/libalpm/be_sync.c index aafed15d..7ab52301 100644 --- a/lib/libalpm/be_sync.c +++ b/lib/libalpm/be_sync.c @@ -301,6 +301,134 @@ int SYMEXPORT alpm_db_update(int force, alpm_db_t *db) return ret; } +int SYMEXPORT alpm_dbs_update(alpm_handle_t *handle, alpm_list_t *dbs, int force) { + char *syncpath; + const char *dbext = handle->dbext; + alpm_list_t *i; + int ret = -1; + mode_t oldmask; + alpm_list_t *payloads = NULL; + + /* Sanity checks */ + CHECK_HANDLE(handle, return -1); + ASSERT(dbs != NULL, return -1); + handle->pm_errno = ALPM_ERR_OK; + + syncpath = get_sync_dir(handle); + ASSERT(syncpath != NULL, return -1); + + /* make sure we have a sane umask */ + oldmask = umask(0022); + + /* attempt to grab a lock */ + if(_alpm_handle_lock(handle)) { + GOTO_ERR(handle, ALPM_ERR_HANDLE_LOCK, cleanup); + } + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + int dbforce = force; + struct dload_payload *payload = NULL; + size_t len; + int siglevel; + + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + ASSERT(db != handle->db_local, GOTO_ERR(handle, ALPM_ERR_WRONG_ARGS, cleanup)); + ASSERT(db->servers != NULL, GOTO_ERR(handle, ALPM_ERR_SERVER_NONE, cleanup)); + + /* force update of invalid databases to fix potential mismatched database/signature */ + if(db->status & DB_STATUS_INVALID) { + dbforce = 1; + } + + CALLOC(payload, 1, sizeof(*payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + + /* set hard upper limit of 128 MiB */ + payload->max_size = 128 * 1024 * 1024; + payload->servers = db->servers; + + /* print server + filename into a buffer */ + len = strlen(db->treename) + strlen(dbext) + 1; + MALLOC(payload->filepath, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(payload->filepath, len, "%s%s", db->treename, dbext); + payload->handle = handle; + payload->force = dbforce; + payload->unlink_on_fail = 1; + + payloads = alpm_list_add(payloads, payload); + + siglevel = alpm_db_get_siglevel(db); + if(siglevel & ALPM_SIG_DATABASE) { + struct dload_payload *sig_payload; + CALLOC(sig_payload, 1, sizeof(*sig_payload), GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + + /* print filename into a buffer (leave space for separator and .sig) */ + len = strlen(db->treename) + strlen(dbext) + 5; + MALLOC(sig_payload->filepath, len, GOTO_ERR(handle, ALPM_ERR_MEMORY, cleanup)); + snprintf(sig_payload->filepath, len, "%s%s.sig", db->treename, dbext); + + sig_payload->handle = handle; + sig_payload->force = 1; + sig_payload->errors_ok = (siglevel & ALPM_SIG_DATABASE_OPTIONAL); + + /* set hard upper limit of 16 KiB */ + sig_payload->max_size = 16 * 1024; + sig_payload->servers = db->servers; + + payloads = alpm_list_add(payloads, sig_payload); + } + } + + ret = _alpm_multi_download(handle, payloads, syncpath); + if(ret < 0) { + goto cleanup; + } + + for(i = dbs; i; i = i->next) { + alpm_db_t *db = i->data; + if(!(db->usage & ALPM_DB_USAGE_SYNC)) { + continue; + } + + /* Cache needs to be rebuilt */ + _alpm_db_free_pkgcache(db); + + /* clear all status flags regarding validity/existence */ + db->status &= ~DB_STATUS_VALID; + db->status &= ~DB_STATUS_INVALID; + db->status &= ~DB_STATUS_EXISTS; + db->status &= ~DB_STATUS_MISSING; + + /* if the download failed skip validation to preserve the download error */ + if(sync_db_validate(db) != 0) { + /* pm_errno should be set */ + ret = -1; + } + } + +cleanup: + _alpm_handle_unlock(handle); + + if(ret == -1) { + /* pm_errno was set by the download code */ + _alpm_log(handle, ALPM_LOG_DEBUG, "failed to sync dbs: %s\n", + alpm_strerror(handle->pm_errno)); + } else { + handle->pm_errno = ALPM_ERR_OK; + } + + if(payloads) { + alpm_list_free_inner(payloads, (alpm_list_fn_free)_alpm_dload_payload_reset); + FREELIST(payloads); + } + free(syncpath); + umask(oldmask); + return ret; +} + /* Forward decl so I don't reorganize the whole file right now */ static int sync_db_read(alpm_db_t *db, struct archive *archive, struct archive_entry *entry, alpm_pkg_t **likely_pkg); diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 670da03d..7cd3e3a4 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -636,6 +636,16 @@ int _alpm_download(struct dload_payload *payload, const char *localpath, } } +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath) +{ + (void)handle; + (void)payloads; + (void)localpath; + return 0; +} + static char *filecache_find_url(alpm_handle_t *handle, const char *url) { const char *filebase = strrchr(url, '/'); @@ -738,6 +748,7 @@ void _alpm_dload_payload_reset(struct dload_payload *payload) FREE(payload->destfile_name); FREE(payload->content_disp_name); FREE(payload->fileurl); + FREE(payload->filepath); *payload = (struct dload_payload){0}; } @@ -746,6 +757,7 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload) ASSERT(payload, return); FREE(payload->fileurl); + FREE(payload->filepath); payload->initial_size += payload->prevprogress; payload->prevprogress = 0; payload->unlink_on_fail = 0; diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index 1e8f75f3..3eb7fbe1 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -31,6 +31,7 @@ struct dload_payload { char *destfile_name; char *content_disp_name; char *fileurl; + char *filepath; /* download URL path */ alpm_list_t *servers; long respcode; off_t initial_size; @@ -53,4 +54,8 @@ void _alpm_dload_payload_reset_for_retry(struct dload_payload *payload); int _alpm_download(struct dload_payload *payload, const char *localpath, char **final_file, const char **final_url); +int _alpm_multi_download(alpm_handle_t *handle, + alpm_list_t *payloads /* struct dload_payload */, + const char *localpath); + #endif /* ALPM_DLOAD_H */