From patchwork Sun Mar 8 20:33:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anatol Pomozov X-Patchwork-Id: 1548 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 1DEE61781B883 for ; Sun, 8 Mar 2020 20:33:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-0.1 required=5.0 tests=DKIM_ADSP_CUSTOM_MED=0.001, DKIM_INVALID=1,DKIM_SIGNED=0.1,FREEMAIL_FROM=0.5,MAILING_LIST_MULTI=-1, RCVD_IN_DNSWL_LOW=-0.7,SPF_HELO_NONE=0.001,T_DMARC_POLICY_NONE=0.01, T_DMARC_SIMPLE_DKIM=0.01 autolearn=ham autolearn_force=no version=3.4.4 X-Spam-BL-Results: [127.0.9.1] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 8 Mar 2020 20:33:53 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 7081419D72EFF9; Sun, 8 Mar 2020 20:33:45 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) (Authenticated sender: luna) by orion.archlinux.org (Postfix) with ESMTPSA id 3C6CD19D72EFF2; Sun, 8 Mar 2020 20:33:45 +0000 (UTC) Authentication-Results: orion.archlinux.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=HY7peGSb Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 307B92A8EF; Sun, 8 Mar 2020 20:33:45 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=HY7peGSb Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id D83392A8EE for ; Sun, 8 Mar 2020 20:33:42 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 8 Mar 2020 20:33:42 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 43B9119D72EFEC for ; Sun, 8 Mar 2020 20:33:37 +0000 (UTC) Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Sun, 8 Mar 2020 20:33:37 +0000 (UTC) Received: by mail-pl1-x643.google.com with SMTP id y1so3123566plp.7 for ; Sun, 08 Mar 2020 13:33:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=VjcL+KBw8uIlO37rq1VrEqlsM5QbpyxGspqtdcNF/IM=; b=HY7peGSb2Ddt0Fl3ItHSvCcc1I3sKld8vwbj2coO28i7njuFNCdV173iW4PQfIp9xt Os40Pd39yj6PVQuG9kwvJtO4zgRN4da0islj5CmHaer30EQodXhRQ4NMdh/MgRXiXdE4 qOEvVyizVT+0DITnHVuOkY39ASMAcsagkqnN0Ge6ij4h50lBCxfOF6WkRn7ZuRaNRcaE CgQJu57VBtUpBqmoO6vHjxwZjyIY+D0NqNXyK8YD0ZTcmJG2LveCkWU/pFC3oBe73ljz v1fNZtpV8Mvpp5FJGU77WsgxFtywNp0XN+WLLB480453/fmVfnlV+uI/GYNld0Jo2Qn6 FFKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=VjcL+KBw8uIlO37rq1VrEqlsM5QbpyxGspqtdcNF/IM=; b=Qbl2kNgIWwvcgQOtxrrW1sodAJmCwx/q7TZEayYnltuGuaVN/vDu3elWzkgQgdA+87 fK2Dn1rkBNDeJBfrmmuGDjGB2i1aTEbpPmLikPUQKtbRQ8PRj6TSmAJ0W8ux4aDASlsG HARebFRlUQsbYE/bnIv7usR/d0rHelKmzXvDYpsb4QiHtD2waZNcR+6KfmX2gtNdtB98 6o55jK59Z2GQWw/lYPQogmjPVEgdX+2k8btcSsMQR3WiMFWFt/PBBsHDjPYktO5RG/63 0WYsszsDhx+O05bjfZwzWnj9USQ091Scrk0xq3MO3g+rKr0AZzakDjlgTL3OABwiqrtb ar4w== X-Gm-Message-State: ANhLgQ2f5PDGAUYoa9BjXhzoSX+nVdAdUc3FyjUS3BkNoltM7qRa4FGc FmraWE/qsc4ATgzdqJyDijDGXyJvOBo= X-Google-Smtp-Source: ADFU+vuhgR6rf9QRQ83g0Tuzs9dvtbuRVcuZO3/9xLevQtLzfNTg9CKBxJ5lu2ETS1nxeBNQOS/XTg== X-Received: by 2002:a17:902:8a8d:: with SMTP id p13mr12681252plo.159.1583699615050; Sun, 08 Mar 2020 13:33:35 -0700 (PDT) Received: from wolf.hsd1.ca.comcast.net ([2601:646:8500:a860:2efd:a1ff:febf:a363]) by smtp.gmail.com with ESMTPSA id a9sm41807003pfo.35.2020.03.08.13.33.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Mar 2020 13:33:34 -0700 (PDT) From: Anatol Pomozov To: pacman-dev@archlinux.org Date: Sun, 8 Mar 2020 13:33:32 -0700 Message-Id: <20200308203332.52326-1-anatol.pomozov@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [pacman-dev] [PATCH v3] Add config option to specify amount of parallel download streams X-BeenThere: pacman-dev@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for pacman development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Discussion list for pacman development Errors-To: pacman-dev-bounces@archlinux.org Sender: "pacman-dev" It includes pacman.conf new 'ParallelDownloads' option that specifies how many concurrent downloads cURL starts in parallel. Add alpm_option_set_parallel_downloads() ALPM function that allows to set this config option programmatically. Signed-off-by: Anatol Pomozov --- doc/pacman.conf.5.asciidoc | 5 ++++ etc/pacman.conf.in | 1 + lib/libalpm/alpm.h | 9 ++++++ lib/libalpm/handle.c | 15 ++++++++++ lib/libalpm/handle.h | 1 + src/pacman/conf.c | 56 ++++++++++++++++++++++++++++++++++++++ src/pacman/conf.h | 2 ++ 7 files changed, 89 insertions(+) diff --git a/doc/pacman.conf.5.asciidoc b/doc/pacman.conf.5.asciidoc index b297e332..a1b708b9 100644 --- a/doc/pacman.conf.5.asciidoc +++ b/doc/pacman.conf.5.asciidoc @@ -205,6 +205,11 @@ Options Disable defaults for low speed limit and timeout on downloads. Use this if you have issues downloading files with proxy and/or security gateway. +*ParallelDownloads*:: + Specifies number of concurrent download streams. The value needs to be a + positive integer. If this config option is not set then only one download + stream is used (i.e. downloads happens sequentially). + Repository Sections ------------------- diff --git a/etc/pacman.conf.in b/etc/pacman.conf.in index 7446944f..44cea408 100644 --- a/etc/pacman.conf.in +++ b/etc/pacman.conf.in @@ -34,6 +34,7 @@ Architecture = auto #TotalDownload CheckSpace #VerbosePkgLists +ParallelDownloads = 5 # PGP signature checking #SigLevel = Optional diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index c2a069ad..5d559db1 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -903,6 +903,15 @@ int alpm_option_set_remote_file_siglevel(alpm_handle_t *handle, int level); int alpm_option_set_disable_dl_timeout(alpm_handle_t *handle, unsigned short disable_dl_timeout); +/** Sets number of parallel streams to download database and package files. + * If the function is not called then the default value of '1' stream + * (i.e. sequential download) is used. + * @param handle the context handle + * @param num_streams number of parallel download streams + * @return 0 on success, -1 on error + */ +int alpm_option_set_parallel_downloads(alpm_handle_t *handle, unsigned int num_streams); + /** @} */ /** @addtogroup alpm_api_databases Database Functions diff --git a/lib/libalpm/handle.c b/lib/libalpm/handle.c index fc7c1faf..8ae08d11 100644 --- a/lib/libalpm/handle.c +++ b/lib/libalpm/handle.c @@ -856,3 +856,18 @@ int SYMEXPORT alpm_option_set_disable_dl_timeout(alpm_handle_t *handle, #endif return 0; } + +int SYMEXPORT alpm_option_set_parallel_downloads(alpm_handle_t *handle, + unsigned int num_streams) +{ + CHECK_HANDLE(handle, return -1); +#ifdef HAVE_LIBCURL + if(num_streams < 1) { + return -1; + } + handle->parallel_downloads = num_streams; +#else + (void)num_streams; /* silence unused variable warnings */ +#endif + return 0; +} diff --git a/lib/libalpm/handle.h b/lib/libalpm/handle.h index c343f6e0..cd7104f9 100644 --- a/lib/libalpm/handle.h +++ b/lib/libalpm/handle.h @@ -61,6 +61,7 @@ struct __alpm_handle_t { /* libcurl handle */ CURL *curl; /* reusable curl_easy handle */ unsigned short disable_dl_timeout; + unsigned int parallel_downloads; /* number of download streams */ #endif #ifdef HAVE_LIBGPGME diff --git a/src/pacman/conf.c b/src/pacman/conf.c index f9de386f..6d731551 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -111,6 +111,8 @@ config_t *config_new(void) newconfig->remotefilesiglevel = ALPM_SIG_USE_DEFAULT; } + /* by default use 1 download stream */ + newconfig->parallel_downloads = 1; newconfig->colstr.colon = ":: "; newconfig->colstr.title = ""; newconfig->colstr.repo = ""; @@ -400,6 +402,32 @@ int config_set_arch(const char *arch) return 0; } +/** + * Parse a string into long number. The input string has to be non-empty + * and represent a number that fits long type. + * @param value the string to parse + * @param result pointer to long where the final result will be stored. + * This result is modified iff the input string parsed successfully. + * @return 0 in case if value parsed successfully, 1 otherwise. + */ +static int parse_number(char *value, long *result) { + char *endptr; + long val; + int invalid; + + errno = 0; /* To distinguish success/failure after call */ + val = strtol(value, &endptr, 10); + invalid = (errno == ERANGE && (val == LONG_MAX || val == LONG_MIN)) + || (*endptr != '\0') + || (endptr == value); + + if(!invalid) { + *result = val; + } + + return invalid; +} + /** * Parse a signature verification level line. * @param values the list of parsed option values @@ -677,6 +705,33 @@ static int _parse_options(const char *key, char *value, return 1; } FREELIST(values); + } else if(strcmp(key, "ParallelDownloads") == 0) { + long number; + int err; + + err = parse_number(value, &number); + if(err) { + pm_printf(ALPM_LOG_WARNING, + _("config file %s, line %d: invalid value for '%s' : '%s'\n"), + file, linenum, "ParallelDownloads", value); + return 1; + } + + if(number < 1) { + pm_printf(ALPM_LOG_WARNING, + _("config file %s, line %d: value for '%s' has to be positive : '%s'\n"), + file, linenum, "ParallelDownloads", value); + return 1; + } + + if(number > INT_MAX) { + pm_printf(ALPM_LOG_WARNING, + _("config file %s, line %d: value for '%s' is too large : '%s'\n"), + file, linenum, "ParallelDownloads", value); + return 1; + } + + config->parallel_downloads = number; } else { pm_printf(ALPM_LOG_WARNING, _("config file %s, line %d: directive '%s' in section '%s' not recognized.\n"), @@ -845,6 +900,7 @@ static int setup_libalpm(void) alpm_option_set_noextracts(handle, config->noextract); alpm_option_set_disable_dl_timeout(handle, config->disable_dl_timeout); + alpm_option_set_parallel_downloads(handle, config->parallel_downloads); for(i = config->assumeinstalled; i; i = i->next) { char *entry = i->data; diff --git a/src/pacman/conf.h b/src/pacman/conf.h index d954e637..42cddfbb 100644 --- a/src/pacman/conf.h +++ b/src/pacman/conf.h @@ -115,6 +115,8 @@ typedef struct __config_t { /* When downloading, display the amount downloaded, rate, ETA, and percent * downloaded of the total download list */ unsigned short totaldownload; + /* number of parallel download streams */ + unsigned int parallel_downloads; /* select -Sc behavior */ unsigned short cleanmethod; alpm_list_t *holdpkg;