From patchwork Mon Jan 24 11:48:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Hesse X-Patchwork-Id: 2018 Return-Path: Delivered-To: patchwork@archlinux.org Received: from mail.archlinux.org [95.216.189.61] by patchwork.archlinux.org with IMAP (fetchmail-6.4.26) for (single-drop); Mon, 24 Jan 2022 11:48:47 +0000 (UTC) Received: from mail.archlinux.org by mail.archlinux.org with LMTP id aHVHEh+S7mFn5gkAK+/4rw (envelope-from ) for ; Mon, 24 Jan 2022 11:48:47 +0000 Received: from lists.archlinux.org (lists.archlinux.org [95.217.236.249]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.archlinux.org (Postfix) with ESMTPS id B6892C6E4A2; Mon, 24 Jan 2022 11:48:45 +0000 (UTC) Received: from lists.archlinux.org (localhost [IPv6:::1]) by lists.archlinux.org (Postfix) with ESMTP id 8A0C8C15B0D; Mon, 24 Jan 2022 11:48:45 +0000 (UTC) X-Original-To: pacman-dev@lists.archlinux.org Delivered-To: pacman-dev@lists.archlinux.org Received: from mx.mylinuxtime.de (mx.mylinuxtime.de [195.201.174.144]) by lists.archlinux.org (Postfix) with ESMTPS id 42837C15AFC for ; Mon, 24 Jan 2022 11:48:43 +0000 (UTC) Received: from leda.eworm.de (p5085a9aa.dip0.t-ipconnect.de [80.133.169.170]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mx.mylinuxtime.de (Postfix) with ESMTPSA id 9C75517F504; Mon, 24 Jan 2022 12:48:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eworm.de; s=mail; t=1643024922; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uj3MwcHVRCLz44AQC6q2i5PE0UMmqgumI4JTNyz66OI=; b=myLdW7lkZ2UQGmxjLpUD7wIgmsyt2dOOeEoz/TsSVrhhefj3knO3jA2H61+H+U0bT85Zjn Z+PZ9JwcvdyAabiKpBpfuLdb5BsURuhJ/55DqZnBGGnm2dBoYAoFVRIfdQp5bZyzvVpHjP z6V+25Wo5nBSRRaq82yzeJDFjEmBvAc= Received: by leda.eworm.de (Postfix, from userid 1000) id 58F431A09E2; Mon, 24 Jan 2022 12:48:12 +0100 (CET) From: Christian Hesse To: Discussion list for pacman development Cc: Christian Hesse Subject: [PATCH v2 1/1] implement CacheServer Date: Mon, 24 Jan 2022 12:48:10 +0100 Message-Id: <20220124114810.22154-1-list@eworm.de> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220124114252.48c3003f@leda.eworm.net> References: <20220124114252.48c3003f@leda.eworm.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.80 X-Stat-Signature: e5m5izs655bwdzr7diemsh6eztr45hrr X-BeenThere: pacman-dev@lists.archlinux.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: Discussion list for pacman development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pacman-dev-bounces@lists.archlinux.org Sender: "pacman-dev" Authentication-Results: mail.archlinux.org; dkim=pass header.d=eworm.de header.s=mail header.b=myLdW7lk; dmarc=none; spf=pass (mail.archlinux.org: domain of pacman-dev-bounces@lists.archlinux.org designates 95.217.236.249 as permitted sender) smtp.mailfrom=pacman-dev-bounces@lists.archlinux.org X-Rspamd-Queue-Id: B6892C6E4A2 X-Spamd-Result: default: False [0.09 / 15.00]; MID_CONTAINS_FROM(1.00)[]; R_MISSING_CHARSET(0.50)[]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; MAILLIST(-0.20)[mailman]; RCVD_IN_DNSWL_MED(-0.20)[95.217.236.249:from]; R_SPF_ALLOW(-0.20)[+ip4:95.217.236.249:c]; R_DKIM_ALLOW(-0.20)[eworm.de:s=mail]; MIME_GOOD(-0.10)[text/plain]; HAS_LIST_UNSUB(-0.01)[]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DMARC_NA(0.00)[eworm.de: no valid DMARC record]; PREVIOUSLY_DELIVERED(0.00)[pacman-dev@lists.archlinux.org]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:95.217.0.0/16, country:DE]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[eworm.de:+]; TO_DN_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[list@eworm.de,pacman-dev-bounces@lists.archlinux.org]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; MIME_TRACE(0.00)[0:+]; RECEIVED_SPAMHAUS_PBL(0.00)[80.133.169.170:received]; NEURAL_HAM(-0.00)[-1.000]; FORGED_SENDER_MAILLIST(0.00)[] X-Rspamd-Server: mail.archlinux.org From: Christian Hesse This implements a new configuration option 'CacheServer'. Adding a cache server makes it ignore the server error limit. We have a struct that stores the server errors. Extend (and rename) this struct to store if this is a cache server. The errors are not increased for cache servers, thus they are never ignored. --- doc/pacman.conf.5.asciidoc | 4 +++ lib/libalpm/alpm.h | 2 +- lib/libalpm/db.c | 12 ++++++--- lib/libalpm/db.h | 1 + lib/libalpm/dload.c | 50 ++++++++++++++++++++++++++++++++------ lib/libalpm/dload.h | 2 ++ src/pacman/conf.c | 16 +++++++++--- src/pacman/conf.h | 1 + src/pacman/pacman-conf.c | 1 + 9 files changed, 74 insertions(+), 15 deletions(-) diff --git a/doc/pacman.conf.5.asciidoc b/doc/pacman.conf.5.asciidoc index 77a3907f..1dfacf22 100644 --- a/doc/pacman.conf.5.asciidoc +++ b/doc/pacman.conf.5.asciidoc @@ -243,6 +243,10 @@ number. *Server =* url:: A full URL to a location where the database, packages, and signatures (if available) for this repository can be found. + +*CacheServer =* url:: + A full URL to a location where packages, and signatures (if available) + for this repository can be found. There's no server error limit. + During parsing, pacman will define the `$repo` variable to the name of the current section. This is often utilized in files specified using the 'Include' diff --git a/lib/libalpm/alpm.h b/lib/libalpm/alpm.h index 50b5e3d2..2b19338e 100644 --- a/lib/libalpm/alpm.h +++ b/lib/libalpm/alpm.h @@ -1325,7 +1325,7 @@ int alpm_db_set_servers(alpm_db_t *db, alpm_list_t *servers); * @param url url of the server * @return 0 on success, -1 on error (pm_errno is set accordingly) */ -int alpm_db_add_server(alpm_db_t *db, const char *url); +int alpm_db_add_server(alpm_db_t *db, const char *url, const int cache); /** Remove a download server from a database. * @param db database pointer diff --git a/lib/libalpm/db.c b/lib/libalpm/db.c index 4a234f9c..17225809 100644 --- a/lib/libalpm/db.c +++ b/lib/libalpm/db.c @@ -36,6 +36,7 @@ #include "alpm.h" #include "package.h" #include "group.h" +#include "dload.h" alpm_db_t SYMEXPORT *alpm_register_syncdb(alpm_handle_t *handle, const char *treename, int siglevel) @@ -137,6 +138,7 @@ alpm_list_t SYMEXPORT *alpm_db_get_servers(const alpm_db_t *db) return db->servers; } +/* Is this used at all? int SYMEXPORT alpm_db_set_servers(alpm_db_t *db, alpm_list_t *servers) { alpm_list_t *i; @@ -150,6 +152,7 @@ int SYMEXPORT alpm_db_set_servers(alpm_db_t *db, alpm_list_t *servers) } return 0; } +*/ static char *sanitize_url(const char *url) { @@ -164,7 +167,7 @@ static char *sanitize_url(const char *url) return newurl; } -int SYMEXPORT alpm_db_add_server(alpm_db_t *db, const char *url) +int SYMEXPORT alpm_db_add_server(alpm_db_t *db, const char *url, const int cache) { char *newurl; @@ -178,8 +181,11 @@ int SYMEXPORT alpm_db_add_server(alpm_db_t *db, const char *url) return -1; } db->servers = alpm_list_add(db->servers, newurl); - _alpm_log(db->handle, ALPM_LOG_DEBUG, "adding new server URL to database '%s': %s\n", - db->treename, newurl); + if(cache) { + server_make_cache(db->handle, newurl); + } + _alpm_log(db->handle, ALPM_LOG_DEBUG, "adding new %sserver URL to database '%s': %s\n", + cache ? "cache " : "", db->treename, newurl); return 0; } diff --git a/lib/libalpm/db.h b/lib/libalpm/db.h index c9400365..50812e28 100644 --- a/lib/libalpm/db.h +++ b/lib/libalpm/db.h @@ -69,6 +69,7 @@ struct _alpm_db_t { char *_path; alpm_pkghash_t *pkgcache; alpm_list_t *grpcache; + alpm_list_t *cacheservers; alpm_list_t *servers; const struct db_operations *ops; diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index a64f405f..94003d2b 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -62,15 +62,16 @@ static int curl_gethost(const char *url, char *buffer, size_t buf_len); * server blacklisting */ const unsigned int server_error_limit = 3; -struct server_error_count { +struct server_opts { char server[HOSTNAME_SIZE]; + unsigned int cache; unsigned int errors; }; -static struct server_error_count *find_server_errors(alpm_handle_t *handle, const char *server) +static struct server_opts *find_server_errors(alpm_handle_t *handle, const char *server) { alpm_list_t *i; - struct server_error_count *h; + struct server_opts *h; char hostname[HOSTNAME_SIZE]; /* key off the hostname because a host may serve multiple repos under * different url's and errors are likely to be host-wide */ @@ -83,7 +84,7 @@ static struct server_error_count *find_server_errors(alpm_handle_t *handle, cons return h; } } - if((h = calloc(sizeof(struct server_error_count), 1)) + if((h = calloc(sizeof(struct server_opts), 1)) && alpm_list_append(&handle->server_errors, h)) { strcpy(h->server, hostname); h->errors = 0; @@ -94,20 +95,50 @@ static struct server_error_count *find_server_errors(alpm_handle_t *handle, cons } } +static int is_cache_server(alpm_handle_t *handle, const char *server) +{ + struct server_opts *h; + if((h = find_server_errors(handle, server))) { + return h->cache; + } + return 0; +} + +static int should_skip_cacheserver(alpm_handle_t *handle, const char *server, const char *filepath) +{ + if(!is_cache_server(handle, server)) { + return 0; + } + if(filepath) { + return strncmp(filepath + strlen(filepath) - 3, ".db", 3) == 0 ? 1 : 0; + } + return 0; +} + + static int should_skip_server(alpm_handle_t *handle, const char *server) { - struct server_error_count *h; + struct server_opts *h; if(server_error_limit && (h = find_server_errors(handle, server)) ) { return h->errors >= server_error_limit; } return 0; } +void server_make_cache(alpm_handle_t *handle, const char *server) +{ + struct server_opts *h; + if((h = find_server_errors(handle, server))) { + h->cache = 1; + } +} + static void server_increment_error(alpm_handle_t *handle, const char *server, int count) { - struct server_error_count *h; + struct server_opts *h; if(server_error_limit && (h = find_server_errors(handle, server)) + && !is_cache_server(handle, server) && !should_skip_server(handle, server) ) { h->errors += count; @@ -411,7 +442,8 @@ static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload struct stat st; alpm_handle_t *handle = payload->handle; - while(payload->servers && should_skip_server(handle, payload->servers->data)) { + while(payload->servers && (should_skip_server(handle, payload->servers->data) + || should_skip_cacheserver(handle, payload->servers->data, payload->filepath))) { payload->servers = payload->servers->next; } if(!payload->servers) { @@ -766,7 +798,9 @@ static int curl_add_payload(alpm_handle_t *handle, CURLM *curlm, ASSERT(!payload->filepath, GOTO_ERR(handle, ALPM_ERR_WRONG_ARGS, cleanup)); } else { const char *server; - while(payload->servers && should_skip_server(handle, payload->servers->data)) { + + while(payload->servers && (should_skip_server(handle, payload->servers->data) + || should_skip_cacheserver(handle, payload->servers->data, payload->filepath))) { payload->servers = payload->servers->next; } diff --git a/lib/libalpm/dload.h b/lib/libalpm/dload.h index 9438e04a..d117244d 100644 --- a/lib/libalpm/dload.h +++ b/lib/libalpm/dload.h @@ -58,6 +58,8 @@ struct dload_payload { #endif }; +void server_make_cache(alpm_handle_t *handle, const char *server); + void _alpm_dload_payload_reset(struct dload_payload *payload); int _alpm_download(alpm_handle_t *handle, diff --git a/src/pacman/conf.c b/src/pacman/conf.c index f9edf75b..04bb478b 100644 --- a/src/pacman/conf.c +++ b/src/pacman/conf.c @@ -172,6 +172,7 @@ void config_repo_free(config_repo_t *repo) return; } free(repo->name); + FREELIST(repo->cacheservers); FREELIST(repo->servers); free(repo); } @@ -781,9 +782,9 @@ static char *replace_server_vars(config_t *c, config_repo_t *r, const char *s) } } -static int _add_mirror(alpm_db_t *db, char *value) +static int _add_mirror(alpm_db_t *db, char *value, const int cache) { - if(alpm_db_add_server(db, value) != 0) { + if(alpm_db_add_server(db, value, cache) != 0) { /* pm_errno is set by alpm_db_setserver */ pm_printf(ALPM_LOG_ERROR, _("could not add server URL to database '%s': %s (%s)\n"), alpm_db_get_name(db), value, alpm_strerror(alpm_errno(config->handle))); @@ -809,8 +810,14 @@ static int register_repo(config_repo_t *repo) repo->usage, repo->name); alpm_db_set_usage(db, repo->usage); + for(i = repo->cacheservers; i; i = alpm_list_next(i)) { + if(_add_mirror(db, i->data, 1) != 0) { + return 1; + } + } + for(i = repo->servers; i; i = alpm_list_next(i)) { - if(_add_mirror(db, i->data) != 0) { + if(_add_mirror(db, i->data, 0) != 0) { return 1; } } @@ -993,6 +1000,9 @@ static int _parse_repo(const char *key, char *value, const char *file, if(strcmp(key, "Server") == 0) { CHECK_VALUE(value); repo->servers = alpm_list_add(repo->servers, strdup(value)); + } else if(strcmp(key, "CacheServer") == 0) { + CHECK_VALUE(value); + repo->cacheservers = alpm_list_add(repo->cacheservers, strdup(value)); } else if(strcmp(key, "SigLevel") == 0) { CHECK_VALUE(value); alpm_list_t *values = NULL; diff --git a/src/pacman/conf.h b/src/pacman/conf.h index f7916ca9..f242f522 100644 --- a/src/pacman/conf.h +++ b/src/pacman/conf.h @@ -37,6 +37,7 @@ typedef struct __colstr_t { typedef struct __config_repo_t { char *name; + alpm_list_t *cacheservers; alpm_list_t *servers; int usage; int siglevel; diff --git a/src/pacman/pacman-conf.c b/src/pacman/pacman-conf.c index a9d1f52b..db648d4c 100644 --- a/src/pacman/pacman-conf.c +++ b/src/pacman/pacman-conf.c @@ -236,6 +236,7 @@ static void dump_repo(config_repo_t *repo) { show_usage("Usage", repo->usage); show_siglevel("SigLevel", repo->siglevel, 0); + show_list_str("CacheServer", repo->cacheservers); show_list_str("Server", repo->servers); }