| Message ID | 20210521080653.25245-1-list@eworm.de |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | [pacman-dev,1/1] support http header to indicate an expected failure | expand |
Christian Hesse <list@eworm.de> on Fri, 2021/05/21 10:06: > By setting an extra HTTP header 'X-Pacman-Expected-Failure' the server can > indicate that the failure is expected. The next server is tried without > error message and without increasing the server's error count. > > This can be used by servers that are not expected to be complete, for > example when serving a local cache. Any comment on this? Where to document the behavior? BTW, this is the matching branch for pacredir: https://github.com/eworm-de/pacredir/commits/404-header
On 05/24/21 at 08:31pm, Christian Hesse wrote: > Christian Hesse <list@eworm.de> on Fri, 2021/05/21 10:06: > > By setting an extra HTTP header 'X-Pacman-Expected-Failure' the server can > > indicate that the failure is expected. The next server is tried without > > error message and without increasing the server's error count. > > > > This can be used by servers that are not expected to be complete, for > > example when serving a local cache. > > Any comment on this? > Where to document the behavior? > > BTW, this is the matching branch for pacredir: > https://github.com/eworm-de/pacredir/commits/404-header I'm not a fan of having this be server-side. I'd rather see https://bugs.archlinux.org/task/23407 implemented.
Andrew Gregory <andrew.gregory.8@gmail.com> on Mon, 2021/05/24 11:50: > On 05/24/21 at 08:31pm, Christian Hesse wrote: > > Christian Hesse <list@eworm.de> on Fri, 2021/05/21 10:06: > > > By setting an extra HTTP header 'X-Pacman-Expected-Failure' the server > > > can indicate that the failure is expected. The next server is tried > > > without error message and without increasing the server's error count. > > > > > > This can be used by servers that are not expected to be complete, for > > > example when serving a local cache. > > > > Any comment on this? > > Where to document the behavior? > > > > BTW, this is the matching branch for pacredir: > > https://github.com/eworm-de/pacredir/commits/404-header > > I'm not a fan of having this be server-side. Would be a perfect fit for me and my use case. :-p For those interested... I created simple flow charts to describe what happens when pacman sends its requests to pacredir. https://git.eworm.de/cgit/pacredir/about/FLOW.md In the seconds chart pacredir returns the http code 404 to pacman just before the mirror is contacted. That is where the extra header should be included. > I'd rather see > https://bugs.archlinux.org/task/23407 implemented. Could work as well. But note that pacredir does handle database files as well. So limiting 'CacheServer' to package files would restrict pacredir's functionality.
On 26/5/21 6:22 pm, Christian Hesse wrote: > Andrew Gregory <andrew.gregory.8@gmail.com> on Mon, 2021/05/24 11:50: >> On 05/24/21 at 08:31pm, Christian Hesse wrote: >>> Christian Hesse <list@eworm.de> on Fri, 2021/05/21 10:06: >>>> By setting an extra HTTP header 'X-Pacman-Expected-Failure' the server >>>> can indicate that the failure is expected. The next server is tried >>>> without error message and without increasing the server's error count. >>>> >>>> This can be used by servers that are not expected to be complete, for >>>> example when serving a local cache. >>> >>> Any comment on this? >>> Where to document the behavior? >>> >>> BTW, this is the matching branch for pacredir: >>> https://github.com/eworm-de/pacredir/commits/404-header >>de/pacredir/commits/404-header >> >> I'm not a fan of having this be server-side. > > Would be a perfect fit for me and my use case. :-p > > For those interested... I created simple flow charts to describe what happens > when pacman sends its requests to pacredir. > https://git.eworm.de/cgit/pacredir/about/FLOW.md > Thanks - that helps a bit. I'm still confused about how pacredir updates databases from a mirror. I can see how it gets a database update from another host. Can you clarify? Tbanks, Allan
Allan McRae <allan@archlinux.org> on Fri, 2021/05/28 12:39: > On 26/5/21 6:22 pm, Christian Hesse wrote: > > Andrew Gregory <andrew.gregory.8@gmail.com> on Mon, 2021/05/24 11:50: > >> On 05/24/21 at 08:31pm, Christian Hesse wrote: > >>> Christian Hesse <list@eworm.de> on Fri, 2021/05/21 10:06: > >>>> By setting an extra HTTP header 'X-Pacman-Expected-Failure' the server > >>>> can indicate that the failure is expected. The next server is tried > >>>> without error message and without increasing the server's error count. > >>>> > >>>> This can be used by servers that are not expected to be complete, for > >>>> example when serving a local cache. > >>> > >>> Any comment on this? > >>> Where to document the behavior? > >>> > >>> BTW, this is the matching branch for pacredir: > >>> https://github.com/eworm-de/pacredir/commits/404-header > >>de/pacredir/commits/404-header > >> > >> I'm not a fan of having this be server-side. > > > > Would be a perfect fit for me and my use case. :-p > > > > For those interested... I created simple flow charts to describe what > > happens when pacman sends its requests to pacredir. > > https://git.eworm.de/cgit/pacredir/about/FLOW.md > > > > Thanks - that helps a bit. > > I'm still confused about how pacredir updates databases from a mirror. > I can see how it gets a database update from another host. Can you clarify? Ok, let's see a more detailed example: * pacman sends a request to pacredir, the header contains a timestamp from synced database file in /var/lib/pacman/sync/: If-Modified-Since: Fri, 28 May 2021 04:38:25 GMT * pacredir sends HEAD requests to pacserve on hosts in local network * pacserve answers with 200, but the header contains a timestamp: Last-Modified: Fri, 28 May 2021 04:38:25 GMT * pacredir sends a 307 with redirect to the host with most recent db file - or 404 if all requested db files are older or of same age as local file
On 28/5/21 7:07 pm, Christian Hesse wrote: > Allan McRae <allan@archlinux.org> on Fri, 2021/05/28 12:39: >> On 26/5/21 6:22 pm, Christian Hesse wrote: >>> Andrew Gregory <andrew.gregory.8@gmail.com> on Mon, 2021/05/24 11:50: >>>> On 05/24/21 at 08:31pm, Christian Hesse wrote: >>>>> Christian Hesse <list@eworm.de> on Fri, 2021/05/21 10:06: >>>>>> By setting an extra HTTP header 'X-Pacman-Expected-Failure' the server >>>>>> can indicate that the failure is expected. The next server is tried >>>>>> without error message and without increasing the server's error count. >>>>>> >>>>>> This can be used by servers that are not expected to be complete, for >>>>>> example when serving a local cache. >>>>> >>>>> Any comment on this? >>>>> Where to document the behavior? >>>>> >>>>> BTW, this is the matching branch for pacredir: >>>>> https://github.com/eworm-de/pacredir/commits/404-header >>>> de/pacredir/commits/404-header >>>> >>>> I'm not a fan of having this be server-side. >>> >>> Would be a perfect fit for me and my use case. :-p >>> >>> For those interested... I created simple flow charts to describe what >>> happens when pacman sends its requests to pacredir. >>> https://git.eworm.de/cgit/pacredir/about/FLOW.md >>> >> >> Thanks - that helps a bit. >> >> I'm still confused about how pacredir updates databases from a mirror. >> I can see how it gets a database update from another host. Can you clarify? > > Ok, let's see a more detailed example: > > * pacman sends a request to pacredir, the header contains a timestamp from > synced database file in /var/lib/pacman/sync/: > If-Modified-Since: Fri, 28 May 2021 04:38:25 GMT > * pacredir sends HEAD requests to pacserve on hosts in local network > * pacserve answers with 200, but the header contains a timestamp: > Last-Modified: Fri, 28 May 2021 04:38:25 GMT > * pacredir sends a 307 with redirect to the host with most recent db file - or > 404 if all requested db files are older or of same age as local file > OK... so pacredir never checks for the latest database availability on a mirror if there is a newer database on one of the hosts. Does this mean the "new" database on the hosts network could be long out of date, but as long as it is newer than the local machine being updated, that is what will be used? The reason I am trying to figure this out is the approach in the bug report only provides cache servers for packages, and not databases. I'm trying to understand why you would cache databases. Allan
Allan McRae <allan@archlinux.org> on Fri, 2021/05/28 20:43: > > Ok, let's see a more detailed example: > > > > * pacman sends a request to pacredir, the header contains a timestamp from > > synced database file in /var/lib/pacman/sync/: > > If-Modified-Since: Fri, 28 May 2021 04:38:25 GMT > > * pacredir sends HEAD requests to pacserve on hosts in local network > > * pacserve answers with 200, but the header contains a timestamp: > > Last-Modified: Fri, 28 May 2021 04:38:25 GMT > > * pacredir sends a 307 with redirect to the host with most recent db file > > - or 404 if all requested db files are older or of same age as local file > > OK... so pacredir never checks for the latest database availability on > a mirror if there is a newer database on one of the hosts. Please note that pacredir does not known about mirrors at all! It tries to redirect in local network - and returns 404 if it can not. It's pacman that skips to the mirrors. That's why pacman should not consider a 404 a permanent error: The next request can result in a successful redirect. > Does this > mean the "new" database on the hosts network could be long out of date, > but as long as it is newer than the local machine being updated, that is > what will be used? Well, out-of-date is a term that does barely match here... pacman does known about the date of its current database files only. So yes, more recent database files are used as long as they are newer than the local ones - even if out-of-date compared with a mirror. That's why the pacredir documentation tells you to run `pacman -Sy` twice to be sure: First run fetches the newest database from local network, second run (where pacredir returns 404) fetches from mirror if a newer version is available. > The reason I am trying to figure this out is the approach in the bug > report only provides cache servers for packages, and not databases. I'm > trying to understand why you would cache databases. Because database file are available on every host anyway. :) Usually it does not add any harm to transfer a database file in local network, even if you have to retransfer an newer one from mirror later. Chance are you do not have to.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Fri, 28 May 2021, Christian Hesse wrote: > Well, out-of-date is a term that does barely match here... pacman does known > about the date of its current database files only. So yes, more recent > database files are used as long as they are newer than the local ones - even > if out-of-date compared with a mirror. > > That's why the pacredir documentation tells you to run `pacman -Sy` twice to > be sure: First run fetches the newest database from local network, second run > (where pacredir returns 404) fetches from mirror if a newer version is > available. Just some side-idea: why not consider all databases which are older than x minutes (where x is configurable) "out of date"? This would rely on having systemtimes in sync, but otherwise would get rid of the cumbersome double-`pacman -Syu`. regards, Erich P.S.: I'm running something similar as you - but by far not as elaborate: two (for redundancy) nginx servers which cache packages and download databases directly. -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE3p92iMrPBP64GmxZCu7JB1Xae1oFAmCw4XYACgkQCu7JB1Xa e1oxlA/9Gw07gaaSs14YXaScYg6bxUZtPLxU308uXzBXnPGWxbPaHdyMTzoKbUcK WK/khLbrU8akJtzcvhS6JUwgWGLhgZ9rKign2oqsHd/Vehzq0AMs4t6E0uBo+t4q IXsNfWE/R7X0KIBfrvSvZBrgZRbAqZ3C2Om0YbtxJIyQHZYzb+zxMix9GH0TvJdU hvZU/nSedSl3tL/5TloFTU7giMmUr4endXwCQkZemGvBtav45CwZNLlTy121WcRI F5qQfr8nAIj46zn4xH3xBVZa5aPBDg4S5Xp0Ey0rPlBWEDqx5rDTdo268hIMX7Fy Wq5ki7+Uaf+AVz7LD95dIExfgwajqUPpO6dctb91IPAAuxTM5afLcff84U3KtprC ugHqArk76m65HKW/ETOQxLGw+2/aexXXGFNoueh3IEXMShyX7reNNlCAA2rTRkuP 6RHXOS/BODNaJCuHzo3g2V3xcTwDztmkGJK6oVEyOZJczeBd3CwW82d0AqOY5+KF kUNvNGHa//QyQsjCRTFcp2Az5trPRmnYI+/uYyGbpll19nW2jnXZuaQbNMwg+GG9 GtA9qBdCuubMdkcW+zIIYv9AHbyiYjxO7b4eVsLSJAgluPaEzc1GVP+lTqmQP/v7 HpMTGB/53rxDHIhnPQJSZMd68RG8eSrsppDHoh+M9dJ2mdberZs= =7LPG -----END PGP SIGNATURE-----
On 28/5/21 10:18 pm, Christian Hesse wrote: > Allan McRae <allan@archlinux.org> on Fri, 2021/05/28 20:43: >>> Ok, let's see a more detailed example: >>> >>> * pacman sends a request to pacredir, the header contains a timestamp from >>> synced database file in /var/lib/pacman/sync/: >>> If-Modified-Since: Fri, 28 May 2021 04:38:25 GMT >>> * pacredir sends HEAD requests to pacserve on hosts in local network >>> * pacserve answers with 200, but the header contains a timestamp: >>> Last-Modified: Fri, 28 May 2021 04:38:25 GMT >>> * pacredir sends a 307 with redirect to the host with most recent db file >>> - or 404 if all requested db files are older or of same age as local file >> >> OK... so pacredir never checks for the latest database availability on >> a mirror if there is a newer database on one of the hosts. > > Please note that pacredir does not known about mirrors at all! > It tries to redirect in local network - and returns 404 if it can not. It's > pacman that skips to the mirrors. > I was using "mirror" to distinguish external servers from hosts on the local network. > That's why pacman should not consider a 404 a permanent error: The next > request can result in a successful redirect. > >> Does this >> mean the "new" database on the hosts network could be long out of date, >> but as long as it is newer than the local machine being updated, that is >> what will be used? > > Well, out-of-date is a term that does barely match here... pacman does known > about the date of its current database files only. So yes, more recent > database files are used as long as they are newer than the local ones - even > if out-of-date compared with a mirror. > > That's why the pacredir documentation tells you to run `pacman -Sy` twice to > be sure: First run fetches the newest database from local network, second run > (where pacredir returns 404) fetches from mirror if a newer version is > available. This does very little to convince me that the CacheServer proposal should be used for databases. A
Erich Eckner <arch@eckner.net> on Fri, 2021/05/28 14:26: > > Well, out-of-date is a term that does barely match here... pacman does > > known about the date of its current database files only. So yes, more > > recent database files are used as long as they are newer than the local > > ones - even if out-of-date compared with a mirror. > > > > That's why the pacredir documentation tells you to run `pacman -Sy` twice > > to be sure: First run fetches the newest database from local network, > > second run (where pacredir returns 404) fetches from mirror if a newer > > version is available. > > Just some side-idea: why not consider all databases which are older than x > minutes (where x is configurable) "out of date"? This would rely on having > systemtimes in sync, but otherwise would get rid of the cumbersome > double-`pacman -Syu`. Great idea... About to add that. :) > P.S.: I'm running something similar as you - but by far not as elaborate: > two (for redundancy) nginx servers which cache packages and download > databases directly. Try pacredir for an out-of-the-box solution. ;) BTW, if my solution with server side headers is accepted... In nginx configuration your location block could look something like this: location / { root /var/cache/pacman/pkg/ add_header X-Pacman-Expected-Failure true; }
Allan McRae <allan@archlinux.org> on Fri, 2021/05/28 22:52: > >> Does this > >> mean the "new" database on the hosts network could be long out of date, > >> but as long as it is newer than the local machine being updated, that is > >> what will be used? > > > > Well, out-of-date is a term that does barely match here... pacman does > > known about the date of its current database files only. So yes, more > > recent database files are used as long as they are newer than the local > > ones - even if out-of-date compared with a mirror. > > > > That's why the pacredir documentation tells you to run `pacman -Sy` twice > > to be sure: First run fetches the newest database from local network, > > second run (where pacredir returns 404) fetches from mirror if a newer > > version is available. > > This does very little to convince me that the CacheServer proposal > should be used for databases. So let's implement CacheServer for package files only as described in FS#23407. Apply my patch anyway to make pacman happy with pacredir. :)
Hi On Fri, May 28, 2021, 05:26 Erich Eckner <arch@eckner.net> wrote: > P.S.: I'm running something similar as you - but by far not as elaborate: > two (for redundancy) nginx servers which cache packages and download > databases directly. > FYI there are plenty of package caching solutions exist. See https://wiki.archlinux.org/title/Pacman/Tips_and_tricks#Network_shared_pacman_cache
On 05/28/21 at 05:55pm, Christian Hesse wrote: > Allan McRae <allan@archlinux.org> on Fri, 2021/05/28 22:52: > > >> Does this > > >> mean the "new" database on the hosts network could be long out of date, > > >> but as long as it is newer than the local machine being updated, that is > > >> what will be used? > > > > > > Well, out-of-date is a term that does barely match here... pacman does > > > known about the date of its current database files only. So yes, more > > > recent database files are used as long as they are newer than the local > > > ones - even if out-of-date compared with a mirror. > > > > > > That's why the pacredir documentation tells you to run `pacman -Sy` twice > > > to be sure: First run fetches the newest database from local network, > > > second run (where pacredir returns 404) fetches from mirror if a newer > > > version is available. > > > > This does very little to convince me that the CacheServer proposal > > should be used for databases. > > So let's implement CacheServer for package files only as described > in FS#23407. > > Apply my patch anyway to make pacman happy with pacredir. :) -1. This sounds like exactly the kind of esoteric setup that we keep the external downloader around for.
Christian Hesse <list@eworm.de> on Fri, 2021/05/28 17:55: > Allan McRae <allan@archlinux.org> on Fri, 2021/05/28 22:52: > > >> Does this > > >> mean the "new" database on the hosts network could be long out of date, > > >> but as long as it is newer than the local machine being updated, that > > >> is what will be used? > > > > > > Well, out-of-date is a term that does barely match here... pacman does > > > known about the date of its current database files only. So yes, more > > > recent database files are used as long as they are newer than the local > > > ones - even if out-of-date compared with a mirror. > > > > > > That's why the pacredir documentation tells you to run `pacman -Sy` > > > twice to be sure: First run fetches the newest database from local > > > network, second run (where pacredir returns 404) fetches from mirror if > > > a newer version is available. > > > > This does very little to convince me that the CacheServer proposal > > should be used for databases. > > So let's implement CacheServer for package files only as described > in FS#23407. > > Apply my patch anyway to make pacman happy with pacredir. :) Now that pacman 6.0.0 is in [core] we should have an idea of a solution at least. Given that a number of caching solutions exist [0] having the header solution around may be not the worst idea. So does this have a chance to be committed? Still the CacheServer directive for package files would not work with database files. Thus it would not be a real solution for use with pacredir. So would you accept a patch to disable the server error limit from configuration and command line switch? That would kind of resolve the issue but result in error messages still being thrown for me, however... Had hoped to get rid of that. [0] https://wiki.archlinux.org/title/Pacman/Tips_and_tricks#Network_shared_pacman_cache
On 2/6/21 12:28 am, Christian Hesse wrote: > Christian Hesse <list@eworm.de> on Fri, 2021/05/28 17:55: >> Allan McRae <allan@archlinux.org> on Fri, 2021/05/28 22:52: >>>>> Does this >>>>> mean the "new" database on the hosts network could be long out of date, >>>>> but as long as it is newer than the local machine being updated, that >>>>> is what will be used? >>>> >>>> Well, out-of-date is a term that does barely match here... pacman does >>>> known about the date of its current database files only. So yes, more >>>> recent database files are used as long as they are newer than the local >>>> ones - even if out-of-date compared with a mirror. >>>> >>>> That's why the pacredir documentation tells you to run `pacman -Sy` >>>> twice to be sure: First run fetches the newest database from local >>>> network, second run (where pacredir returns 404) fetches from mirror if >>>> a newer version is available. >>> >>> This does very little to convince me that the CacheServer proposal >>> should be used for databases. >> >> So let's implement CacheServer for package files only as described >> in FS#23407. >> >> Apply my patch anyway to make pacman happy with pacredir. :) > > Now that pacman 6.0.0 is in [core] we should have an idea of a solution at > least. Given that a number of caching solutions exist [0] having the header > solution around may be not the worst idea. So does this have a chance to be > committed? > Andrew already gave it a -1. That means there is zero chance of it being committed. > Still the CacheServer directive for package files would not work with database > files. Thus it would not be a real solution for use with pacredir. > So would you accept a patch to disable the server error limit from > configuration and command line switch? That would kind of resolve the > issue but result in error messages still being thrown for me, however... > Had hoped to get rid of that. Going through other caching solutions, CacheServer ignoring missing packages would solve most issues. So I'll implement that when I have a spare 30 minutes. Once that is done pacredir can probably do "pacman -Sy; pacman -Sy; pacman -Su" as a workaround until we handle dbs too. Allan
diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index f6a4f012..e6e0a6f5 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -277,6 +277,7 @@ static size_t dload_parseheader_cb(void *ptr, size_t size, size_t nmemb, void *u const char *fptr, *endptr = NULL; const char * const cd_header = "Content-Disposition:"; const char * const fn_key = "filename="; + const char * const xpef_header = "X-Pacman-Expected-Failure:"; struct dload_payload *payload = (struct dload_payload *)user; long respcode; @@ -300,6 +301,13 @@ static size_t dload_parseheader_cb(void *ptr, size_t size, size_t nmemb, void *u } } + /* By setting an extra HTTP header 'X-Pacman-Expected-Failure' the server can + indicate that the failure is expected. The next server is tried without + error message and without increasing the server's error count. */ + if(_alpm_raw_ncmp(xpef_header, ptr, strlen(xpef_header)) == 0) { + payload->errors_ok = 1; + } + curl_easy_getinfo(payload->curl, CURLINFO_RESPONSE_CODE, &respcode); if(payload->respcode != respcode) { payload->respcode = respcode;