| Message ID | 20210909084122.1020752-1-archlinux@cy.md |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [pacman-dev] libalpm: Log URLs when retrying | expand |
On 09/09/2021 09:41, Vladimir Panteleev via pacman-dev wrote: > Allow finding which mirror was used to fetch a file. > > This makes it a bit easier to debug situations in which mirrors serve > bad files with HTTP 200. > > Signed-off-by: Vladimir Panteleev <archlinux@cy.md> > --- > lib/libalpm/dload.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c > index 4322318b..022bc7fd 100644 > --- a/lib/libalpm/dload.c > +++ b/lib/libalpm/dload.c > @@ -427,6 +427,9 @@ static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload > len = strlen(server) + strlen(payload->filepath) + 2; > MALLOC(payload->fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); > snprintf(payload->fileurl, len, "%s/%s", server, payload->filepath); > + _alpm_log(handle, ALPM_LOG_DEBUG, > + "%s: retrying from %s\n", > + payload->remote_name, payload->fileurl); > > > fflush(payload->localf); > Doesn't pacman already do this though? :: Retrieving packages... glibc-2.33-5-x86_64 downloading... debug: glibc-2.33-5-x86_64.pkg.tar.zst: url is http://test.com/glibc-2.33-5-x86_64.pkg.tar.zst debug: glibc-2.33-5-x86_64.pkg.tar.zst: maxsize 10303557 debug: glibc-2.33-5-x86_64.pkg.tar.zst: opened tempfile for download: /var/cache/pacman/pkg/glibc-2.33-5-x86_64.pkg.tar.zst.part (wb) debug: glibc-2.33-5-x86_64.pkg.tar.zst: curl returned result 28 from transfer error: failed retrieving file 'glibc-2.33-5-x86_64.pkg.tar.zst' from test.com : Connection timeout after 10001 ms debug: glibc-2.33-5-x86_64.pkg.tar.zst: tempfile found, attempting continuation from 0 bytes debug: glibc-2.33-5-x86_64.pkg.tar.zst: curl returned result 0 from transfer debug: glibc-2.33-5-x86_64.pkg.tar.zst: response code 200 debug: glibc-2.33-5-x86_64.pkg.tar.zst.sig: url is http://mirror.bytemark.co.uk/archlinux/core/os/x86_64/glibc-2.33-5-x86_64.pkg.tar.zst.sig debug: glibc-2.33-5-x86_64.pkg.tar.zst.sig: maxsize 16384 debug: glibc-2.33-5-x86_64.pkg.tar.zst.sig: opened tempfile for download: /var/cache/pacman/pkg/glibc-2.33-5-x86_64.pkg.tar.zst.sig.part (wb) debug: glibc-2.33-5-x86_64.pkg.tar.zst.sig: curl returned result 0 from transfer debug: glibc-2.33-5-x86_64.pkg.tar.zst.sig: response code 200
On 03/10/2021 13:42, Vladimir Panteleev wrote: > On 27/09/2021 20.40, morganamilo at archlinux.org (Morgan Adamiec) wrote: >> On 09/09/2021 09:41, Vladimir Panteleev via pacman-dev wrote: >>> Allow finding which mirror was used to fetch a file. >>> >>> This makes it a bit easier to debug situations in which mirrors serve >>> bad files with HTTP 200. >>> >>> Signed-off-by: Vladimir Panteleev <archlinux at cy.md> >>> --- >>> lib/libalpm/dload.c | 3 +++ >>> 1 file changed, 3 insertions(+) >>> >>> diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c >>> index 4322318b..022bc7fd 100644 >>> --- a/lib/libalpm/dload.c >>> +++ b/lib/libalpm/dload.c >>> @@ -427,6 +427,9 @@ static int curl_retry_next_server(CURLM *curlm, >>> CURL *curl, struct dload_payload >>> len = strlen(server) + strlen(payload->filepath) + 2; >>> MALLOC(payload->fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, >>> -1)); >>> snprintf(payload->fileurl, len, "%s/%s", server, >>> payload->filepath); >>> + _alpm_log(handle, ALPM_LOG_DEBUG, >>> + "%s: retrying from %s\n", >>> + payload->remote_name, payload->fileurl); >>> fflush(payload->localf); >>> >> >> Doesn't pacman already do this though? > > Not when retrying due to e.g. HTTP 404. > > BTW, I missed your message because (AFAICS) I was not CC'd - I think > "reply to all" makes more sense on a mailing list. (Alternatively > looking forward to contributions being acceptable via GitLab.) > > Thanks. > Still I seems to work with 404's too :: Retrieving packages... filesystem-2018.1-2-x86_64 downloading... --- debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: url is http://mirror.bytemark.co.uk/archlinux/core/os/x86_64/filesystem-2018.1-2-x86_64.pkg.tar.xz --- debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: maxsize 7972 debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: opened tempfile for download: /var/cache/pacman/pkg/filesystem-2018.1-2-x86_64.pkg.tar.xz.part (wb) debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: curl returned result 0 from transfer debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: response code 404 error: failed retrieving file 'filesystem-2018.1-2-x86_64.pkg.tar.xz' from mirror.bytemark.co.uk : The requested URL returned error: 404 debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: tempfile found, attempting continuation from 0 bytes debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: curl returned result 0 from transfer debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: response code 200 --- debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: url is https://ia601006.us.archive.org/6/items/archlinux_pkg_filesystem/filesystem-2018.1-2-x86_64.pkg.tar.xz.sig --- debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: maxsize 16384 debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: opened tempfile for download: /var/cache/pacman/pkg/filesystem-2018.1-2-x86_64.pkg.tar.xz.sig.part (wb) debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: curl returned result 0 from transfer debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: response code 200 debug: curl_download_internal return code is 0 checking keyring...
On Sun, 3 Oct 2021 at 23:22, Morgan Adamiec <morganamilo@archlinux.org> wrote: > Still I seems to work with 404's too > > :: Retrieving packages... > filesystem-2018.1-2-x86_64 downloading... > --- > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: url is > http://mirror.bytemark.co.uk/archlinux/core/os/x86_64/filesystem-2018.1-2-x86_64.pkg.tar.xz > --- > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: maxsize 7972 > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: opened tempfile for > download: > /var/cache/pacman/pkg/filesystem-2018.1-2-x86_64.pkg.tar.xz.part (wb) > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: curl returned result 0 > from transfer > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: response code 404 > error: failed retrieving file 'filesystem-2018.1-2-x86_64.pkg.tar.xz' > from mirror.bytemark.co.uk : The requested URL returned error: 404 > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: tempfile found, attempting > continuation from 0 bytes > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: curl returned result 0 > from transfer > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: response code 200 > --- > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: url is > https://ia601006.us.archive.org/6/items/archlinux_pkg_filesystem/filesystem-2018.1-2-x86_64.pkg.tar.xz.sig > --- > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: maxsize 16384 > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: opened tempfile for > download: > /var/cache/pacman/pkg/filesystem-2018.1-2-x86_64.pkg.tar.xz.sig.part (wb) > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: curl returned result 0 > from transfer > debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: response code 200 > debug: curl_download_internal return code is 0 > checking keyring... So, in your log, what URL did pacman download filesystem-2018.1-2-x86_64.pkg.tar.xz from? We could make an educated guess that it is from https://ia601006.us.archive.org/6/items/archlinux_pkg_filesystem/filesystem-2018.1-2-x86_64.pkg.tar.xz , because that's where it downloaded the signature from. This assumes that pacman resumes downloading the signature from the mirror which last successfully served a package file. However, that is never explicitly stated in the log, and is what this patch aims to address. Hope this helps. :)
On 04/10/2021 08:10, Vladimir Panteleev wrote: > On Sun, 3 Oct 2021 at 23:22, Morgan Adamiec <morganamilo@archlinux.org> wrote: >> Still I seems to work with 404's too >> >> :: Retrieving packages... >> filesystem-2018.1-2-x86_64 downloading... >> --- >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: url is >> http://mirror.bytemark.co.uk/archlinux/core/os/x86_64/filesystem-2018.1-2-x86_64.pkg.tar.xz >> --- >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: maxsize 7972 >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: opened tempfile for >> download: >> /var/cache/pacman/pkg/filesystem-2018.1-2-x86_64.pkg.tar.xz.part (wb) >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: curl returned result 0 >> from transfer >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: response code 404 >> error: failed retrieving file 'filesystem-2018.1-2-x86_64.pkg.tar.xz' >> from mirror.bytemark.co.uk : The requested URL returned error: 404 >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: tempfile found, attempting >> continuation from 0 bytes >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: curl returned result 0 >> from transfer >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz: response code 200 >> --- >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: url is >> https://ia601006.us.archive.org/6/items/archlinux_pkg_filesystem/filesystem-2018.1-2-x86_64.pkg.tar.xz.sig >> --- >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: maxsize 16384 >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: opened tempfile for >> download: >> /var/cache/pacman/pkg/filesystem-2018.1-2-x86_64.pkg.tar.xz.sig.part (wb) >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: curl returned result 0 >> from transfer >> debug: filesystem-2018.1-2-x86_64.pkg.tar.xz.sig: response code 200 >> debug: curl_download_internal return code is 0 >> checking keyring... > > > So, in your log, what URL did pacman download > filesystem-2018.1-2-x86_64.pkg.tar.xz from? > > We could make an educated guess that it is from > https://ia601006.us.archive.org/6/items/archlinux_pkg_filesystem/filesystem-2018.1-2-x86_64.pkg.tar.xz > , because that's where it downloaded the signature from. This assumes > that pacman resumes downloading the signature from the mirror which > last successfully served a package file. However, that is never > explicitly stated in the log, and is what this patch aims to address. > Hope this helps. :) > You know, I didn't see the.sig on the end there. In that cause this may make sense.
diff --git a/lib/libalpm/dload.c b/lib/libalpm/dload.c index 4322318b..022bc7fd 100644 --- a/lib/libalpm/dload.c +++ b/lib/libalpm/dload.c @@ -427,6 +427,9 @@ static int curl_retry_next_server(CURLM *curlm, CURL *curl, struct dload_payload len = strlen(server) + strlen(payload->filepath) + 2; MALLOC(payload->fileurl, len, RET_ERR(handle, ALPM_ERR_MEMORY, -1)); snprintf(payload->fileurl, len, "%s/%s", server, payload->filepath); + _alpm_log(handle, ALPM_LOG_DEBUG, + "%s: retrying from %s\n", + payload->remote_name, payload->fileurl); fflush(payload->localf);
Allow finding which mirror was used to fetch a file. This makes it a bit easier to debug situations in which mirrors serve bad files with HTTP 200. Signed-off-by: Vladimir Panteleev <archlinux@cy.md> --- lib/libalpm/dload.c | 3 +++ 1 file changed, 3 insertions(+)