From patchwork Sun Oct 6 17:21:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Fleischer X-Patchwork-Id: 1251 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id 3EBB21441C688 for ; Sun, 6 Oct 2019 17:21:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on apollo X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1,DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1, MAILING_LIST_MULTI=-1,RCVD_IN_DNSWL_MED=-2.3,SPF_HELO_NONE=0.001, T_DMARC_POLICY_NONE=0.01 autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Sun, 6 Oct 2019 17:21:44 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 9E9BE15AB45AB8; Sun, 6 Oct 2019 17:21:41 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS; Sun, 6 Oct 2019 17:21:41 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 8B4392CADC; Sun, 6 Oct 2019 17:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1570382501; bh=FJTPtrLJB5IKPny2QKNu95GjN1415RZT7KhtoDZKgl0=; h=From:To:Subject:Date:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe; b=S3wkyEL75O+c5PYSBb5xqhAWm+NE/SEDDVX9LwoYZuvXMp2fqXSGdmZRof7heGdn0 4qkythHcUyJs8ST2wua2ULm9Pw22Vs83IjF5FGqu4RU75+8WrxFWgkH6SKXfsThZGO lWINhxsrva0okv7LYIOxMWnPQjzlsBSGaJOh8tvOGnMdRkZCkePD49Ij2s24dqIbKA WQeQqkpuAerFtDjyU/fix7xVeaXLMVQe54lTatlsw7YeXDmFwMxBCVZEl3XiDT3mk4 EsvxCy0/MMzH9P1D5faup6BOCTOGb2eDuBTndguMhquoDvIBXbqCVhgOZz9yNAJXK1 GHIZ+xe7qW2vh727x6DFn2zrIQFcAAYvtqzg4c4+8pYZaMaF8CLHwmqlacXPgd1D7t lFtJOKSsXz3kuP9iS1z7auzLcycNftqKjMfbYVkZ26yCjm35B55pq8xjICQFvwOT9M 8cBAKh9ZkrHIf1VXJKsrLDnf5133J19X6m/UhI4CqFQpcC82AeKB1OeWdcCD6podQh 5cZvaAfvEgOet2xe+t3qzQqOt+ycPcsfbPRGA3uGvf3hkgEGzkTUbF+XkebwGytXWA Lv869gnpIy6t+mTQZRue3068OGf5Dk/+n56uGVaTC//07IdjZHTQ5jax30rLRA3OsZ S7UBlfkD3nAXSFdNUJr0ZEpA= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 3E0302CAD8 for ; Sun, 6 Oct 2019 17:21:37 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Sun, 6 Oct 2019 17:21:37 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 2220815AB459D0 for ; Sun, 6 Oct 2019 17:21:36 +0000 (UTC) Received: from localhost (unknown [72.138.14.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: lfleischer) by orion.archlinux.org (Postfix) with ESMTPSA id A947115AB459CF for ; Sun, 6 Oct 2019 17:21:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=orion; t=1570382496; bh=FJTPtrLJB5IKPny2QKNu95GjN1415RZT7KhtoDZKgl0=; h=From:To:Subject:Date; b=HqNWm8HCLXF9Iq1ty2eswnsz/eTeHLUH0W60qLXIUMb8+6UPIcqBaN5dEjm7l11JH q2kKAkPEvzRuGOcnDv9uGm6N0VBb2XP11oVlqrvDggWMU7tiZH1WoLgPbKzoqpAzN8 NWpg7mppUrO+ZRBLkvPN58bHlQIFitcQfp63JEOSKNMLfP0LlGvLVb5lStrM/8QRFA UmpbBKjW3Uwk/ev6M0QAjCm6ioaY2o4NKlq6vcOqdpv+P9Y+njbHA2g38uKkwpGniB Ymay0XW/ErOQhj+Fx/7Zc72zI/q3wi4zY87fpAy44+U1bNCVG3KYPEWo2q+S+II7SD Nc7cqdm7kZTyIXBhZ5A0MvHEI1/qQGNEtubazmD4wDjvZlU/nX/2Q98FkPGZt6jDrS gtASu1aMXa7/N7U9pV83b1ESSW7jwfSloVtzxLOPFC9PluAuRWAPjXqnfceiFAYc8Z V9gaAOuGkRvdrQX4idstum0z4gwplwvXfGvRu/ugnO0rlqSzxmFaqKNrIZhuiGbqMx omRpQAoBofcXsGc0suTf9J8u+7wXVUYFD9bOWXRvhjD1m30X9eWpZRR9h3CAzWSGSF zyomCr7gUk5drynMkEUUoMmZ6+wmeOZlfPuj2jzOeFMjNEFNfl/bsQQOqDaoUuoOzz HxFb4x6emEb0g2ctfb19hSCY= From: Lukas Fleischer To: aur-dev@archlinux.org Subject: [PATCH] Cache package provider and dependency information Date: Sun, 6 Oct 2019 13:21:56 -0400 Message-Id: <20191006172156.6842-1-lfleischer@archlinux.org> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 X-BeenThere: aur-dev@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Arch User Repository \(AUR\) Development" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: aur-dev-bounces@archlinux.org Sender: "aur-dev" The package provider and dependency queries are quite CPU-intensive and usually yield rather small result sets. Cache these values if the global caching mechanism is enabled. Signed-off-by: Lukas Fleischer --- We will test this in our production environment for a while to see whether it makes any difference in server load. There are probably more things that we easily cache and there's a lot of room for improvement in general (e.g. increase the TTL and automatically invalidate the cache when a new package version is pushed via Git). web/lib/cachefuncs.inc.php | 17 ++++++++++++++++ web/lib/pkgfuncs.inc.php | 40 +++++++++++--------------------------- 2 files changed, 28 insertions(+), 29 deletions(-) diff --git a/web/lib/cachefuncs.inc.php b/web/lib/cachefuncs.inc.php index 881ad8f..d0e0f6f 100644 --- a/web/lib/cachefuncs.inc.php +++ b/web/lib/cachefuncs.inc.php @@ -79,4 +79,21 @@ function db_cache_value($dbq, $key, $ttl=600) { return $value; } +# Run a simple db query, retrieving and/or caching the result set if APC is +# available for use. Accepts an optional TTL value (defaults to 600 seconds). +function db_cache_result($dbq, $key, $ttl=600) { + $dbh = DB::connect(); + $status = false; + $value = get_cache_value($key, $status); + if (!$status) { + $result = $dbh->query($dbq); + if (!$result) { + return false; + } + $value = $result->fetchAll(PDO::FETCH_NUM); + set_cache_value($key, $value, $ttl); + } + return $value; +} + ?> diff --git a/web/lib/pkgfuncs.inc.php b/web/lib/pkgfuncs.inc.php index 00f0cc4..44e4167 100644 --- a/web/lib/pkgfuncs.inc.php +++ b/web/lib/pkgfuncs.inc.php @@ -222,17 +222,7 @@ function pkg_providers($name) { $q.= "UNION "; $q.= "SELECT 0, Name FROM OfficialProviders "; $q.= "WHERE Provides = " . $dbh->quote($name); - $result = $dbh->query($q); - - if (!$result) { - return array(); - } - - $providers = array(); - while ($row = $result->fetch(PDO::FETCH_NUM)) { - $providers[] = $row; - } - return $providers; + return db_cache_result($q, 'providers:' . $name); } /** @@ -244,26 +234,18 @@ function pkg_providers($name) { * @return array All package dependencies for the package */ function pkg_dependencies($pkgid, $limit) { - $deps = array(); $pkgid = intval($pkgid); - if ($pkgid > 0) { - $dbh = DB::connect(); - $q = "SELECT pd.DepName, dt.Name, pd.DepDesc, "; - $q.= "pd.DepCondition, pd.DepArch, p.ID "; - $q.= "FROM PackageDepends pd "; - $q.= "LEFT JOIN Packages p ON pd.DepName = p.Name "; - $q.= "LEFT JOIN DependencyTypes dt ON dt.ID = pd.DepTypeID "; - $q.= "WHERE pd.PackageID = ". $pkgid . " "; - $q.= "ORDER BY pd.DepName LIMIT " . intval($limit); - $result = $dbh->query($q); - if (!$result) { - return array(); - } - while ($row = $result->fetch(PDO::FETCH_NUM)) { - $deps[] = $row; - } + if (!$pkgid) { + return array(); } - return $deps; + $q = "SELECT pd.DepName, dt.Name, pd.DepDesc, "; + $q.= "pd.DepCondition, pd.DepArch, p.ID "; + $q.= "FROM PackageDepends pd "; + $q.= "LEFT JOIN Packages p ON pd.DepName = p.Name "; + $q.= "LEFT JOIN DependencyTypes dt ON dt.ID = pd.DepTypeID "; + $q.= "WHERE pd.PackageID = ". $pkgid . " "; + $q.= "ORDER BY pd.DepName LIMIT " . intval($limit); + return db_cache_result($q, 'dependencies:' . $name); } /**