From patchwork Tue Jan 7 20:43:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emil Velikov via arch-projects X-Patchwork-Id: 1437 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id B679C16169380 for ; Tue, 7 Jan 2020 20:44:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.3 (2019-12-06) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1,DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1, MAILING_LIST_MULTI=-1,RCVD_IN_DNSWL_MED=-2.3,SPF_HELO_NONE=0.001, T_DMARC_POLICY_NONE=0.01 autolearn=ham autolearn_force=no version=3.4.3 X-Spam-BL-Results: [127.0.9.2] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Tue, 7 Jan 2020 20:44:06 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id 9E28817ED8C8BC; Tue, 7 Jan 2020 20:44:04 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [5.9.250.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) (Authenticated sender: luna) by orion.archlinux.org (Postfix) with ESMTPSA id 40C7A17ED8C8B5; Tue, 7 Jan 2020 20:44:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=orion; t=1578429844; bh=I+8koa5LrN9abEQ/fGkRhVXC1M3Xv85vO8heUrriDL4=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc; b=qhJpBhApD5Z0VqCMhxgGUAPQ5eJaX2UX+rf2fddxaSARGKfEAgn9rsSQS+MM9Wv/x ZFSx0U/wpBsKJD1cPSxQdnwMLsbWTC6Q5ndPTgDysfx3t8Jgs0dYNnVBLmyUBdOTlg k8XOWbsw4Q67gAdVyQF5PD1l8cL4ws2ytWMQqrqlDCys5lWPcbLRRbKgfMkOjTg6sA kUGibLvjsHBUF/JW9pqihX195IsziHndMH2nz/yjvkisp6LAp60eUOkGKyAlMA1NLS /hHTdoquU3mPjjIzkQv6XxifyVXdMaw2dCKdGUThUdhhcFemeyAlrvic40n8H4MO0o tqejzCTAqFadC64jbFCgqX1jkdjanSiGeDDj1UsuKambCDHdxkBfAa0Vz7qI2HOrAY ZfUh71lUQlVM5oe2Rqnx4MKfpPGW9vsxGHUQyQ6wdu/ScQQKTzTm5tyrz/aJpOUD7I P1OAsrvqLXRKgKguptHupAmdjJctiKJh6m2qMrZB/Vyy/VgQCD5ifBS4gZrNsrdVTv OjdSvmBg4fHdGvmitd6GgwpNDPYn3k1o0SPeKyQPob/giiV51Zz4J9bvRaC3tLlows CuCZXbgRKggVJi1Pj+HAf4jZ67iy4tOUj5eRlWxkXseRj499PLdtiXQwHuVoeyvI1p X0m7a8EekXq8BhI4AMqU/S1Q= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 2F27F2BE5C; Tue, 7 Jan 2020 20:44:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=archlinux.org; s=luna2; t=1578429844; bh=I+8koa5LrN9abEQ/fGkRhVXC1M3Xv85vO8heUrriDL4=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc; b=Wfw/O47Yi8X1tFMTvyCNeTyNg7xRnQ2Is4wPnUTYwHYZEjAlo7YmovH0dlEzPjIn5 npgecbVAH/pb7YM8UuiilKQe2JPjT698cZSGK+mT976ZSkmckmmmCZZXvnjiYG9Q3I hxXr5mD1xyK8m4dnwIV1cT6hmt/4VMA5Uo97mOSd7cEAmGPcS+0Ekc5CmIk0nD1il5 bQfii3GAJPA908NF03YqWaMwAAet8OondV4qvldSZLGiLeypcC5bYqaZb3ro/JmsHw CGG31wciZ5l/Z6xZrgEKw+Bp3VIGbJefYQNIQ76p20k+O8noELL2QBqEYwNLPFmFSg VL3Kva/XPYxP+cgqI3biLIh8aGxj2IT+95cDdHcSmAh9RXdL75MkNvbbM61gJNaVLV +/E8Sfy8biZasMTsg4CGmsmiSFwJQucNwheQlhnI3RQAodLO5UfMqiljLAh2oLQjdd 3StDlxOYlYds5Vmn3lz0D0CHfeG5fSdiDy74fJBY18Gbf7LTNL3pVWBmpxJi5xe9mo imndzZ3GLaCkD+6UrWEbQ0HaMWta+RvkFLH48fhpj+QwLNNhRUFfHS6RW6r+IqVbYE q9koWptTBVoHKOSD1Ggtxc2rEuCFHv3GUfKsvfUR/8mrj01edJ7MHUVD1YSwULPJw3 as5iFurJ9kpWEf0KTM6xOx5Y= Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id D328A2BE5B for ; Tue, 7 Jan 2020 20:43:59 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by luna.archlinux.org (Postfix) with ESMTPS for ; Tue, 7 Jan 2020 20:43:59 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id EB7E417ED8C8A3; Tue, 7 Jan 2020 20:43:55 +0000 (UTC) Received: from localhost (jolteon.felixc.at [45.131.68.110]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: felixonmars) by orion.archlinux.org (Postfix) with ESMTPSA id 6D5E517ED8C8A2; Tue, 7 Jan 2020 20:43:55 +0000 (UTC) To: arch-projects@archlinux.org, felixonmars@archlinux.org Date: Wed, 8 Jan 2020 04:43:00 +0800 Message-Id: <20200107204300.978636-1-felixonmars@archlinux.org> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 Subject: [arch-projects] [namcap] [PATCH] Add: rule to detect Python dependencies X-BeenThere: arch-projects@archlinux.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Arch Linux projects development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Felix Yan via arch-projects Reply-To: Arch Linux projects development discussion Cc: Felix Yan Errors-To: arch-projects-bounces@archlinux.org Sender: "arch-projects" --- Namcap/rules/__init__.py | 1 + Namcap/rules/pydepends.py | 122 +++++++++++++++++++++++++ Namcap/tests/package/test_pydepends.py | 68 ++++++++++++++ 3 files changed, 191 insertions(+) create mode 100644 Namcap/rules/pydepends.py create mode 100644 Namcap/tests/package/test_pydepends.py diff --git a/Namcap/rules/__init__.py b/Namcap/rules/__init__.py index 525dbc6..01d1b96 100644 --- a/Namcap/rules/__init__.py +++ b/Namcap/rules/__init__.py @@ -43,6 +43,7 @@ from . import ( perllocal, permissions, py_mtime, + pydepends, rpath, scrollkeeper, shebangdepends, diff --git a/Namcap/rules/pydepends.py b/Namcap/rules/pydepends.py new file mode 100644 index 0000000..efc6735 --- /dev/null +++ b/Namcap/rules/pydepends.py @@ -0,0 +1,122 @@ +# -*- coding: utf-8 -*- +# +# namcap rules - pydepends +# Copyright (C) 2020 Felix Yan +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +# + +from collections import defaultdict +import ast +import sys +import sysconfig +import Namcap.package +from Namcap.ruleclass import * + + +def finddepends(liblist): + """ + Find packages owning a list of libraries + + Returns: + dependlist -- a dictionary { package => set(libraries) } + orphans -- the list of libraries without owners + """ + dependlist = defaultdict(set) + + pymatches = {} + + knownlibs = set(liblist) + foundlibs = set() + + workarounds = { + "python": sys.builtin_module_names + } + + for pkg in Namcap.package.get_installed_packages(): + for j, fsize, fmode in pkg.files: + if not j.startswith("usr/lib/python3"): + continue + + for k in knownlibs: + if j.endswith("site-packages/" + k + "/") or j.endswith("site-packages/" + k + ".py") or \ + j.endswith("site-packages/" + k + ".so") or \ + j.endswith("site-packages/" + k + sysconfig.get_config_var('EXT_SUFFIX')) or \ + j.endswith("lib-dynload/" + k + sysconfig.get_config_var('EXT_SUFFIX')) or \ + j.count("/") == 3 and j.endswith("/" + k + ".py") or \ + j.count("/") == 4 and j.endswith("/" + k + "/") or \ + pkg.name in workarounds and k in workarounds[pkg.name]: + dependlist[pkg.name].add(k) + foundlibs.add(k) + + orphans = list(knownlibs - foundlibs) + return dependlist, orphans + + +def get_imports(file): + root = ast.parse(file.read()) + + for node in ast.walk(root): + if isinstance(node, ast.Import): + for module in node.names: + yield module.name.split('.')[0] + elif isinstance(node, ast.ImportFrom): + if node.module and node.level == 0: + yield node.module.split('.')[0] + + +class PythonDependencyRule(TarballRule): + name = "pydepends" + description = "Checks python dependencies" + def analyze(self, pkginfo, tar): + liblist = defaultdict(set) + own_liblist = set() + + for entry in tar: + if not entry.isfile() or not entry.name.endswith('.py'): + continue + own_liblist.add(entry.name[:-3]) + f = tar.extractfile(entry) + for module in get_imports(f): + liblist[module].add(entry.name) + f.close() + + for lib in own_liblist: + liblist.pop(lib, None) + + dependlist, orphans = finddepends(liblist) + + # Handle "no package associated" errors + self.warnings.extend([("library-no-package-associated %s", i) + for i in orphans]) + + # Print link-level deps + for pkg, libraries in dependlist.items(): + if isinstance(libraries, set): + files = list(libraries) + needing = set().union(*[liblist[lib] for lib in libraries]) + reasons = pkginfo.detected_deps.setdefault(pkg, []) + reasons.append(( + "libraries-needed %s %s", + (str(files), str(list(needing))) + )) + self.infos.append(("link-level-dependence %s in %s", (pkg, str(files)))) + + # Check for packages in testing + for i in dependlist.keys(): + p = Namcap.package.load_testing_package(i) + q = Namcap.package.load_from_db(i) + if p is not None and q is not None and p["version"] == q["version"] : + self.warnings.append(("dependency-is-testing-release %s", i)) diff --git a/Namcap/tests/package/test_pydepends.py b/Namcap/tests/package/test_pydepends.py new file mode 100644 index 0000000..bb0dfbb --- /dev/null +++ b/Namcap/tests/package/test_pydepends.py @@ -0,0 +1,68 @@ +# -*- coding: utf-8 -*- +# +# namcap tests - pydepends +# Copyright (C) 2020 Felix Yan +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 +# USA +# + +import os +from Namcap.tests.makepkg import MakepkgTest +import Namcap.rules.pydepends + + +class PyDependsTest(MakepkgTest): + pkgbuild = """ +pkgname=__namcap_test_pydepends +pkgver=1.0 +pkgrel=1 +pkgdesc="A package" +arch=('any') +url="http://www.example.com/" +license=('GPL') +depends=('python-six') +source=() +build() { + cd "${srcdir}" + echo "import six, pyalpm" > main.py +} +package() { + install -D -m 755 "$srcdir/main.py" "$pkgdir/usr/bin/main.py" +} +""" + def test_pydepends(self): + "Package with missing pacman dependency" + pkgfile = "__namcap_test_pydepends-1.0-1-any.pkg.tar" + with open(os.path.join(self.tmpdir, "PKGBUILD"), "w") as f: + f.write(self.pkgbuild) + self.run_makepkg() + pkg, r = self.run_rule_on_tarball( + os.path.join(self.tmpdir, pkgfile), + Namcap.rules.pydepends.PythonDependencyRule + ) + self.assertEqual(pkg.detected_deps['pyalpm'], [ + ('libraries-needed %s %s', + (str(["pyalpm"]), str(["usr/bin/main.py"]))) + ] + ) + e, w, i = Namcap.depends.analyze_depends(pkg) + self.assertEqual(e, [ + ('dependency-detected-not-included %s (%s)', + ('pyalpm', "libraries ['pyalpm'] needed in files ['usr/bin/main.py']")) + ]) + self.assertEqual(w, []) + +# vim: set ts=4 sw=4 noet: