From patchwork Mon Jun 1 16:49:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Fr=C3=A9d=C3=A9ric_Mangano-Tarumi?= X-Patchwork-Id: 1656 Return-Path: Delivered-To: patchwork@archlinux.org Received: from apollo.archlinux.org (localhost [127.0.0.1]) by apollo.archlinux.org (Postfix) with ESMTP id B7444191D54F2 for ; Mon, 1 Jun 2020 16:49:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on apollo.archlinux.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1,LOCAL_FAKEBUSINESS=0.5, MAILING_LIST_MULTI=-1,RCVD_IN_DNSWL_MED=-2.3,RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01,SPF_HELO_NONE=0.001,T_DMARC_POLICY_NONE=0.01 autolearn=ham autolearn_force=no version=3.4.4 X-Spam-BL-Results: [127.0.9.2] [127.0.0.19] Received: from orion.archlinux.org (orion.archlinux.org [88.198.91.70]) by apollo.archlinux.org (Postfix) with ESMTPS for ; Mon, 1 Jun 2020 16:49:49 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id F07FF1C7391DE5; Mon, 1 Jun 2020 16:49:44 +0000 (UTC) Received: from luna.archlinux.org (luna.archlinux.org [IPv6:2a01:4f8:160:3033::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits)) (No client certificate requested) (Authenticated sender: luna) by orion.archlinux.org (Postfix) with ESMTPSA id A74A41C7391DDF; Mon, 1 Jun 2020 16:49:44 +0000 (UTC) Authentication-Results: orion.archlinux.org; dkim=pass (1024-bit key) header.d=mg0.fr header.i=@mg0.fr header.b=Jj4okNQi Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 8394029CD0; Mon, 1 Jun 2020 16:49:44 +0000 (UTC) Authentication-Results: luna.archlinux.org; dkim=pass (1024-bit key) header.d=mg0.fr header.i=@mg0.fr header.b=Jj4okNQi Received: from luna.archlinux.org (luna.archlinux.org [127.0.0.1]) by luna.archlinux.org (Postfix) with ESMTP id 9A4B329CCB for ; Mon, 1 Jun 2020 16:49:41 +0000 (UTC) Received: from orion.archlinux.org (orion.archlinux.org [IPv6:2a01:4f8:160:6087::1]) by luna.archlinux.org (Postfix) with ESMTPS for ; Mon, 1 Jun 2020 16:49:41 +0000 (UTC) Received: from orion.archlinux.org (localhost [127.0.0.1]) by orion.archlinux.org (Postfix) with ESMTP id EA4A61C7391DDD for ; Mon, 1 Jun 2020 16:49:37 +0000 (UTC) Received: from tsubame.mg0.fr (tsubame.mg0.fr [IPv6:2001:41d0:401:3100::402b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by orion.archlinux.org (Postfix) with ESMTPS for ; Mon, 1 Jun 2020 16:49:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mg0.fr; s=tsubame; h=Message-ID:Subject:To:From:Date:cc; bh=GCUJ6U/XF5wWPirv8tTfYqGa+0BU9KvWjzTw/+2F5kY=; b=Jj4okNQitUzwYj9zEMRFiI7BMn vs9dFLgEQ5sY4MUrqTDQ78lMnWK6VmkxBlXDKMnAQ7BsinA3t/A3hvhwCXRvBqxtLPboJDMRF4eDl xh3THZ1XlkJdgX8sPWq26DMPwBC6rkcJXzIrclxNe80MVCyv2KPs7QDDvcfzQxKyRBGg=; Received: from fmang by tsubame.mg0.fr with local (Exim 4.93) (envelope-from ) id 1jfnd3-0026J2-3u for aur-dev@archlinux.org; Mon, 01 Jun 2020 18:49:37 +0200 Date: Mon, 1 Jun 2020 18:49:37 +0200 From: =?utf-8?b?RnLDqWTDqXJpYw==?= Mangano-Tarumi To: aur-dev@archlinux.org Subject: [PATCH v2 1/2] aurweb.spawn: Integrate FastAPI and nginx Message-ID: <20200601164937.GA500896@tsubame.mg0.fr> References: <159093460328.993942.12941354672188187108@typhoon> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <159093460328.993942.12941354672188187108@typhoon> X-BeenThere: aur-dev@archlinux.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Arch User Repository \(AUR\) Development" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: aur-dev-bounces@archlinux.org Sender: "aur-dev" aurweb.spawn used to launch only PHP’s built-in server. Now it spawns a dummy FastAPI application too. Since both stacks spawn their own HTTP server, aurweb.spawn also spawns nginx as a reverse proxy to mount them under the same base URL, defined by aur_location in the configuration. --- .gitlab-ci.yml | 2 +- TESTING | 3 +- aurweb/asgi.py | 8 +++++ aurweb/spawn.py | 80 +++++++++++++++++++++++++++++++++++++------- conf/config.defaults | 7 ++++ 5 files changed, 86 insertions(+), 14 deletions(-) create mode 100644 aurweb/asgi.py diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 74784fce..f6260ebb 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -11,7 +11,7 @@ before_script: base-devel git gpgme protobuf pyalpm python-mysql-connector python-pygit2 python-srcinfo python-bleach python-markdown python-sqlalchemy python-alembic python-pytest python-werkzeug - python-pytest-tap + python-pytest-tap python-fastapi uvicorn nginx test: script: diff --git a/TESTING b/TESTING index a5e08cb8..31e3bcbd 100644 --- a/TESTING +++ b/TESTING @@ -12,7 +12,8 @@ INSTALL. 2) Install the necessary packages: # pacman -S --needed php php-sqlite sqlite words fortune-mod \ - python python-sqlalchemy python-alembic + python python-sqlalchemy python-alembic \ + python-fastapi uvicorn nginx Ensure to enable the pdo_sqlite extension in php.ini. diff --git a/aurweb/asgi.py b/aurweb/asgi.py new file mode 100644 index 00000000..5f30471a --- /dev/null +++ b/aurweb/asgi.py @@ -0,0 +1,8 @@ +from fastapi import FastAPI + +app = FastAPI() + + +@app.get("/hello/") +async def hello(): + return {"message": "Hello from FastAPI!"} diff --git a/aurweb/spawn.py b/aurweb/spawn.py index 5fa646b5..0506afa4 100644 --- a/aurweb/spawn.py +++ b/aurweb/spawn.py @@ -10,8 +10,10 @@ configuration anyway. import atexit import argparse +import os import subprocess import sys +import tempfile import time import urllib @@ -20,6 +22,7 @@ import aurweb.schema children = [] +temporary_dir = None verbosity = 0 @@ -35,10 +38,42 @@ class ProcessExceptions(Exception): super().__init__("\n- ".join(messages)) +def generate_nginx_config(): + """ + Generate an nginx configuration based on aurweb's configuration. + The file is generated under `temporary_dir`. + Returns the path to the created configuration file. + """ + aur_location = aurweb.config.get("options", "aur_location") + aur_location_parts = urllib.parse.urlsplit(aur_location) + config_path = os.path.join(temporary_dir, "nginx.conf") + config = open(config_path, "w") + # We double nginx's braces because they conflict with Python's f-strings. + config.write(f""" + events {{}} + daemon off; + error_log /dev/stderr info; + pid {os.path.join(temporary_dir, "nginx.pid")}; + http {{ + access_log /dev/stdout; + server {{ + listen {aur_location_parts.netloc}; + location / {{ + proxy_pass http://{aurweb.config.get("php", "bind_address")}; + }} + location /hello {{ + proxy_pass http://{aurweb.config.get("fastapi", "bind_address")}; + }} + }} + }} + """) + return config_path + + def spawn_child(args): """Open a subprocess and add it to the global state.""" if verbosity >= 1: - print(f"Spawning {args}", file=sys.stderr) + print(f":: Spawning {args}", file=sys.stderr) children.append(subprocess.Popen(args)) @@ -52,10 +87,29 @@ def start(): if children: return atexit.register(stop) - aur_location = aurweb.config.get("options", "aur_location") - aur_location_parts = urllib.parse.urlsplit(aur_location) - htmldir = aurweb.config.get("options", "htmldir") - spawn_child(["php", "-S", aur_location_parts.netloc, "-t", htmldir]) + + print("{ruler}\n" + "Spawing PHP and FastAPI, then nginx as a reverse proxy.\n" + "Check out {aur_location}\n" + "Hit ^C to terminate everything.\n" + "{ruler}" + .format(ruler=("-" * os.get_terminal_size().columns), + aur_location=aurweb.config.get('options', 'aur_location'))) + + # PHP + php_address = aurweb.config.get("php", "bind_address") + htmldir = aurweb.config.get("php", "htmldir") + spawn_child(["php", "-S", php_address, "-t", htmldir]) + + # FastAPI + host, port = aurweb.config.get("fastapi", "bind_address").rsplit(":", 1) + spawn_child(["python", "-m", "uvicorn", + "--host", host, + "--port", port, + "aurweb.asgi:app"]) + + # nginx + spawn_child(["nginx", "-p", temporary_dir, "-c", generate_nginx_config()]) def stop(): @@ -73,7 +127,7 @@ def stop(): try: p.terminate() if verbosity >= 1: - print(f"Sent SIGTERM to {p.args}", file=sys.stderr) + print(f":: Sent SIGTERM to {p.args}", file=sys.stderr) except Exception as e: exceptions.append(e) for p in children: @@ -99,9 +153,11 @@ if __name__ == '__main__': help='increase verbosity') args = parser.parse_args() verbosity = args.verbose - start() - try: - while True: - time.sleep(60) - except KeyboardInterrupt: - stop() + with tempfile.TemporaryDirectory(prefix="aurweb-") as tmpdirname: + temporary_dir = tmpdirname + start() + try: + while True: + time.sleep(60) + except KeyboardInterrupt: + stop() diff --git a/conf/config.defaults b/conf/config.defaults index 86fe765c..ed495168 100644 --- a/conf/config.defaults +++ b/conf/config.defaults @@ -41,9 +41,16 @@ cache = none cache_pkginfo_ttl = 86400 memcache_servers = 127.0.0.1:11211 +[php] +; Address PHP should bind when spawned in development mode by aurweb.spawn. +bind_address = 127.0.0.1:8081 ; Directory containing aurweb's PHP code, required by aurweb.spawn. ;htmldir = /path/to/web/html +[fastapi] +; Address uvicorn should bind when spawned in development mode by aurweb.spawn. +bind_address = 127.0.0.1:8082 + [ratelimit] request_limit = 4000 window_length = 86400