the realm of Amber Brown

Komaci

2019.10.04

“Nephila komaci is a species of golden orb-web spider. It is the largest web-spinning spider known.” - Wikipedia

One of the best events of the past decade of Python development has been the standardisation and adoption of ASGI, the Asynchronous Server Gateway Interface.

As you may guess, it is a replacement for WSGI. It looks similar enough at a glance – a callable, with some more callables to send data, and some dictionaries that contain data – but it’s natively event-driven design means that it does more than HTTP, and doesn’t rely on synchronousness. It’s agnostic to the event driven system, instead using coroutines as a shim to allow frameworks like trio and Twisted to be both ASGI protocol servers and clients, instead of just asyncio. It also has multiple independent implementations, giving those that adopt it the ability to get going in production today, rather than making the mistake of WSGI and being standardised before production-grade implementations were available.

It’s a promising standard, and I think it’s time Twisted threw our hat in the ring. Although Daphne, the reference implementation, is built on Twisted, there’s several reasons why adopting ASGI as a protocol server and as a client is important for Twisted.

Firstly, the current ASGI protocol servers do not suit the needs of users that currently use Twisted Web. Implementations like Daphne, hypercorn, and uvicorn are very oriented towards being an “app runner” – you give it something to import, it looks for an app in that imported file, and then the server starts serving that app. While some Twisted Web users might operate their applications like this, the use of Twisted to write an application that happens to contain a web server (or many) is an appealing ability to many. As such, an ASGI server that “drops in” to existing Twisted Web uses, while integrating with Twisted-specific APIs such as Endpoints, is useful.

Secondly, ASGI fixes what I call the “web2” problem. Twisted can disconnect itself from being a complete “web development framework”, and instead focus on its strength – being a solid networking base. This does not mean that Twisted APIs such as Resource and Request have to go away, but it means that users that want Twisted’s capabilities and portability, but not its complexity, can theoretically pick another ASGI web framework (such as Django or – maybe – Flask) to write their actual webapp in. Of course, these web development frameworks might require asyncio-interop to function initially, but there’s nothing fundamentally stopping these web frameworks awaiting on Deferreds instead, due to pluggable coroutine schedulers. This widens Twisted’s user base to potentially everyone using an ASGI web framework, and can contribute to breaking up a potential asyncio monoculture within production ASGI servers.

Thirdly, ASGI provides a reasonable base upon which Twisted-native web frameworks could be written. Although these are not as widely useful as general-purpose ASGI web frameworks, the ability for a user of Twisted to construct a custom framework around the clearly defined ASGI spec (instead of the at-times-troublesome Twisted Web APIs) lowers the barrier to entry by reducing the amount of the stack said user has to learn about to be functional.

With these points and use-cases in mind, I introduce Komaci. It’s a web server built on top of Twisted, in pure Python, with complete spec conformance and Twisted integration as major design goals. In service of these goals, it uses the excellent h11 and h2 libraries to implement the on-wire protocols, both sans-I/O libraries with extensive test coverage of their own. It is designed to be a full replacement for twisted.web.http, twisted.web.server, and twisted.web.resource. h One element of being truly drop-in is the ability to run existing Twisted Web apps. Komaci ships with komaci.ASGISite, a drop-in replacement for twisted.web.server.Site. It, like Site, accepts a twisted.web.resource.Resource, but instead of being a ProtocolFactory served directly, is instead an ASGI app. This means that Komaci can serve it, as well as any ASGI server capable of hosting Twisted ASGI apps.

Although not ‘production-ready’ yet, it boasts a comprehensive test suite.

View my JetItUp infographic