Sage Owl

The owl asks but never answers.


Twisted in 2017

written by Amber Brown on 2016-03-15

Twisted boasts one of the more mature codebases in the Python world, with a high emphasis on quality and compatibility. However, it's hardly on the cutting edge of Python developments -- elements of Twisted's packaging, testing, and documentation are behind the Python-wide curve, and this is something that needs to change for the project's continued success. Twisted also does not fully take advantage of some newer Python features, and includes a lot of functionality which may be better placed in new external libraries.

One of the complaints that I have heard is that people have little overarching idea of what is going on. A small development team communicating mostly over IRC and hundreds of separate issue tickets means that very few have a broad understanding of where the project is going. I, however, am one of those people, which is why I've decided to write this -- my vision of what, by 2017, Twisted will have changed.

Packaging

Twisted's packaging has historically been more home grown than many other projects -- its 15 year history meaning that it predates many of the things in Python packaging that we take for granted. For example, Twisted 1.0 was released mere days after the PyPI PEP was originally published, and several years before Virtualenv and pip were released.

2015 was the year of some modernisation, with the past "subproject" layout being retired and Twisted moving to being released only as a single package. You used to have the option of installing the whole package as you would today, or just Core (which contained the Reactor and Deferred parts), and then any selection of Web, Conch, Mail, etc. The problem with this was that Twisted never really was developed or even used as these separate projects, and it only led to complications, confusion, and surprising import errors (on some Ubuntu installations, from twisted import web won't import, because web isn't installed, but core is). There is still remaining historical artifacts, like the separate topfiles directories having their own NEWS and README files, and the main NEWS file having the separate modules with the same version listed.

Twisted's release process is unique in that it has "topfiles" and a "Version" class of its own. The two utilities that allowed the use of these tools have been split out, into towncrier (a complete rewrite) and Incremental. These two tools are not yet used by Twisted, but there are open tickets and PRs on Klein to trial it.

The other uniqueness is that we do not use setup.py sdist to build our tarballs, instead having custom code which once supported the old subproject layout. This means, for one, that we don't have tarballs that twine can upload to PyPI. There is an open ticket to just use setup.py sdist instead.

Twisted also has historically used the pre qualifier for release candidates. Although PEP440 allows its use, Setuptools has now started normalising to rc. Since the future of Twisted versioning is Incremental, there's an open ticket for changing pre to rc on its issue tracker.

Binary distribution has changed a lot over Twisted's history -- there was once a Twisted PPA for Ubuntu and MSI/exe installers for Windows, but since virtualenv has become the preferred option for development and deployment, and these binary packages don't support the use of virtualenvs, wheels have become the only method of binary distribution for Twisted. Unfortunately we can't ship universal wheels as we have C extensions -- mainly for Windows now, but also one for testing. There is a ticket open on Twisted which would put these C extensions in another optional package, meaning that Twisted itself becomes pure Python, with optional extensions, and can be distributed as a universal wheel. This external C extension package, of course, would also be distributed as a wheel for binary platforms we support (Windows, OS X).

One part of this C extension change, as well as the testing changes I'll go into later, will be moving Twisted into a src/ directory. This is now best practice in Python, and even if it wasn't, it's useful to Twisted in this area. The C extension package and Twisted would work in lock-step, being published in two separate packages to avoid problems related to Linux being a mess for binary distribution and some embedded platforms having compiler issues. Therefore, one src/ directory, with a "Twisted" package and a "C extensions" package, seems to be the best way forward. There's a ticket filed on Twisted for this.

Testing

Work started during the Twisted Fellowship for Twisted to move to Git, hosted on GitHub. This migration process is still underway, and there is a lot yet to do, but it does have a lot of interesting changes for testing Twisted. One such change is the future adoption of BuildBot Nine, which should be a nice new coat of paint on this process.

Twisted's test suite leaped into 2015 with the porting of Trial to Python 3, and our builders running nearly every test in a virtualenv. We do, however, need to bring it into 2016, and officially adopt Tox as a first-class testing citizen. It's currently not used on our buildbots, with our builders currently creating a virtualenv manually, installing dependencies into it, and running the test suite 'in-place', without installing Twisted. This has led to some Python 3 specific issues, which can only be detected and fixed by running the tests when installed.

As Twisted's Git migration progresses and PRs are eventually accepted, having tox environments be our standard method of testing Twisted will mean that we can run them on Travis without too much trouble. Of course, Travis will never replace our BuildBot infrastructure -- we test on eight Linux distros, FreeBSD, Windows, and OS X, and Travis can only provide one or two Linux distros and OS X. But, it can provide a helpful "screening" service -- most reviews generally have a "there's a pyflakes warning" which could be caught by Travis running on each PR, without requiring our full build fleet. There is a ticket open on Braid for this.

Documentation

I'll be the first to admit, Twisted's documentation has awful discoverability. Twisted's documentation was never a site unto itself, instead having twistedmatrix.com acting as such -- but nowadays, the front page of documentation is something that matters especially. Users (including yours truly) expect to be able to go to <project>.readthedocs.org and get something immediately useful, and we do not give that, instead dumping a page of links to pages of links. There's a ticket open on Twisted for exploring fixing this.

We also have a lot of outdated documentation, and a lot of it does not operate on Python 3. This is not something that works well on a ticket basis, unfortunately, and will require a sustained effort to reorganise and update all of this documentation into something we can be proud of in 2017.

Ecosystem Projects

2016 needs to be the year of project tooling unification. As mentioned above in packaging, towncrier and Incremental are two "Twisted release concepts" split into packages that do not require Twisted themselves. This means that ecosystem projects, such as Klein, Treq, and the former Divmod projects, can share these common tools (plus any others that are needed) and the wheel is not reinvented over and over.

Many Twisted ecosystem projects are HTTP related -- Klein, Treq, Divmod Mantissa, and txmongo all use Twisted's HTTP support. An important part of HTTP in 2016 and even moreso in 2017 is TLS, as the scourge of internet blackhats and nation-state actors have made it unsafe to operate unencrypted connections over the public internet, private cloud, or even on private home networks. Twisted's TLS support is among the best in the world, but we can take steps to make it even better. Dropping support for Twisteds that do not support good TLS (that is, versions below 14.0) is a step that will need to be considered for all ecosystem projects and applications that depend on Twisted. There's an open ticket for officially discouraging the use of Twisteds older than 14.0.

Features

Ah yes, the thing everyone really wants.

Twisted's HTTP/2 support, being developed by Cory Benfield under the auspices of the Hyper project is on the horizon. This will have a lot of ramifications for the ecosystem, as existing HTTP/1.1 frameworks such as Klein need to make sure they adapt. Progress can be followed on the Twisted ticket.

Although not directly Twisted, Glyph's txsni may gain Let's Encrypt support, in support of Divmod Mantissa supporting LE. Such work would mean that using Let's Encrypt with a txsni using Twisted server would be nearly "hands free", with automatic certificate requests and authenticating LE challenges.

Using Python 3 and Twisted is now a reality for a lot of projects, with the 50% ported (by lines of code) mark being reached in 2015. There are open tickets on Twisted for supporting yield from and supporting the new async and await keywords. Unfortunately, this is no simple task, and will require a blog post of its own to outline why it is -- but it is on the radar.

There is, of course, other features to be had in 2016 -- but these are the ones I'm really looking forward to.

Wrapup

I hope this has given you an idea of what is going on in Twisted this year. This hasn't gone into a lot of features, or truly exciting things to the common developer, but without constant incremental fixes to our policy, our code, and our surrounding systems, the headlining features won't come as easy.

So, here's to Twisted, and a successful 2016.

If my work on any of the above aligns with your interests, and you'd like to support it, I'm available for contract work. Additionally, you can sponsor Twisted through the Software Freedom Conservancy, a 501(c)3 non-profit organization.

Special thanks to Cory Benfield for pointing out some post-publish typos and weirdness.