Sage Owl

The owl asks but never answers.


Marching Ever Forward

written by Amber Brown on 2016-10-28

Twisted is a large ship to turn, with a lot of tooling, infrastructure, and process being accumulated in the fourteen years since 1.0. Luckily, I am a stubborn little owl, and I think we've managed to make some huge steps since I wrote on this topic last.

First, though, I'd like to thank the people that made this year so great for Twisted -- people such as Adi Roiban, Craig Rodrigues, Mark Williams, and nearly innumerable other individuals that have contributed patches and reviews, as well as corporations such as Rackspace that employ Glyph, and HPE that employ Cory Benfield. Upstream dependencies such as PyOpenSSL (thanks JP and Hynek!) and Cryptography (thanks Alex, Paul, et al!) let us do awesome things as well, and are an important part of what we offer the user. We literally couldn't do it without you. 💜

If I could summarise the most important of Twisted's changes this year in a single word, it would be process. We have made leaps and bounds in the way our developers work, and very much for the better. Twisted has now moved some elements of the developer process to GitHub, utilising pull requests instead of uploading patches to Trac, and GitHub Reviews instead of unfriendly Trac comments. In support of this drive to making things work on GitHub, we have also enabled Travis-CI and Appveyor on twisted/twisted, giving contributors without access to our buildbots a wide array of Python versions and OSs (2.7/3.3/3.4/3.5 on Linux, 2.7/3.5 on macOS, 2.7/3.5 on Windows) to test on. These public CI services come within fractions of a percent of our total coverage as recorded on the full BuildBot infrastructure, removing the status of non-committers as second-class citizens as far as access to knowing whether their patch actually works. The full-scale adoption of tox in our Travis, Appveyor, and BuildBot testing environments means that users can also test their code locally as if it were being tested by CI. Even though we still use Trac for our bug tracking due to major deficiencies in GitHub Issues, you can now log onto our Trac using your GitHub account -- making it way easier to use (and maintain!!).

Simplifying our packaging is of major importance to me, as the Release Manager. Since the writing of that post, Twisted now uses Incremental as its versioning system, replacing twisted.python.versions.Version (which now just re-exports Incremental). This brings Twisted in line with other Twisted projects like Klein, and replaces a non-trivial portion of our custom release tooling. Towncrier has not yet hit primetime but it is, too, on the radar, and will hopefully be implemented in Twisted early next year. Further simplification of packaging came in the full adoption of setuptools, wheel (for Windows), and twine. Uploading release candidates to PyPI has also made the "testing" period of them much easier to take advantage of!

Although our documentation had no major overhauls this year, a few tweaks to pydoctor have meant that private API is hidden from users by default, making the API docs much easier to browse.

As far as features, Twisted and the ecosystem has had a very good year. HTTP/2 server support (spearheaded by Cory Benfield and their Hyper project) landed in Twisted 16.3, and txacme came onto the scene not long after, making it is extremely simple to implement a Twisted Web server that speaks HTTP/2 and automatically provisions TLS certificates using txacme, with very little to no effort. rproxy is an example of a web server, being a small reverse proxy that if configured to send HSTS headers, on a new enough OpenSSL, will get an A+ on Qualys SSL Test. The new twist utility, also, improves upon the existing twistd implementation, but with a much simpler and clearer implementation.

It would be remiss of me not to also discuss Python 3 support, my personal little Twisted project. In the time since the original writing, our Python 3 tests now cover in excess of 70% of the codebase (compared to Python 2's tests covering 91%). This year alone we have ported Perspective Broker, large swathes of Conch, syslog support, inotify support, twisted.pair, SOCKS, the memcache client, lots of Names, XMPP support, AMP, and our WSGI runner, among dozens of other smaller and private modules. Official Py3+macOS support (including a ported and fully supported CFReactor) and Py3+Windows support (including a ported IOCPReactor) has been merged this year as well, rounding off the big 3 OSs. Asyncio interop has also been a major focus, with the new asyncioreactor allowing Twisted and asyncio to run side-by-side, twisted.internet.defer.ensureDeferred allowing you to use the same async and await (with bonus yield from) syntax, and some changes in Python 3.6's asyncio moving us closer to the possibility of Deferreds and asyncio Futures operating in harmony. I put a lot of work into this, even discussing it at the Python Language Summit, but there's still plenty to do.

Overall, Twisted has had a big year, and there is far more in the five releases (and six when I get 16.5 out the door very soon!) than I could ever cover here. If knowing all the details interests you, maybe you should check out our NEWS file -- it'll have all the details you'll ever want.

As far as next year? Well, I've got plans that I'm looking forward to sharing... soon!

If my work on any of the above aligns with your interests, and you'd like to support it, I'm available for contract work. Additionally, you can sponsor Twisted through the Software Freedom Conservancy, a 501(c)3 non-profit organization. Sponsoring Twisted means that they can get the funds to hire developers to work on essential Twisted maintenance tasks and development, so your money will be put to good use!