"WTF is Twisted? (or; owl amongst the ponies)" is a talk that introduces the Twisted asynchronous programming framework, how it works, and what uses it.

1. WTF is Twisted?
(or: Owl Amongst The Ponies)

2. About HawkOwl
• hawkowl@atleastfornow.net
• @hawkieowl on Twitter
• hawkowl on GitHub, Keybase &
ThoughtStreams
• PGP Key Fingerprint:
2100 5027 897E C8AE 1D94
A905 2308 B479 D392 4A11
• Twisted Core Committer and Release
Manager (13.2 & 14.0)
• Is not actually an owl

3. Twisted in One Slide
• Twisted is a framework for writing network
applications in Python
• Best known for “Deferreds”, Twisted’s async
abstraction
• Contains protocol implementations for HTTP,
SMTP, IMAP, DNS, etc.

4. Why would you want to use
Twisted?
• Higher performance in some common situations
• Stable, mature codebase
• Protocol implementations for lots of things
• High test coverage
• Great community consisting of excellent
individuals who do excellent work

5. What makes Twisted especially
different from other projects?
• Twisted has a reputation for being the “black
sheep” in the Python community, mostly
because we do things “weirdly” and “hard”
• But, surprise! It turns out the Twisted developers
were right all along (see: asyncio)
• Different set of principals, from both the
longevity of the project and their specific
problems

6. The Tenets of Twisted
(the time-tested, battle-hardened principals of the
Twisted project, as interpreted by HawkOwl)

7. Subclassing is bad.
Composition is the new inheritance!

8. Multiple inheritance is
really bad.
Handy hint: If you’ve written the word “super”, it’s
already too late.

9. Time-based releases
are king.
For large, evolving frameworks, it’s better to gradually
change than have a massive app migration effort.

10. Early APIs > No APIS
You won’t know how your API works in the real world until
other people use it. You can always deprecate it later.

11. All code needs tests.
No exceptions.
It doesn’t matter if you’re fixing an urgent release regression
- an eye on coverage saves you a lot of time later.

12. All code needs reviewing.
No exceptions.
Code review is invaluable, even if it does descend into
bikeshedding if they can’t find anything wrong.

13. Only bad APIs can’t
be unit tested.
Can’t unit test your API? Too bad, time to refactor it.

14. Build slaves are
amazing.
Test your code on everything. Virtual machines are
cheap, compatibility problems down the line are not.

15. –David Reid, Glyph, et al
“Call a function with some arguments and
inspect some return values.”

16. How Twisted Works
(a short, mostly-correct overview)

17. The Event Loop
(Spin me right round, baby, right round)

18. The Reactor
• Twisted, being an event-driven framework,
provides an event loop, or “reactor”.
• This is an infinite loop, wrapped around a blocking
call that waits until some I/O is ready
• Once the I/O is ready, this fires off an event to run
the code that was waiting for it
• Runs other tasks while I/O blocked tasks are
waiting

19. Using the Reactor
• To utilise the reactor, you use Deferreds.
• Deferreds are an “IOU” for an actual result
sometime in the future
• A Deferred contains a callback chain, a series of
functions run in sequence after it gets a result
• Deferreds get their results from events occurring -
such as I/O being received or some task being
finished

20. Benefits
• For I/O bound applications, allows you to
efficiently use CPU resources, as the reactor
runs other things in the meantime
• Integration with common event loops - for
example, GTK or QT, to use your Twisted code
into your GTK or QT applications

21. The Callback Chain
(Hey, I just met you, and this is crazy,
but here’s a function, so call it maybe?)

22. The Callback Chain
• The easiest way to understand the callback
chain is to visualise it as a production line.
• A production line is made up of discrete blocks -
each doing a separate thing.
• Let’s imagine a such a production line…

23. Lets say that we are building a toy car, and we
need to do the following:
1. Create toy car body from materials
2. Paint racing stripes on car body
3. Add wheels to the car body
4. Put the completed car in a box

24. Here is how we would do that in psuedo-code
Deferreds:
!
! d = Deferred()!
!d.addCallback(buildCarBody)!
d.addCallback(paintStripes)!
d.addCallback(addWheels)!
d.addCallback(placeInBox)!
# Start the callback chain!
d.callback(rawMaterials)

25. Deferreds
• Each function in the chain is called with the
result of the previous one as its first argument
• Functions in the callback chain can return either
a result or a Deferred that fires with a result
sometime in the future

26. Lists of Deferreds
• If you have several Deferred operations to do at
the same time, and want your callback to fire
only when they are all done, use
DeferredList.!
• Pass it a list of Deferreds, and it will return a
Deferred that fires with a list containing the
results of each when they have all finished.

27. Error Handling
• Error handling is done by errbacks.
• Errbacks are like callbacks, but are run when an
exception is raised.
• An errback will catch all errors that occur on the
chain before it.

28. Deferred Benefits
• Forces you to break up your logic into sections,
broken up over external interface accesses
• Encourages reuse, as these sections quite
commonly do one thing to an input and return it
• Can be cancelled (eg. if a client disconnected
and you don’t care about the result anymore)

29. Deferred Benefits
• Discourages global mutable state by having
different execution threads share the same
global state
• No global mutable state means you can scale
easier
• If your global state is databases, just spin up
more Twisted instances to use all your CPUs

30. Extending the Callback Chain
(Because sometimes things like to do other things)

31. Deferreds returning
Deferreds
• Deferreds can contain their own callback chain -
meaning that one Deferred returned may have
several callbacks being called inside it
• This is invisible to the “top level” of callbacks
• You can extend your callback chain at any time

32. What can Twisted do?
(Enough of the boring stuff!)

33. Web
• Klein, a Flask-like API in Twisted
• Saratoga, a HTTP API framework
• Autobahn, a Websockets implementation
• Hendrix, a Django deployment framework
• Static file serving:
twistd -n web --port tcp:8080 --path /tmp

34. Klein (Flask in Twisted)
from klein import Klein!
app = Klein()!
!
@app.route('/')!
def home(request):!
return 'Hello, world!'!
!
app.run("localhost", 8080)

35. Saratoga (HTTP API)
{!
"metadata": {!
"name": “APIExample", "versions": [1]!
},!
"endpoints": [{!
! ! "endpoint": “example", "getProcessors": [!
{"versions": [1]}!
]}!
]}!
from saratoga.api import SaratogaAPI!
import json!
!
class APIExample(object):!
class v1(object):!
def example_GET(self, request, params):!
return {"hello": "world"}!
!
api = SaratogaAPI(APIExample, json.load(open("apidef.json")))!
api.run()

36. Email
• Complete SMTP and ESMTP server and clients
• POP3 and IMAP4 servers and clients
• Support for TLS-secured email out of the box
For more info, see:
http://twistedmatrix.com/documents/
current/mail/index.html

37. DNS
• A caching, recursive DNS server out of the box
twistd -n dns -c —port 10053 --recursive
• DNS client APIs
• DNS server toolkit
For more info, see:
http://twistedmatrix.com/documents/
current/names/howto/custom-server.html

38. Chat
• IRC Server
twistd -n words --irc-port tcp:8400
--group roomname --auth memory:usr:pass
• IRC client (for making IRC bots)
• XMPP client
For more info, see:
http://twistedmatrix.com/documents/
current/words/examples/

39. Want to know more?
• Twisted Docs
http://twistedmatrix.com/documents/current/
• Iffy’s Twisted FTW
http://iffycan.com/twistedftw/
• HawkOwl’s Technicals
http://technicals.atleastfornow.net/

40. Questions?
(sane answers not guaranteed)