An overview of asynchronous programming in the Python development language utilizing the Twisted framework.

2. Who Am I?
Developer/Evangelist at LaunchKey
Organizer of Las Vegas Python User Group
Founder/Co-Organizer of Las Vegas PHP UG
Co-Organizer of Las Vegas Developer UG
National Junior Basketball Coach

3. What is Asynchronous
Programming?
• Non-Linear
• Event Driven
• Functional
• Depends on non-blocking code

4. Why Asynchronous?
One thread for all connections with the following
benefits:
Shared application level objects and data reduces
memory footprint per connection
Limited threads allows for better CPU utilization of each
core thus fewer cores overall.
Both of the above enable large applications to run on
smaller virtual machines in the cloud.

5. What’s the Catch?
Functional non-blocking code development has a
learning curve.
Developing applications with objects that persist
beyond a single request can be more difficult than
standard WSGI applications.
Requires asynchronous libraries which can be
scarce in Python.

6. Twisted
Established Asynchronous Python Framework
1.0 Release was in March 2004
Utilizes asynchronous I/O
Server runs in Python 2.6+ and 3.2+

7. Deferred = Asynchronous
Analogous to promises/futures
Returned by asynchronous and concurrent calls
Use callbacks to process result of fulfillment
Process callbacks in the order they are registered with
each being passed the result of the predecessor
Can be quickly created for static data with defer.sucess
and defer.failure functions

8. Callback Example
d = defer.Deferred()
d.addCallback(lambda x: x + “ is an “)
d.addCallback(lambda x: x + “ example!“)
d.callback(“This”)
>>> This is an example!

9. Deferred Error Handling
Deferred objects catch all callback Exceptions
Exceptions are wrapped in a Failure object which can
also be returned by a callback or errback
If no errbacks remain, exception is swallowed and
processing is stopped
Remaining callbacks before next errback are skipped
If errback returns non-Failure, callback processing
resumes

10. Callback Non-Failure Flow
Deferred callback(result)
• callback - success
• callback – success
• errback - skipped
• errback – skipped

11. Errback Non-Failure Flow
Deferred callback(result)
• callback - success
• errback - failure
• callback – success
• errback – skipped

12. Errback Failure Flow
Deferred callback(result)
• callback - success
• callback – failure
• errback - failure
• errback – success

13. Threads = Concurrent
Threads run in thread pool managed by reactor
callInThread executes function in its own thread with
supplied data
deferToThread works like callInThread but returns a
Deferred object and processes the callbacks when
complete
Use threads for blocking code

14. Reactor
Manages the main event loop
Waits for and dispatches events or messages
May have predefined services to provide access to
network communications, threading, and event
dispatching. Available services are operating system
dependent

15. Application Hierarchy
Protocols
Service
Application Mail
SSL
Web
Mail
SMTP
UDP
StatsD

16. Application
Starts and stops the reactor
Manages services
Handles logging
Can run as daemon

17. Service
Registers with the reactor
Processed in the main event loop
Manages connections
Communicates via Protocols

18. Predefined Services
File Descriptor
Process
Threads
Time (scheduling)
SSL
TCP
UDP
UNIX (UNIX sockets)
Socket (3rd Party
Sockets)

19. Protocol
Utilized for interaction with services
Services with stateful connections utilize protocol
factories to generate a protocol instance per
connection.
Provided with method to send data to connection in
the form of transports or producers

20. Protocol Factory
Builds protocol instances
Injected into every protocol instance
Meant to perform inter-instance interaction
Can be used as testable registry for registry pattern
implementations

21. Included Protocols
AMP
Basic
Dict
Finger
FTP
GPS
HTB
HTTP
Ident
Loopback
Memcache
Mice
PCP
Portforward
Postfix
Shoutcast
SIP
SOCKSv4
Stateful
Telnet
TLS
Wire

22. Available Protocols
Mongo
LDAP
Dict
Finger
FTP
GPS
HTB
HTTP
Ident
Loopback
Memcache
Mice
PCP
Portforward
Postfix
Shoutcast
SIP
SOCKSv4
Stateful
Telnet
TLS
Wire
MongoDB*
LDAP*
* External packages maintained by Twisted Matrix Labs

23. Twisted Web
Web server built on Twisted HTTP Protocol
Can be stand alone web server
Can embedded as part of an application
Supports all of the features of most modern web
servers

24. Web Server Features
Virtual hosts
Proxy/Reverse proxy
Scripts (hello.py)
Static files
CGI
WSGI
URL rewrites
MIME processors
Sessions

25. Simple Web Application
Site object is root Resource
Resource object represents URL segment
Tree hierarchy of resources
Can directly return string or write response via
request object for deferred processing

26. Example Hierarchy
/
foo/
bar/
bat/
fizz/buz

27. Example Hierarchy Code
root = RootResource()
foo = FooResource()
foo.putChild(“bar”, BarResource())
foo.putChild(“bat”, BatResource())
root.putChild(“foo”, foo)
root.putChild(“fiz/buzz”, FizBuzzResource())
reactor.listenTCP(8080, Site(root))
reactor.start()

28. Concurrent Example
from twisted.web import resource
class DemoResource(resource.Resource):
def render_GET(self, request):
return "<html>foo</html>"

29. Deferred Example
def render_GET(self, request):
def process_resp(html, request):
request.write(html)
request.finish()
deferred = self.async_client.request()
deferred.addCallback(process_resp, request)
return NOT_DONE_YET

30. Demo Time

31. Example on GitHub:
https://github.com/aenglander/demo-python-twisted-
websocket
@adam_englander
#launchkey on freenode.net
#vegastech on freenode.net
adam@launchkey.com