Note: This document is relevant for the version of Twisted that were current previous to IPC10. Even at the time of its release, there were errata issued to make it current. It is remaining unaltered for historical purposes but it is no longer accurate.
Twisted is a framework for writing asynchronous,
event-driven networked programs in Python -- both clients and
servers. In addition to abstractions for low-level system calls
like select(2) and socket(2), it also
includes a large number of utility functions and classes, which
make writing new servers easy. Twisted includes support for
popular network protocols like HTTP and SMTP, support for GUI
frameworks like GTK+/GNOME and
Tk and many other classes designed to make network
programs easy. Whenever possible, Twisted uses Python's
introspection facilities to save the client programmer as much
work as possible. Even though Twisted is still work in
progress, it is already usable for production systems -- it can
be used to bring up a Web server, a mail server or an IRC
server in a matter of minutes, and require almost no
configuration.
Keywords: internet, network, framework, event-based, asynchronous
Python lends itself to writing frameworks. Python has a simple class model, which facilitates inheritance. It has dynamic typing, which means code needs to assume less. Python also has built-in memory management, which means application code does not need to track ownership. Thus, when writing a new application, a programmer often finds himself writing a framework to make writing this kind of application easier. Twisted evolved from the need to write high-performance interoperable servers in Python, and making them easy to use (and difficult to use incorrectly).
There are three ways to write network programs:
When dealing with many connections in one thread, the scheduling is the responsibility of the application, not the operating system, and is usually implemented by calling a registered function when each connection is ready to for reading or writing -- commonly known as event-driven, or callback-based, programming.
Since multi-threaded programming is often tricky, even with
high level abstractions, and since forking Python processes has
many disadvantages, like Python's reference counting not
playing well with copy-on-write and problems with shared state,
it was felt the best option was an event-driven framework. A
benefit of such approach is that by letting other event-driven
frameworks take over the main loop, server and client code are
essentially the same - making peer-to-peer a reality. While
Twisted includes its own event loop, Twisted can already
interoperate with GTK+'s and Tk's
mainloops, as well as provide an emulation of event-based I/O
for Jython (specific support for the Swing toolkit is planned).
Client code is never aware of the loop it is running under, as
long as it is using Twisted's interface for registering for
interesting events.
Some examples of programs which were written using the
Twisted framework are twisted.web (a web server),
twisted.mail (a mail server, supporting both SMTP
and POP3, as well as relaying), twisted.words (a
chat application supporting integration between a variety of IM
protocols, like IRC, AOL Instant Messenger's TOC and
Perspective Broker, a remote-object protocol native to
Twisted), im (an instant messenger which connects
to twisted.words) and faucet (a GUI client for the
twisted.reality interactive-fiction framework).
Twisted can be useful for any network or GUI application
written in Python.
However, event-driven programming still contains some tricky aspects. As each callback must be finished as soon as possible, it is not possible to keep persistent state in function-local variables. In addition, some programming techniques, such as recursion, are impossible to use. Event-driven programming has a reputation of being hard to use due to the frequent need to write state machines. Twisted was built with the assumption that with the right library, event-driven programming is easier then multi-threaded programming. Twisted aims to be that library.
Twisted includes both high-level and low-level support for protocols. Most protocol implementation by twisted are in a package which tries to implement "mechanisms, not policy". On top of those implementations, Twisted includes usable implementations of those protocols: for example, connecting the abstract HTTP protocol handler to a concrete resource-tree, or connecting the abstract mail protocol handler to deliver mail to maildirs according to domains. Twisted tries to come with as much functionality as possible out of the box, while not constraining a programmer to a choice between using a possibly-inappropriate class and rewriting the non-interesting parts himself.
Twisted also includes Perspective Broker, a simple remote-object framework, which allows Twisted servers to be divided into separate processes as the end deployer (rather then the original programmer) finds most convenient. This allows, for example, Twisted web servers to pass requests for specific URLs with co-operating servers so permissions are granted according to the need of the specific application, instead of being forced into giving all the applications all permissions. The co-operation is truly symmetrical, although typical deployments (such as the one which the Twisted web site itself uses) use a master/slave relationship.
Twisted is not alone in the niche of a Python network framework. One of the better known frameworks is Medusa. Medusa is used, among other things, as Zope's native server serving HTTP, FTP and other protocols. However, Medusa is no longer under active development, and the Twisted development team had a number of goals which would necessitate a rewrite of large portions of Medusa. Twisted seperates protocols from the underlying transport layer. This seperation has the advantages of resuability (for example, using the same clients and servers over SSL) and testability (because it is easy to test the protocol with a much lighter test harness) among others. Twisted also has a very flexible main-loop which can interoperate with third-party main-loops, making it usable in GUI programs too.
Python comes out of the box with "batteries included".
However, it seems that many Python projects rewrite some basic
parts: logging to files, parsing options and high level
interfaces to reflection. When the Twisted project found itself
rewriting those, it moved them into a separate subpackage,
which does not depend on the rest of the twisted framework.
Hopefully, people will use twisted.python more and
solve interesting problems instead. Indeed, it is one of
Twisted's goals to serve as a repository for useful Python
code.
One useful module is twisted.python.reflect,
which has methods like prefixedMethods, which
returns all methods with a specific prefix. Even though some
modules in Python itself implement such functionality (notably,
urllib2), they do not expose it as a function
usable by outside code. Another useful module is
twisted.python.hook, which can add pre-hooks and
post-hooks to methods in classes.
# Add all method names beginning with opt_ to the given # dictionary. This cannot be done with dir(), since # it does not search in superclasses dct = {} reflect.addMethodNamesToDict(self.__class__, dct, "opt_") # Sum up all lists, in the given class and superclasses, # which have a given name. This gives us "different class # semantics": attributes do not override, but rather append flags = [] reflect.accumulateClassList(self.__class__, 'optFlags', flags) # Add lock-acquire and lock-release to all methods which # are not multi-thread safe for methodName in klass.synchronized: hook.addPre(klass, methodName, _synchPre) hook.addPost(klass, methodName, _synchPost)Listing 1: Using
twisted.python.reflectandtwisted.python.hook
The twisted.python subpackage also contains a
high-level interface to getopt which supplies as much power as
plain getopt while avoiding long
if/elif chains and making many common
cases easier to use. It uses the reflection interfaces in
twisted.python.reflect to find which options the
class is interested in, and constructs the argument to
getopt. Since in the common case options' values
are just saved in instance attributes, it is very easy to
indicate interest in such options. However, for the cases
custom code needs to be run for an option (for example,
counting how many -v options were given to
indicate verbosity level), it will call a method which is named
correctly.
class ServerOptions(usage.Options): # Those are (short and long) options which # have no argument. The corresponding attribute # will be true iff this option was given optFlags = [['nodaemon','n'], ['profile','p'], ['threaded','t'], ['quiet','q'], ['no_save','o']] # This are options which require an argument # The default is used if no such option was given # Note: since options can only have string arguments, # putting a non-string here is a reliable way to detect # whether the option was given optStrings = [['logfile','l',None], ['file','f','twistd.tap'], ['python','y',''], ['pidfile','','twistd.pid'], ['rundir','d','.']] # For methods which can be called multiple times # or have other unusual semantics, a method will be called # Twisted assumes that the option needs an argument if and only if # the method is defined to accept an argument. def opt_plugin(self, pkgname): pkg = __import__(pkgname) self.python = os.path.join(os.path.dirname( os.path.abspath(pkg.__file__)), 'config.tac') # Most long options based on methods are aliased to short # options. If there is only one letter, Twisted knows it is a short # option, so it is "-g", not "--g" opt_g = opt_plugin try: config = ServerOptions() config.parseOptions() except usage.error, ue: print "%s: %s" % (sys.argv[0], ue) sys.exit(1)Listing 2:
twistd's Usage Code
Unlike getopt, Twisted has a useful abstraction
for the non-option arguments: they are passed as arguments to
the parsedArgs method. This means too many
arguments, or too few, will cause a usage error, which will be
flagged. If an unknown number of arguments is desired,
explicitly using a tuple catch-all argument will work.
The formats of configuration files have shown two visible trends over the years. On the one hand, more and more programmability has been added, until sometimes they become a new language. The extreme end of this trend is using a regular programming language, such as Python, as the configuration language. On the other hand, some configuration files became more and more machine editable, until they become a miniature database formates. The extreme end of that trend is using a generic database tool.
Both trends stem from the same rationale -- the need to use a powerful general purpose tool instead of hacking domain specific languages. Domain specific languages are usually ad-hoc and not well designed, having neither the power of general purpose languages nor the predictable machine editable format of generic databases.
Twisted combines these two trends. It can read the configuration either from a Python file, or from a pickled file. To some degree, it integrates the approaches by auto-pickling state on shutdown, so the configuration files can migrate from Python into pickles. Currently, there is no way to go back from pickles to equivalent Python source, although it is planned for the future. As a proof of concept, the RPG framework Twisted Reality already has facilities for creating Python source which evaluates into a given Python object.
from twisted.internet import main from twisted.web import proxy, server site = server.Site(proxy.ReverseProxyResource('www.yahoo.com', 80, '/')) application = main.Application('web-proxy') application.listenOn(8080, site)Listing 3: The configuration file for a reverse web proxy
Twisted's main program, twistd, can receive
either a pickled twisted.internet.main.Application
or a Python file which defines a variable called
application. The application can be saved at any
time by calling its save method, which can take an
optional argument to save to a different file name. It would be
fairly easy, for example, to have a Twisted server which saves
the application every few seconds to a file whose name depends
on the time. Usually, however, one settles for the default
behavior which saves to a shutdown file. Then, if
the shutdown configuration proves suitable, the regular pickle
is replaced by the shutdown file. Hence, on the fly
configuration changes, regardless of complexity, can always
persist.
There are several client/server protocols which let a
suitably privileged user to access to application variable and
change it on the fly. The first, and least common denominator,
is telnet. The administrator can telnet into twisted, and issue
Python statements to her heart's content. For example, one can
add ports to listen on to the application, reconfigure the web
servers and various other ways by simple accessing
__main__.application. Some proof of concepts for a
simple suite of command-line utilities to control a Twisted
application were written, including commands which allow an
administrator to shut down the server or save the current state
to a tap file. These are especially useful on Microsoft
Windows(tm) platforms, where the normal UNIX way of
communicating shutdown requests via signals are less
reliable.
If reconfiguration on the fly is not necessary, Python
itself can be used as the configuration editor. Loading the
application is as simple as unpickling it, and saving it is
done by calling its save method. It is quite easy
to add more services or change existing ones from the Python
interactive mode.
A more sophisticated way to reconfigure the application on
the fly is via the manhole service. Manhole is a client/server
protocol based on top of Perspective Broker, Twisted's
translucent remote-object protocol which will be covered later.
Manhole has a graphical client called gtkmanhole
which can access the server and change its state. Since Twisted
is modular, it is possible to write more services for user
friendly configuration. For example, through-the-web
configuration is planned for several services, notably
mail.
For cases where a third party wants to distribute both the
code for a server and a ready to run configuration file, there
is the plugin configuration. Philosophically similar to the
--python option to twistd, it
simplifies the distribution process. A plugin is an archive
which is ready to be unpacked into the Python module path. In
order to keep a clean tree, twistd extends the
module path with some Twisted-specific paths, like the
directory TwistedPlugins in the user's home
directory. When a plugin is unpacked, it should be a Python
package which includes, alongside __init__.py a
file named config.tac. This file should define a
variable named application, in a similar way to
files loaded with --python. The plugin way of
distributing configurations is meant to reduce the temptation
to put large amount of codes inside the configuration file
itself.
Putting class and function definition inside the configuration files would make the persistent servers which are auto-generated on shutdown useless, since they would not have access to the classes and functions defined inside the configuration file. Thus, the plugin method is intended so classes and functions can still be in regular, importable, Python modules, but still allow third parties distribute powerful configurations. Plugins are used by some of the Twisted Reality virtual worlds.
Port is the Twisted class which represents a
socket listening on a port. Currently, twisted supports both
internet and unix-domain sockets, and there are SSL classes
with identical interface. A Port is only
responsible for handling the transfer layer. It calls
accept on the socket, checks that it actually
wants to deal with the connection and asks its factory for a
protocol. The factory is usually a subclass of
twisted.protocols.protocol.Factory, and its most
important method is buildProtocol. This should
return something that adheres to the protocol interface, and is
usually a subclass of
twisted.protocols.protocol.Protocol.
from twisted.protocols import protocol from twisted.internet import main, tcp class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.Factory() factory.protocol = Echo port = tcp.Port(8000, factory) app = main.Application("echo") app.addPort(port) app.run()Listing 4: A Simple Twisted Application
The factory is responsible for two tasks: creating new protocols, and keeping global configuration and state. Since the factory builds the new protocols, it usually makes sure the protocols have a reference to it. This allows protocols to access, and change, the configuration. Keeping state information in the factory is the primary reason for keeping an abstraction layer between ports and protocols. Examples of configuration information is the root directory of a web server or the user database of a telnet server. Note that it is possible to use the same factory in two different Ports. This can be used to run the same server bound to several different addresses but not to all of them, or to run the same server on a TCP socket and a UNIX domain sockets.
A protocol begins and ends its life with
connectionMade and connectionLost;
both are called with no arguments. connectionMade
is called when a connection is first established. By then, the
protocol has a transport attribute. The
transport attribute is a Transport -
it supports write and loseConnection.
Both these methods never block: write actually
buffers data which will be written only when the transport is
signalled ready to for writing, and loseConnection
marks the transport for closing as soon as there is no buffered
data. Note that transports do not have a
read method: data arrives when it arrives, and the
protocol must be ready for its dataReceived
method, or its connectionLost method, to be
called. The transport also supports a getPeer
method, which returns parameters about the other side of the
transport. For TCP sockets, this includes the remote IP and
port.
# A tcp port-forwarder # A StupidProtocol sends all data it gets to its peer. # A StupidProtocolServer connects to the host/port, # and initializes the client connection to be its peer # and itself to be the client's peer from twisted.protocols import protocol class StupidProtocol(protocol.Protocol): def connectionLost(self): self.peer.loseConnection();del self.peer def dataReceived(self, data): self.peer.write(data) class StupidProtocolServer(StupidProtocol): def connectionMade(self): clientProtocol = StupidProtocol() clientProtocol.peer = self.transport self.peer = tcp.Client(self.factory.host, self.factory.port, clientProtocol) # Create a factory which creates StupidProtocolServers, and # has the configuration information they assume def makeStupidFactory(host, port): factory = protocol.Factory() factory.host, factory.port = host, port factory.protocol = StupidProtocolServer return factoryListing 5: TCP forwarder code
While Twisted has the ability to let other event loops take
over for integration with GUI toolkits, it usually uses its own
event loop. The event loop code uses global variables to
maintain interested readers and writers, and uses Python's
select() function, which can accept any object
which has a fileno() method, not only raw file
descriptors. Objects can use the event loop interface to
indicate interest in either reading to or writing from a given
file descriptor. In addition, for those cases where time-based
events are needed (for example, queue flushing or periodic POP3
downloads), Twisted has a mechanism for repeating events at
known delays. While far from being real-time, this is enough
for most programs' needs.
Unfortunately, handling arbitrary data chunks is a hard way
to code a server. This is why twisted has many classes sitting
in submodules of the twisted.protocols package which give
higher level interface to the data. For line oriented
protocols, LineReceiver translates the low-level
dataReceived events into lineReceived
events. However, the first naive implementation of
LineReceiver proved to be too simple. Protocols
like HTTP/1.1 or Freenet have packets which begin with header
lines that include length information, and then byte streams.
LineReceiver was rewritten to have a simple
interface for switching at the protocol layer between
line-oriented parts and byte-stream parts.
Another format which is gathering popularity is Dan J.
Bernstein's netstring format. This format keeps ASCII text as
ASCII, but allows arbitrary bytes (including nulls and
newlines) to be passed freely. However, netstrings were never
designed to be used in event-based protocols where over-reading
is unavoidable. Twisted makes sure no user will have to deal
with the subtle problems handling netstrings in event-driven
programs by providing NetstringReceiver.
For even higher levels, there are the protocol-specific protocol classes. These translate low-level chunks into high-level events such as "HTTP request received" (for web servers), "approve destination address" (for mail servers) or "get user information" (for finger servers). Many RFCs have been thus implemented for Twisted (at latest count, more then 12 RFCs have been implemented). One of Twisted's goals is to be a repository of event-driven implementations for various protocols in Python.
class DomainSMTP(SMTP): def validateTo(self, helo, destination): try: user, domain = string.split(destination, '@', 1) except ValueError: return 0 if domain not in self.factory.domains: return 0 if not self.factory.domains[domain].exists(user, domain, self): return 0 return 1 def handleMessage(self, helo, origin, recipients, message): # No need to check for existence -- only recipients which # we approved at the validateTo stage are passed here for recipient in recipients: user, domain = string.split(recipient, '@', 1) self.factory.domains[domain].saveMessage(origin, user, message, domain)Listing 6: Implementation of virtual domains using the SMTP protocol class
Copious documentation on writing new protocol abstraction exists, since this is the largest amount of code written -- much like most operating system code is device drivers. Since many different protocols have already been implemented, there are also plenty of examples to draw on. Usually implementing the client-side of a protocol is particularly challenging, since protocol designers tend to assume much more state kept on the client side of a connection then on the server side.
twisted.tap Package and
mktapSince one of Twisted's configuration formats are pickles,
which are tricky to edit by hand, Twisted evolved a framework
for creating such pickles. This framework is contained in the
twisted.tap package and the mktap
script. New servers, or new ways to configure existing servers,
can easily participate in the twisted.tap framework by creating
a twisted.tap submodule.
All twisted.tap submodules must conform to a
rigid interface. The interface defines functions to accept the
command line parameters, and functions to take the processed
command line parameters and add servers to
twisted.main.internet.Application. Existing
twisted.tap submodules use
twisted.python.usage, so the command line format
is consistent between different modules.
The mktap utility gets some generic options,
and then the name of the server to build. It imports a
same-named twisted.tap submodule, and lets it
process the rest of the options and parameters. This makes sure
that the process configuring the main.Application
is agnostic for where it is used. This allowed
mktap to grow the --append option,
which appends to an existing pickle rather then creating a new
one. This option is frequently used to post-add a telnet server
to an application, for net-based on the fly configuration
later.
When running mktap under UNIX, it saves the
user id and group id inside the tap. Then, when feeding this
tap into twistd, it changes to this user/group id
after binding the ports. Such a feature is necessary in any
production-grade server, since ports below 1024 require root
privileges to use on UNIX -- but applications should not run as
root. In case changing to the specified user causes difficulty
in the build environment, it is also possible to give those
arguments to mktap explicitly.
from twisted.internet import tcp, stupidproxy from twisted.python import usage usage_message = """ usage: mktap stupid [OPTIONS] Options are as follows: --port <#>, -p: set the port number to <#>. --host <host>, -h: set the host to <host> --dest_port <#>, -d: set the destination port to <#> """ class Options(usage.Options): optStrings = [["port", "p", 6666], ["host", "h", "localhost"], ["dest_port", "d", 6665]] def getPorts(app, config): s = stupidproxy.makeStupidFactory(config.host, int(config.dest_port)) return [(int(config.port), s)]Listing 7:
twisted.tap.stupid
The twisted.tap framework is one of the reasons
servers can be set up with little knowledge and time. Simply
running mktap with arguments can bring up a web
server, a mail server or an integrated chat server -- with
hardly any need for maintainance. As a working
proof-on-concept, the tap2deb utility exists to
wrap up tap files in Debian packages, which include scripts for
running and stopping the server and interact with
init(8) to make sure servers are automatically run
on start-up. Such programs can also be written to interface
with the Red Hat Package Manager or the FreeBSD package
management systems.
% mktap --uid 33 --gid 33 web --static /var/www --port 80 % tap2deb -t web.tap -m 'Moshe Zadka <moshez@debian.org>' % su password: # dpkg -i .build/twisted-web_1.0_all.debListing 8: Bringing up a web server on a Debian system
Sometimes, threads are unavoidable or hard to avoid. Many
legacy programs which use threads want to use Twisted, and some
vendor APIs have no non-blocking version -- for example, most
database systems' API. Twisted can work with threads, although
it supports only one thread in which the main select loop is
running. It can use other threads to simulate non-blocking API
over a blocking API -- it spawns a thread to call the blocking
API, and when it returns, the thread calls a callback in the
main thread. Threads can call callbacks in the main thread
safely by adding those callbacks to a list of pending events.
When the main thread is between select calls, it searches
through the list of pending events, and executes them. This is
used in the twisted.enterprise package to supply
an event driven interfaces to databases, which uses Python's DB
API.
Twisted tries to optimize for the common case -- no threads.
If there is need for threads, a special call must be made to
inform the twisted.python.threadable module that
threads will be used. This module is implemented differently
depending on whether threads will be used or not. The decision
must be made before importing any modules which use threadable,
and so is usually done in the main application. For example,
twistd has a command line option to initialize
threads.
Twisted also supplies a module which supports a threadpool, so the common task of implementing non-blocking APIs above blocking APIs will be both easy and efficient. Threads are kept in a pool, and dispatch requests are done by threads which are not working. The pool supports a maximum amount of threads, and will throw exceptions when there are more requests than allowable threads.
One of the difficulties about multi-threaded systems is
using locks to avoid race conditions. Twisted uses a mechanism
similar to Java's synchronized methods. A class can declare a
list of methods which cannot safely be called at the same time
from two different threads. A function in threadable then uses
twisted.python.hook to transparently add
lock/unlock around these methods. This allows Twisted classes
to be written without thought about threading, except for one
localized declaration which does not entail any performance
penalty for the single-threaded case.
Mail servers have a history of security flaws. Sendmail is by now the poster boy of security holes, but no mail servers, bar maybe qmail, are free of them. Like Dan Bernstein of qmail fame said, mail cannot be simply turned off -- even the simplest organization needs a mail server. Since Twisted is written in a high-level language, many problems which plague other mail servers, notably buffer overflows, simply do not exist. Other holes are avoidable with correct design. Twisted Mail is a project trying to see if it is possible to write a high quality high performance mail server entirely in Python.
Twisted Mail is built on the SMTP server and client protocol classes. While these present a level of abstraction from the specific SMTP line semantics, they do not contain any message storage code. The SMTP server class does know how to divide responsibility between domains. When a message arrives, it analyzes the recipient's address, tries matching it with one of the registered domain, and then passes validation of the address and saving the message to the correct domain, or refuses to handle the message if it cannot handle the domain. It is possible to specify a catch-all domain, which will usually be responsible for relaying mails outwards.
While correct relaying is planned for the future, at the moment we have only so-called "smarthost" relaying. All e-mail not recognized by a local domain is relayed to a single outside upstream server, which is supposed to relay the mail further. This is the configuration for most home machines, which are Twisted Mail's current target audience.
Since the people involved in Twisted's development were reluctant to run code that runs as a super user, or with any special privileges, it had to be considered how delivery of mail to users is possible. The solution decided upon was to have Twisted deliver to its own directory, which should have very strict permissions, and have users pull the mail using some remote mail access protocol like POP3. This means only a user would write to his own mail box, so no security holes in Twisted would be able to adversely affect a user.
Future plans are to use a Perspective Broker-based service to hand mail to users to a personal server using a UNIX domain socket, as well as to add some more conventional delivery methods, as scary as they may be.
Because the default configuration of Twisted Mail is to be an integrated POP3/SMTP servers, it is ideally suited for the so-called POP toaster configuration, where there are a multitude of virtual users and domains, all using the same IP address and computer to send and receive mails. It is fairly easy to configure Twisted as a POP toaster. There are a number of deployment choices: one can append a telnet server to the tap for remote configuration, or simple scripts can add and remove users from the user database. The user database is saved as a directory, where file names are keys and file contents are values, so concurrency is not usually a problem.
% mktap mail -d foobar.com=$HOME/Maildir/ -u postmaster=secret -b \ -p 110 -s 25 % twistd -f mail.tapBringing up a simple mail-server
Twisted's native mail storage format is Maildir, a format
that requires no locking and is safe and atomic. Twisted
supports a number of standardized extensions to Maildir,
commonly known as Maildir++. Most importantly, it supports
deletion as simply moving to a subfolder named
Trash, so mail is recoverable if accessed through
a protocol which allows multiple folders, like IMAP. However,
Twisted itself currently does not support any such protocol
yet.
Twisted was originally designed to support multi-player games; a simulated "real world" environment. Experience with game systems of that type is enlightening as to the nature of computing on the whole. Almost all services on a computer are modeled after some simulated real-world activity. For example, e-"mail", or "document publishing" on the web. Even "object-oriented" programming is based around the notion that data structures in a computer simulate some analogous real-world objects.
All such networked simulations have a few things in common. They each represent a service provided by software, and there is usually some object where "global" state is kept. Such a service must provide an authentication mechanism. Often, there is a representation of the authenticated user within the context of the simulation, and there are also objects aside from the user and the simulation itself that can be accessed.
For most existing protocols, Twisted provides these
abstractions through twisted.internet.passport.
This is so named because the most important common
functionality it provides is authentication. A simulation
"world" as described above -- such as an e-mail system,
document publishing archive, or online video game -- is
represented by subclass of Service, the
authentication mechanism by an Authorizer (which
is a set of Identities), and the user of the
simulation by a Perspective. Other objects in the
simulation may be represented by arbitrary python objects,
depending upon the implementation of the given protocol.
New problem domains, however, often require new protocols, and re-implementing these abstractions each time can be tedious, especially when it's not necessary. Many efforts have been made in recent years to create generic "remote object" or "remote procedure call" protocols, but in developing Twisted, these protocols were found to require too much overhead in development, be too inefficient at runtime, or both.
Perspective Broker is a new remote-object protocol designed
to be lightweight and impose minimal constraints upon the
development process and use Python's dynamic nature to good
effect, but still relatively efficient in terms of bandwidth
and CPU utilization. twisted.spread.pb serves as a
reference implementation of the protocol, but implementation of
Perspective Broker in other languages is already underway.
spread is the twisted subpackage
dealing with remote calls and objects, and has nothing to do
with the spread toolkit.
Perspective Broker extends
twisted.internet.passport's abstractions to be
concrete objects rather than design patterns. Rather than
having a Protocol implementation translate between
sequences of bytes and specifically named methods (as in the
other Twisted Protocols), Perspective Broker
defines a direct mapping between network messages and
quasi-arbitrary method calls.
In a server application where a large number of clients may
be interacting at once, it is not feasible to have an
arbitrarily large number of OS threads blocking and waiting for
remote method calls to return. Additionally, the ability for
any client to call any method of an object would present a
significant security risk. Therefore, rather than attempting to
provide a transparent interface to remote objects,
twisted.spread.pb is "translucent", meaning that
while remote method calls have different semantics than local
ones, the similarities in semantics are mirrored by
similarities in the syntax. Remote method calls impose as
little overhead as possible in terms of volume of code, but "as
little as possible" is unfortunately not "nothing".
twisted.spread.pb defines a method naming
standard for each type of remotely accessible object. For
example, if a client requests a method call with an expression
such as myPerspective.doThisAction(), the remote
version of myPerspective would be sent the message
perspective_doThisAction. Depending on the manner
in which an object is accessed, other method prefixes may be
observe_, view_, or
remote_. Any method present on a remotely
accessible object, and named appropriately, is considered to be
published -- since this is accomplished with
getattr, the definition of "present" is not just
limited to methods defined on the class, but instances may have
arbitrary callable objects associated with them as long as the
name is correct -- similarly to normal python objects.
Remote method calls are made on remote reference objects
(instances of pb.RemoteReference) by calling a
method with an appropriate name. However, that call will not
block -- if you need the result from a remote method call, you
pass in one of the two special keyword arguments to that method
-- pbcallback or pberrback.
pbcallback is a callable object which will be
called when the result is available, and pberrback
is a callable object which will be called if there was an
exception thrown either in transmission of the call or on the
remote side.
In the case that neither pberrback or
pbcallback is provided,
twisted.spread.pb will optimize network usage by
not sending confirmations of messages.
# Server Side class MyObject(pb.Referenceable): def remote_doIt(self): return "did it" # Client Side ... def myCallback(result): print result # result will be 'did it' def myErrback(stacktrace): print 'oh no, mr. bill!' print stacktrace myRemoteReference.doIt(pbcallback=myCallback, pberrback=myErrback)Listing 9: A remotely accessible object and accompanying call
Considering the problem of remote object access in terms of a simulation demonstrates a requirement for the knowledge of an actor with certain actions or requests. Often, when processing message, it is useful to know who sent it, since different results may be required depending on the permissions or state of the caller.
A simple example is a game where certain an object is invisible, but players with the "Heightened Perception" enchantment can see it. When answering the question "What objects are here?" it is important for the room to know who is asking, to determine which objects they can see. Parallels to the differences between "administrators" and "users" on an average multi-user system are obvious.
Perspective Broker is named for the fact that it does not
broker only objects, but views of objects. As a user of the
twisted.spread.pb module, it is quite easy to
determine the caller of a method. All you have to do is
subclass Viewable.
Before any arguments sent by the client, the actor (specifically, the Perspective instance through which this object was retrieved) will be passed as the first argument to any# Server Side class Greeter(pb.Viewable): def view_greet(self, actor): return "Hello %s!\n" % actor.perspectiveName # Client Side ... remoteGreeter.greet(pbcallback=sys.stdout.write) ...Listing 10: An object responding to its calling perspective
view_xxx methods.
In a simulation of any decent complexity, client and server will wish to share structured data. Perspective Broker provides a mechanism for both transferring (copying) and sharing (caching) that state.
Whenever an object is passed as an argument to or returned
from a remote method call, that object is serialized using
twisted.spread.jelly; a serializer similar in some
ways to Python's native pickle. Originally,
pickle itself was going to be used, but there were
several security issues with the pickle code as it
stands. It is on these issues of security that
pickle and twisted.spread.jelly part
ways.
While twisted.spread.jelly handles a few basic
types such as strings, lists, dictionaries and numbers
automatically, all user-defined types must be registered both
for serialization and unserialization. This registration
process is necessary on the sending side in order to determine
if a particular object is shared, and whether it is shared as
state or behavior. On the receiving end, it's necessary to
prevent arbitrary code from being run when an object is
unserialized -- a significant security hole in
pickle for networked applications.
On the sending side, the registration is accomplished by
making the object you want to serialize a subclass of one of
the "flavors" of object that are handled by Perspective Broker.
A class may be Referenceable,
Viewable, Copyable or
Cacheable. These four classes correspond to
different ways that the object will be seen remotely.
Serialization flavors are mutually exclusive -- these 4 classes
may not be mixed in with each other.
Referenceable: The remote side will refer to
this object directly. Methods with the prefix
remote_ will be callable on it. No state will be
transferred.Viewable: The remote side will refer to a
proxy for this object, which indicates what perspective
accessed this; as discussed above. Methods with the prefix
view_ will be callable on it, and have an
additional first argument inserted (the perspective that
called the method). No state will be transferred.Copyable: Each time this object is
serialized, its state will be copied and sent. No methods are
remotely callable on it. By default, the state sent will be
the instance's __dict__, but a method
getStateToCopyFor(perspective) may be defined
which returns an arbitrary serializable object for
state.Cacheable: The first time this object is
serialized, its state will be copied and sent. Each
subsequent time, however, a reference to the original object
will be sent to the receiver. No methods will be remotely
callable on this object. By default, again, the state sent
will be the instance's __dict__but a method
getStateToCacheAndObserveFor(perspective,
observer) may be defined to return alternative state.
Since the state for this object is only sent once, the
observer argument is an object representative of
the receiver's representation of the Cacheable
after unserialization -- method calls to this object will be
resolved to methods prefixed with observe_,
on the receiver's RemoteCache of this
object. This may be used to keep the receiver's cache
up-to-date as relevant portions of the Cacheable
object change.The previous samples of code have shown how an individual object will interact over a previously-established PB connection. In order to get to that connection, you need to do some set-up work on both the client and server side; PB attempts to minimize this effort.
There are two different approaches for setting up a PB server, depending on your application's needs. In the simplest case, where your application does not deal with the abstractions above -- services, identities, and perspectives -- you can simply publish an object on a particular port.
from twisted.spread import pb from twisted.internet import main class Echoer(pb.Root): def remote_echo(self, st): print 'echoing:', st return st if __name__ == '__main__': app = main.Application("pbsimple") app.listenOn(8789, pb.BrokerFactory(Echoer())) app.run()Listing 11: Creating a simple PB server
Listing 11 shows how to publish a simple object which responds to a single message, "echo", and returns whatever argument is sent to it. There is very little to explain: the "Echoer" class is a pb.Root, which is a small subclass of Referenceable designed to be used for objects published by a BrokerFactory, so Echoer follows the same rule for remote access that Referenceable does. Connecting to this service is almost equally simple.
from twisted.spread import pb from twisted.internet import main def gotObject(object): print "got object:",object object.echo("hello network", pbcallback=gotEcho) def gotEcho(echo): print 'server echoed:',echo main.shutDown() def gotNoObject(reason): print "no object:",reason main.shutDown() pb.getObjectAt("localhost", 8789, gotObject, gotNoObject, 30) main.run()Listing 12: A client for Echoer objects.
The utility function pb.getObjectAt retrieves
the root object from a hostname/port-number pair and makes a
callback (in this case, gotObject) if it can
connect and retrieve the object reference successfully, and an
error callback (gotNoObject) if it cannot connect
or the connection times out.
gotObject receives the remote reference, and
sends the echo message to it. This call is
visually noticeable as a remote method invocation by the
distinctive pbcallback keyword argument. When the
result from that call is received, gotEcho will be
called, notifying us that in fact, the server echoed our input
("hello network").
While this setup might be useful for certain simple types of applications where there is no notion of a "user", the additional complexity necessary for authentication and service segregation is worth it. In particular, re-use of server code for things like chat (twisted.words) is a lot easier with a unified notion of users and authentication.
from twisted.spread import pb from twisted.internet import main class SimplePerspective(pb.Perspective): def perspective_echo(self, text): print 'echoing',text return text class SimpleService(pb.Service): def getPerspectiveNamed(self, name): return SimplePerspective(name, self) if __name__ == '__main__': import pbecho app = main.Application("pbecho") pbecho.SimpleService("pbecho",app).getPerspectiveNamed("guest")\ .makeIdentity("guest") app.listenOn(pb.portno, pb.BrokerFactory(pb.AuthRoot(app))) app.save("start")Listing 13: A PB server using twisted's "passport" authentication.
In terms of the "functionality" it offers, this server is identical. It provides a method which will echo some simple object sent to it. However, this server provides it in a manner which will allow it to cooperate with multiple other authenticated services running on the same connection, because it uses the central Authorizer for the application.
On the line that creates the SimpleService,
several things happen.
Application instance.getPerspectiveNamed method.SimplePerspective has an
Identity generated for it, and persistently
added to the Application's
Authorizer. The created identity will have the
same name as the perspective ("guest"), and the password
supplied (also, "guest"). It will also have a reference to
the service "pbecho" and a perspective named "guest", by
name. The Perspective.makeIdentity utility
method prevents having to deal with the intricacies of the
passport Authorizer system when one doesn't
require strongly separate Identitys and
Perspectives.Also, this server does not run itself, but instead persists to a file which can be run with twistd, offering all the usual amenities of daemonization, logging, etc. Once the server is run, connecting to it is similar to the previous example.
from twisted.spread import pb from twisted.internet import main def success(message): print "Message received:",message main.shutDown() def failure(error): print "Failure...",error main.shutDown() def connected(perspective): perspective.echo("hello world", pbcallback=success, pberrback=failure) print "connected." pb.connect(connected, failure, "localhost", pb.portno, "guest", "guest", "pbecho", "guest", 30) main.run()Listing 14: Connecting to an Authorized Service
This introduces a new utility -- pb.connect.
This function takes a long list of arguments and manages the
handshaking and challenge/response aspects of connecting to a
PB service perspective, eventually calling back to indicate
either success or failure. In this particular example, we are
connecting to localhost on the default PB port (8787),
authenticating to the identity "guest" with the password
"guest", requesting the perspective "guest" from the service
"pbecho". If this can't be done within 30 seconds, the
connection will abort.
In these examples, I've attempted to show how Twisted makes
event-based scripting easier; this facilitates the ability to
run short scripts as part of a long-running process. However,
event-based programming is not natural to procedural scripts;
it is more generally accepted that GUI programs will be
event-driven whereas scripts will be blocking. An alternative
client to our SimpleService using GTK illustrates
the seamless meshing of Twisted and GTK.
from twisted.internet import main, ingtkernet from twisted.spread.ui import gtkutil import gtk ingtkernet.install() class EchoClient: def __init__(self, echoer): l.hide() self.echoer = echoer w = gtk.GtkWindow(gtk.WINDOW_TOPLEVEL) vb = gtk.GtkVBox(); b = gtk.GtkButton("Echo:") self.entry = gtk.GtkEntry(); self.outry = gtk.GtkEntry() w.add(vb) map(vb.add, [b, self.entry, self.outry]) b.connect('clicked', self.clicked) w.connect('destroy', gtk.mainquit) w.show_all() def clicked(self, b): txt = self.entry.get_text() self.entry.set_text("") self.echoer.echo(txt, pbcallback=self.outry.set_text) l = gtkutil.Login(EchoClient, None, initialService="pbecho") l.show_all() gtk.mainloop()Listing 15: A Twisted GUI application
Although PB will be interesting to those people who wish to
write custom clients for their networked applications, many
prefer or require a web-based front end. Twisted's built-in web
server has been designed to accommodate this desire, and the
presentation framework that one would use to write such an
application is twisted.web.widgets. Web.Widgets
has been designed to work in an event-based manner, without
adding overhead to the designer or the developer's
work-flow.
Surprisingly, asynchronous web interfaces fit very well into the normal uses of purpose-built web toolkits such as PHP. Any experienced PHP, Zope, or WebWare developer will tell you that separation of presentation, content, and logic is very important. In practice, this results in a "header" block of code which sets up various functions which are called throughout the page, some of which load blocks of content to display. While PHP does not enforce this, it is certainly idiomatic. Zope enforces it to a limited degree, although it still allows control structures and other programmatic elements in the body of the content.
In Web.Widgets, strict enforcement of this principle coincides very neatly with a "hands-free" event-based integration, where much of the work of declaring callbacks is implicit. A "Presentation" has a very simple structure for evaluating Python expressions and giving them a context to operate in. The "header" block which is common to many templating systems becomes a class, which represents an enumeration of events that the template may generate, each of which may be responded to either immediately or latently.
For the sake of simplicity, as well as maintaining
compatibility for potential document formats other than HTML,
Presentation widgets do not attempt to parse their template as
HTML tags. The structure of the template is "HTML Text
%%%%python_expression()%%%% more HTML Text". Every set
of 4 percent signs (%%%%) switches back and forth between
evaluation and printing.
No control structures are allowed in the template. This was originally thought to be a potentially major inconvenience, but with use of the Web.Widgets code to develop a few small sites, it has seemed trivial to encapsulate any table-formatting code within a method; especially since those methods can take string arguments if there's a need to customize the table's appearance.
The namespace for evaluating the template expressions is obtained by scanning the class hierarchy for attributes, and getting each of those attributes from the current instance. This means that all methods will be bound methods, so indicating "self" explicitly is not required. While it is possible to override the method for creating namespaces, using this default has the effect of associating all presentation code for a particular widget in one class, along with its template. If one is working with a non-programmer designer, and the template is in an external file, it is always very clear to the designer what functionality is available to them in any given scope, because there is a list of available methods for any given class.
A convenient event to register for would be a response from
the PB service that we just implemented. We can use the
Deferred class in order to indicate to the widgets
framework that certain work has to be done later. This is a
Twisted convention which one can currently use in PB as well as
webwidgets; any framework which needs the ability to defer a
return value until later should use this facility. Elements of
the page will be rendered from top to bottom as data becomes
available, so the page will not be blocked on rendering until
all deferred elements have been completed.
from twisted.spread import pb from twisted.python import defer from twisted.web import widgets class EchoDisplay(widgets.Presentation): template = """<H1>Welcome to my widget, displaying %%%%echotext%%%%.</h1> <p>Here it is: %%%%getEchoPerspective()%%%%</p>""" echotext = 'hello web!' def getEchoPerspective(self): d = defer.Deferred() pb.connect(d.callback, d.errback, "localhost", pb.portno, "guest", "guest", "pbecho", "guest", 1) d.addCallbacks(self.makeListOf, self.formatTraceback) return ['<b>',d,'</b>'] def makeListOf(self, echoer): d = defer.Deferred() echoer.echo(self.echotext, pbcallback=d.callback, pberrback=d.errback) d.addCallbacks(widgets.listify, self.formatTraceback) return [d] if __name__ == "__main__": from twisted.web import server from twisted.internet import main a = main.Application("pbweb") gdgt = widgets.Gadget() gdgt.widgets['index'] = EchoDisplay() a.listenOn(8080, server.Site(gdgt)) a.run()Listing 16: an event-based web widget.
Each time a Deferred is returned as part of the page, the
page will pause rendering until the deferred's
callback method is invoked. When that callback is
made, it is inserted at the point in the page where rendering
left off.
If necessary, there are options within web.widgets to allow a widget to postpone or cease rendering of the entire page -- for example, it is possible to write a FileDownload widget, which will override the rendering of the entire page and replace it with a file download.
The final goal of web.widgets is to provide a framework which encourages the development of usable library code. Too much web-based code is thrown away due to its particular environment requirements or stylistic preconceptions it carries with it. The goal is to combine the fast-and-loose iterative development cycle of PHP with the ease of installation and use of Zope's "Product" plugins.
It is unfortunately well beyond the scope of this paper to cover all the functionality that Twisted provides, but it serves as a good overview. It may seem as though twisted does anything and everything, but there are certain features we never plan to implement because they are simply outside the scope of the project.
Despite the multiple ways to publish and access objects, Twisted does not have or support an interface definition language. Some developers on the Twisted project have experience with remote object interfaces that require explicit specification of all datatypes during the design of an object's interface. We feel that such interfaces are in the spirit of statically-typed languages, and are therefore suited to the domain of problems where statically-typed languages excel. Twisted has no plans to implement a protocol schema or static type-checking mechanism, as the efficiency gained by such an approach would be quickly lost again by requiring the type conversion between Python's dynamic types and the protocol's static ones. Since one of the key advantages of Python is its extremely flexible dynamic type system, we felt that a dynamically typed approach to protocol design would share some of those advantages.
Twisted does not assume that all data is stored in a
relational database, or even an efficient object database.
Currently, Twisted's configuration state is all stored in
memory at run-time, and the persistent parts of it are pickled
at one go. There are no plans to move the configuration objects
into a "real" database, as we feel it is easier to keep a naive
form of persistence for the default case and let
application-specific persistence mechanisms handle persistence.
Consequently, there is no object-relational mapping in Twisted;
twisted.enterprise is an interface to the
relational paradigm, not an object-oriented layer over it.
There are other things that Twisted will not do as well, but these have been frequently discussed as possibilities for it. The general rule of thumb is that if something will increase the required installation overhead, then Twisted will probably not do it. Optional additions that enhance integration with external systems are always welcome: for example, database drivers for Twisted or a CORBA IDL for PB objects.
Twisted is still a work in progress. The number of protocols in the world is infinite for all practical purposes, and it would be nice to have a central repository of event-based protocol implementations. Better integration with frameworks and operating systems is also a goal. Examples for integration opportunities are automatic creation of installer for "tap" files (for Red Hat Packager-based distributions, FreeBSD's package management system or Microsoft Windows(tm) installers), and integration with other event-dispatch mechanisms, such as win32's native message dispatch.
A still-nascent feature of Twisted, which this paper only
touches briefly upon, is twisted.enterprise: it is
planned that Twisted will have first-class database support
some time in the near future. In particular, integration
between twisted.web and twisted.enterprise to allow developers
to have SQL conveniences that they are used to from other
frameworks.
Another direction that we hope Twisted will progress in is standardization and porting of PB as a messaging protocol. Some progress has already been made in that direction, with XEmacs integration nearly ready for release as of this writing.
Tighter integration of protocols is also a future goal, such an FTP server that can serve the same resources as a web server, or a web server that allows users to change their POP3 password. While Twisted is already a very tightly integrated framework, there is always room for more integration. Of course, all this should be done in a flexible way, so the end-user will choose which components to use -- and have those components work well together.
As shown, Twisted provides a lot of functionality to the
Python network programmer, while trying to be in his way as
little as possible. Twisted gives good tools for both someone
trying to implement a new protocol, or someone trying to use an
existing protocol. Twisted allows developers to prototype and
develop object communication models with PB, without designing
a byte-level protocol. Twisted tries to have an easy way to
record useful deployment options, via the
twisted.tap and plugin mechanisms, while making it
easy to generate new forms of deployment. And last but not
least, even Twisted is written in a high-level language and
uses its dynamic facilities to give an easy API, it has
performance which is good enough for most situations -- for
example, the web server can easily saturate a T1 line serving
dynamic requests on low-end machines.
While still an active project, Twisted can already used for production programs. Twisted can be downloaded from the main Twisted site (http://www.twistedmatrix.com) where there is also documentation for using and programming Twisted.
We wish to thank Sean Riley, Allen Short, Chris Armstrong, Paul Swartz, Jürgen Hermann, Benjamin Bruheim, Travis B. Hartwell, and Itamar Shtull-Trauring for being a part of the Twisted development team with us.
Thanks also to Jason Asbahr, Tommi Virtanen, Gavin Cooper, Erno Kuusela, Nick Moffit, Jeremy Fincher, Jerry Hebert, Keith Zaback, Matthew Walker, and Dan Moniz, for providing insight, commentary, bandwidth, crazy ideas, and bug-fixes (in no particular order) to the Twisted team.