# Twisted from Scratch, or The Evolution of Finger

## Introduction

Twisted is a big system. People are often daunted when they approach it. It's hard to know where to start looking.

This guide builds a full-fledged Twisted application from the ground up, using most of the important bits of the framework. There is a lot of code, but don't be afraid.

The application we are looking at is a finger service, along the lines of the familiar service traditionally provided by UNIX™ servers. We will extend this service slightly beyond the standard, in order to demonstrate some of Twisted's higher-level features.

## Refuse Connections

    from twisted.internet import reactor
    reactor.run()

Source listing - listings/finger/finger01.py

This example only runs the reactor. Nothing at all will happen until we interrupt the program. It will consume almost no CPU resources. Not very useful, perhaps -- but this is the skeleton inside which the Twisted program will grow.

### The Reactor

You don't call Twisted, Twisted calls you. The `reactor` is Twisted's main event loop. There is exactly one reactor in any running Twisted application. Once started it loops over and over again, responding to network events, and making scheduled calls to code.

## Do Nothing

    from twisted.internet import protocol, reactor
    class FingerProtocol(protocol.Protocol):
        pass
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
    reactor.listenTCP(1079, FingerFactory())
    reactor.run()

Source listing - listings/finger/finger02.py

Here, we start listening on port 1079. The 1079 is a reminder that eventually, we want to run on port 79, the standard port for finger servers. We define a protocol which does not respond to any events. Thus, connections to 1079 will be accepted, but the input ignored.

## Drop Connections

    from twisted.internet import protocol, reactor
    class FingerProtocol(protocol.Protocol):
        def connectionMade(self):
            self.transport.loseConnection()
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
    reactor.listenTCP(1079, FingerFactory())
    reactor.run()

Source listing - listings/finger/finger03.py

Here we add to the protocol the ability to respond to the event of beginning a connection -- by terminating it. Perhaps not an interesting behavior, but it is already close to behaving according to the letter of the protocol. After all, there is no requirement to send any data to the remote connection in the standard. The only problem, as far as the standard is concerned, is that we terminate the connection too soon. A client which is slow enough will see his send() of the username result in an error.

## Read Username, Drop Connections

    from twisted.internet import protocol, reactor
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.transport.loseConnection()
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
    reactor.listenTCP(1079, FingerFactory())
    reactor.run()

Source listing - listings/finger/finger04.py

Here we make `FingerProtocol` inherit from `LineReceiver`, so that we get data-based events on a line-by-line basis. We respond to the event of receiving the line with shutting down the connection.

Congratulations, this is the first standard-compliant version of the code. However, usually people actually expect some data about users to be transmitted.

## Read Username, Output Error, Drop Connections

    from twisted.internet import protocol, reactor
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.transport.write("No such user\r\n")
            self.transport.loseConnection()
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
    reactor.listenTCP(1079, FingerFactory())
    reactor.run()

Source listing - listings/finger/finger05.py

Finally, a useful version. Granted, the usefulness is somewhat limited by the fact that this version only prints out a No such user message. It could be used for devastating effect in honey-pots, of course.

## Output From Empty Factory

    # Read username, output from empty factory, drop connections
    from twisted.internet import protocol, reactor
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.transport.write(self.factory.getUser(user)+"\r\n")
            self.transport.loseConnection()
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
        def getUser(self, user): return "No such user"
    reactor.listenTCP(1079, FingerFactory())
    reactor.run()

Source listing - listings/finger/finger06.py

The same behavior, but finally we see what usefulness the factory has: as something that does not get constructed for every connection, it can be in charge of the user database. In particular, we won't have to change the protocol if the user database back-end changes.

## Output from Non-empty Factory

    # Read username, output from non-empty factory, drop connections
    from twisted.internet import protocol, reactor
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.transport.write(self.factory.getUser(user)+"\r\n")
            self.transport.loseConnection()
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
        def __init__(self, **kwargs): self.users = kwargs
        def getUser(self, user):
            return self.users.get(user, "No such user")
    reactor.listenTCP(1079, FingerFactory(moshez='Happy and well'))
    reactor.run()

Source listing - listings/finger/finger07.py

Finally, a really useful finger database. While it does not supply information about logged in users, it could be used to distribute things like office locations and internal office numbers. As hinted above, the factory is in charge of keeping the user database: note that the protocol instance has not changed. This is starting to look good: we really won't have to keep tweaking our protocol.

## Use Deferreds

    # Read username, output from non-empty factory, drop connections
    # Use deferreds, to minimize synchronicity assumptions
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user
            ).addErrback(lambda _: "Internal error in server"
            ).addCallback(lambda m:
                (self.transport.write(m+"\r\n"),
                 self.transport.loseConnection()))
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
        def __init__(self, **kwargs): self.users = kwargs
        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))
    reactor.listenTCP(1079, FingerFactory(moshez='Happy and well'))
    reactor.run()

Source listing - listings/finger/finger08.py

But, here we tweak it just for the hell of it. Yes, while the previous version worked, it did assume the result of getUser is always immediately available. But what if instead of an in memory database, we would have to fetch result from a remote Oracle? Or from the web? Or, or...

## Run 'finger' Locally

    # Read username, output from factory interfacing to OS, drop connections
    from twisted.internet import protocol, reactor, defer, utils
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user
            ).addErrback(lambda _: "Internal error in server"
            ).addCallback(lambda m:
                (self.transport.write(m+"\r\n"),
                 self.transport.loseConnection()))
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
        def getUser(self, user):
            return utils.getProcessOutput("finger", [user])
    reactor.listenTCP(1079, FingerFactory())
    reactor.run()

Source listing - listings/finger/finger09.py

...from running a local command? Yes, this version runs finger locally with whatever arguments it is given, and returns the standard output. This is probably insecure, so you probably don't want a real server to do this without a lot more validation of the user input. This will do exactly what the standard version of the finger server does.

## Read Status from the Web

The web. That invention which has infiltrated homes around the world finally gets through to our invention. Here we use the built-in Twisted web client, which also returns a deferred. Finally, we manage to have examples of three different database back-ends, which do not change the protocol class. In fact, we will not have to change the protocol again until the end of this tutorial: we have achieved, here, one truly usable class.

    # Read username, output from factory interfacing to web, drop connections
    from twisted.internet import protocol, reactor, defer, utils
    from twisted.protocols import basic
    from twisted.web import client
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user
            ).addErrback(lambda _: "Internal error in server"
            ).addCallback(lambda m:
                (self.transport.write(m+"\r\n"),
                 self.transport.loseConnection()))
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
        def __init__(self, prefix): self.prefix=prefix
        def getUser(self, user):
            return client.getPage(self.prefix+user)
    reactor.listenTCP(1079, FingerFactory(prefix='http://livejournal.com/~'))
    reactor.run()

Source listing - listings/finger/finger10.py

## Use Application

Up until now, we faked. We kept using port 1079, because really, who wants to run a finger server with root privileges? Well, the common solution is privilege shedding: after binding to the network, become a different, less privileged user. We could have done it ourselves, but Twisted has a built-in way to do it. Create a snippet as above, defining an application object. That object will have uid and gid attributes. When running it (later we will see how) it will bind to ports, shed privileges and then run.

    # Read username, output from non-empty factory, drop connections
    # Use deferreds, to minimize synchronicity assumptions
    # Write application. Save in 'finger.tpy'
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user
            ).addErrback(lambda _: "Internal error in server"
            ).addCallback(lambda m:
                (self.transport.write(m+"\r\n"),
                 self.transport.loseConnection()))
    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
        def __init__(self, **kwargs): self.users = kwargs
        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

    application = service.Application('finger', uid=1, gid=1)
    factory = FingerFactory(moshez='Happy and well')
    internet.TCPServer(79, factory).setServiceParent(
        service.IServiceCollection(application))

Source listing - listings/finger/finger11.py

## twistd

This is how to run Twisted Applications -- files which define an 'application'. twistd (TWISTed Daemonizer) does everything a daemon can be expected to -- shuts down stdin/stdout/stderr, disconnects from the terminal and can even change runtime directory, or even the root filesystems. In short, it does everything so the Twisted application developer can concentrate on writing his networking code.

    root% twistd -ny finger.tac # just like before
    root% twistd -ny finger.tac  # also works
    root% twistd -ny finger.tac # usually a .tac is more of an app
    root% twistd -y finger.tac # daemonize, keep pid in twistd.pid
    root% twistd -y finger.tac --pidfile=finger.pid
    root% twistd -y finger.tac --rundir=/
    root% twistd -y finger.tac --chroot=/var
    root% twistd -y finger.tac -l /var/log/finger.log
    root% twistd -y finger.tac --syslog # just log to syslog
    root% twistd -y finger.tac --syslog --prefix=twistedfinger # use given prefix

## Setting Message By Local Users

Now that port 1079 is free, maybe we can run on it a different server, one which will let people set their messages. It does no access control, so anyone who can login to the machine can set any message. We assume this is the desired behavior in our case. Testing it can be done by simply:

    % nc localhost 1079   # or telnet localhost 1079
    moshez
    Giving a tutorial now, sorry!
    ^D

    # But let's try and fix setting away messages, shall we?
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user).addErrback(
                lambda _:"Internal error in server"
                ).addCallback(
                lambda m: (self.transport.write(m+"\r\n"),
                           self.transport.loseConnection()))

    class FingerFactory(protocol.ServerFactory):
        protocol = FingerProtocol
        def __init__(self, **kwargs): self.users = kwargs
        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

    class FingerSetterProtocol(basic.LineReceiver):
        def connectionMade(self): self.lines = []
        def lineReceived(self, line): self.lines.append(line)
        def connectionLost(self, reason):
            self.factory.setUser(*self.lines[:2])
            # first line: user    second line: status

    class FingerSetterFactory(protocol.ServerFactory):
        protocol = FingerSetterProtocol
        def __init__(self, ff): self.setUser = ff.users.__setitem__

    ff = FingerFactory(moshez='Happy and well')
    fsf = FingerSetterFactory(ff)

    application = service.Application('finger', uid=1, gid=1)
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79,ff).setServiceParent(serviceCollection)
    internet.TCPServer(1079,fsf).setServiceParent(serviceCollection)

Source listing - listings/finger/finger12.py

## Use Services to Make Dependencies Sane

The previous version had the setter poke at the innards of the finger factory. It's usually not a good idea: this version makes both factories symmetric by making them both look at a single object. Services are useful for when an object is needed which is not related to a specific network server. Here, we moved all responsibility for manufacturing factories into the service. Note that we stopped subclassing: the service simply puts useful methods and attributes inside the factories. We are getting better at protocol design: none of our protocol classes had to be changed, and neither will have to change until the end of the tutorial.

    # Fix asymmetry
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user
            ).addErrback(lambda _: "Internal error in server"
            ).addCallback(lambda m:
                          (self.transport.write(m+"\r\n"),
                           self.transport.loseConnection()))

    class FingerSetterProtocol(basic.LineReceiver):
        def connectionMade(self): self.lines = []
        def lineReceived(self, line): self.lines.append(line)
        def connectionLost(self,reason): self.factory.setUser(*self.lines[:2])
        # first line: user   second line: status

    class FingerService(service.Service):
        def __init__(self, *args, **kwargs):
            self.parent.__init__(self, *args)
            self.users = kwargs
        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))
        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol, f.getUser = FingerProtocol, self.getUser
            return f
        def getFingerSetterFactory(self):
            f = protocol.ServerFactory()
            f.protocol, f.setUser = FingerSetterProtocol, self.users.__setitem__
            return f

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('finger', moshez='Happy and well')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79,f.getFingerFactory()
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(1079,f.getFingerSetterFactory()
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger13.py

## Read Status File

This version shows how, instead of just letting users set their messages, we can read those from a centrally managed file. We cache results, and every 30 seconds we refresh it. Services are useful for such scheduled tasks.

    moshez: happy and well
    shawn: alive

sample /etc/users file - listings/finger/etc.users

    # Read from file
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user
            ).addErrback(lambda _: "Internal error in server"
            ).addCallback(lambda m:
                          (self.transport.write(m+"\r\n"),
                           self.transport.loseConnection()))

    class FingerService(service.Service):
        def __init__(self, filename):
            self.users = {}
            self.filename = filename
            self._read()
        def _read(self):
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user=user.strip()
                status=status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)
        def startService(self):
            def _read(self):
                for line in file(self.file):
                    user, status = line.split(':', 1)
                    user=user.strip()
                    status=status.strip()
                    self.users[user] = status
                self.call = reactor.callLater(30, _read)
            _read()
            service.Service.startService(self)
        def stopService(self):
            service.Service.stopService(self)
            self.call.cancel()
        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))
        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol, f.getUser = FingerProtocol, self.getUser,
            f.startService = self.startService
            return f

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    finger = internet.TCPServer(79, f.getFingerFactory())
    finger.setServiceParent(service.IServiceCollection(application))

Source listing - listings/finger/finger14.py

## Announce on Web, Too

The same kind of service can also produce things useful for other protocols. For example, in twisted.web, the factory itself (the site) is almost never subclassed -- instead, it is given a resource, which represents the tree of resources available via URLs. That hierarchy is navigated by site, and overriding it dynamically is possible with getChild.

    # Read from file, announce on the web!
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic
    from twisted.web import resource, server, static
    import cgi

    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user
            ).addErrback(lambda _: "Internal error in server"
            ).addCallback(lambda m:
             (self.transport.write(m+"\r\n"),self.transport.loseConnection()))

    class FingerService(service.Service):
        def __init__(self, file):
            self.file = file
            self._read()
        def _read(self):
            self.users = {}
            for line in file(self.file):
                user, status = line.split(':', 1)
                user=user.strip()
                status=status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)
        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))
        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol, f.getUser = FingerProtocol, self.getUser
            f.startService = self.startService
            return f
        def getResource(self):
            r = resource.Resource()
            r.getChild = (lambda path, request:
                          static.Data('<h1>%s</h1><p>%s</p>' %
                          tuple(map(cgi.escape,
                          [path,self.users.get(path,
                          "No such user.<p/>http://this.site/user")])),
                          'text/html'))
            return r

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79,f.getFingerFactory()
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000,server.Site(f.getResource())
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger15.py

## Announce on IRC, Too

This is the first time there is client code. IRC clients often act a lot like servers: responding to events from the network. The reconnecting client factory will make sure that severed links will get re-established, with intelligent tweaked exponential back-off algorithms. The IRC client itself is simple: the only real hack is getting the nickname from the factory in connectionMade.

    # Read from file, announce on the web, irc
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic, irc
    from twisted.web import resource, server, static
    import cgi
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user
            ).addErrback(lambda _: "Internal error in server"
            ).addCallback(lambda m:
             (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
    class FingerSetterProtocol(basic.LineReceiver):
        def connectionMade(self): self.lines = []
        def lineReceived(self, line): self.lines.append(line)
        def connectionLost(self,reason): self.factory.setUser(*self.lines[:2])
    class IRCReplyBot(irc.IRCClient):
        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)
        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                self.factory.getUser(msg
                ).addErrback(lambda _: "Internal error in server"
                ).addCallback(lambda m: irc.IRCClient.msg(self, user, msg+': '+m))

    class FingerService(service.Service):
        def __init__(self, file):
            self.file = file
            self._read()
        def _read(self):
            self.users = {}
            for line in file(self.file):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)
        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))
        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol, f.getUser = FingerProtocol, self.getUser
            return f
        def getResource(self):
            r = resource.Resource()
            r.getChild = (lambda path, request:
                          static.Data('<h1>%s</h1><p>%s</p>' %
                          tuple(map(cgi.escape,
                          [path,self.users.get(path,
                          "No such user <p/> usage: site/user")])),
                          'text/html'))
            return r

        def getIRCBot(self, nickname):
            f = protocol.ReconnectingClientFactory()
            f.protocol,f.nickname,f.getUser = IRCReplyBot,nickname,self.getUser
            return f

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79,f.getFingerFactory()
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000,server.Site(f.getResource())
                       ).setServiceParent(serviceCollection)
    internet.TCPClient('irc.freenode.org', 6667, f.getIRCBot('fingerbot')
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger16.py

## Add XML-RPC Support

In Twisted, XML-RPC support is handled just as though it was another resource. That resource will still support GET calls normally through render(), but that is usually left unimplemented. Note that it is possible to return deferreds from XML-RPC methods. The client, of course, will not get the answer until the deferred is triggered.

    # Read from file, announce on the web, irc, xml-rpc
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic, irc
    from twisted.web import resource, server, static, xmlrpc
    import cgi
    class FingerProtocol(basic.LineReceiver):
        def lineReceived(self, user):
            self.factory.getUser(user
             ).addErrback(lambda _: "Internal error in server"
             ).addCallback(lambda m:
              (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
    class FingerSetterProtocol(basic.LineReceiver):
        def connectionMade(self): self.lines = []
        def lineReceived(self, line): self.lines.append(line)
        def connectionLost(self,reason): self.factory.setUser(*self.lines[:2])
    class IRCReplyBot(irc.IRCClient):
        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)
        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                self.factory.getUser(msg
                ).addErrback(lambda _: "Internal error in server"
                ).addCallback(lambda m: irc.IRCClient.msg(self, user, msg+': '+m))
    class FingerService(service.Service):
        def __init__(self, filename):
            self.filename = filename
            self._read()
        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)
        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))
        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol, f.getUser = FingerProtocol, self.getUser
            return f
        def getResource(self):
            r = resource.Resource()
            r.getChild = (lambda path, request:
                          static.Data('<h1>%s</h1><p>%s</p>' %
                          tuple(map(cgi.escape,
                          [path,self.users.get(path, "No such user")])),
                          'text/html'))
            x = xmlrpc.XMLRPC()
            x.xmlrpc_getUser = self.getUser
            r.putChild('RPC2', x)
            return r
        def getIRCBot(self, nickname):
            f = protocol.ReconnectingClientFactory()
            f.protocol,f.nickname,f.getUser = IRCReplyBot,nickname,self.getUser
            return f

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, f.getFingerFactory()
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(f.getResource())
                       ).setServiceParent(serviceCollection)
    internet.TCPClient('irc.freenode.org', 6667, f.getIRCBot('fingerbot')
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger17.py

A simple client to test the XMLRPC finger:

    # testing xmlrpc finger

    import xmlrpclib
    server = xmlrpclib.Server('http://127.0.0.1:8000/RPC2')
    print server.getUser('moshez')

Source listing - listings/finger/fingerXRclient.py

## Write Readable Code

The last version of the application had a lot of hacks. We avoided sub-classing, didn't support things like user listings over the web, and removed all blank lines -- all in the interest of code which is shorter. Here we take a step back, subclass what is more naturally a subclass, make things which should take multiple lines take them, etc. This shows a much better style of developing Twisted applications, though the hacks in the previous stages are sometimes used in throw-away prototypes.

    # Do everything properly
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic, irc
    from twisted.web import resource, server, static, xmlrpc
    import cgi

    def catchError(err):
        return "Internal error in server"

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value+'\n')
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class FingerSetterProtocol(basic.LineReceiver):

        def connectionMade(self):
            self.lines = []

        def lineReceived(self, line):
            self.lines.append(line)

        def connectionLost(self,reason):
            self.factory.setUser(*self.lines[:2])

    class IRCReplyBot(irc.IRCClient):

        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)
                d.addErrback(catchError)
                d.addCallback(lambda m: "Status of %s: %s" % (msg, m))
                d.addCallback(lambda m: self.msg(user,m))

    class UserStatusTree(resource.Resource):
        def __init__(self, service):
            resource.Resource.__init__(self)
            self.service = service

        def render(self, request):
            d = self.service.getUsers()
            def formatUsers(users):
                l = ['<li><a href="%s">%s</a></li>' % (user, user)
                     for user in users]
                return '<ul>'+''.join(l)+'</ul>'
            d.addCallback(formatUsers)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

        def getChild(self, path, request):
            if path=="":
                return UserStatusTree(self.service)
            else:
                return UserStatus(path, self.service)

    class UserStatus(resource.Resource):

        def __init__(self, user, service):
            resource.Resource.__init__(self)
            self.user = user
            self.service = service

        def render(self, request):
            d = self.service.getUser(self.user)
            d.addCallback(cgi.escape)
            d.addCallback(lambda m:
                          '<h1>%s</h1>'%self.user+'<p>%s</p>'%m)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

    class UserStatusXR(xmlrpc.XMLRPC):

        def __init__(self, service):
            xmlrpc.XMLRPC.__init__(self)
            self.service = service

        def xmlrpc_getUser(self, user):
            return self.service.getUser(user)

    class FingerService(service.Service):

        def __init__(self, filename):
            self.filename = filename
            self._read()

        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

        def getFingerFactory(self):
            f = protocol.ServerFactory()
            f.protocol = FingerProtocol
            f.getUser = self.getUser
            return f

        def getResource(self):
            r = UserStatusTree(self)
            x = UserStatusXR(self)
            r.putChild('RPC2', x)
            return r

        def getIRCBot(self, nickname):
            f = protocol.ReconnectingClientFactory()
            f.protocol = IRCReplyBot
            f.nickname = nickname
            f.getUser = self.getUser
            return f

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, f.getFingerFactory()
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(f.getResource())
                       ).setServiceParent(serviceCollection)
    internet.TCPClient('irc.freenode.org', 6667, f.getIRCBot('fingerbot')
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger18.py

## Write Maintainable Code

In the last version, the service class was three times longer than any other class, and was hard to understand. This was because it turned out to have multiple responsibilities. It had to know how to access user information, by rereading the file every half minute, but also how to display itself in a myriad of protocols. Here, we used the component-based architecture that Twisted provides to achieve a separation of concerns. All the service is responsible for, now, is supporting getUser/getUsers. It declares its support via the __implements__ keyword. Then, adapters are used to make this service look like an appropriate class for various things: for supplying a finger factory to TCPServer, for supplying a resource to site's constructor, and to provide an IRC client factory for TCPClient. All the adapters use are the methods in FingerService they are declared to use: getUser/getUsers. We could, of course, skip the interfaces and let the configuration code use things like FingerFactoryFromService(f) directly. However, using interfaces provides the same flexibility inheritance gives: future subclasses can override the adapters.

    # Do everything properly, and componentize
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic, irc
    from twisted.python import components
    from twisted.web import resource, server, static, xmlrpc
    import cgi

    class IFingerService(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def getUsers(self):
            """Return a deferred returning a list of strings"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    def catchError(err):
        return "Internal error in server"

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value+'\n')
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class IFingerFactory(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerFactory,

        protocol = FingerProtocol

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(FingerFactoryFromService,
                               IFingerService,
                               IFingerFactory)

    class FingerSetterProtocol(basic.LineReceiver):

          def connectionMade(self):
              self.lines = []

          def lineReceived(self, line):
              self.lines.append(line)

          def connectionLost(self,reason):
              if len(self.lines) == 2:
                  self.factory.setUser(*self.lines)

    class IFingerSetterFactory(components.Interface):

        def setUser(self, user, status):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerSetterFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerSetterFactory,

        protocol = FingerSetterProtocol

        def __init__(self, service):
            self.service = service

        def setUser(self, user, status):
            self.service.setUser(user, status)

    components.registerAdapter(FingerSetterFactoryFromService,
                               IFingerSetterService,
                               IFingerSetterFactory)

    class IRCReplyBot(irc.IRCClient):

        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)
                d.addErrback(catchError)
                d.addCallback(lambda m: "Status of %s: %s" % (msg, m))
                d.addCallback(lambda m: self.msg(user, m))

    class IIRCClientFactory(components.Interface):

        '''
        @ivar nickname
        '''

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol"""

    class IRCClientFactoryFromService(protocol.ClientFactory):

        __implements__ = IIRCClientFactory,

        protocol = IRCReplyBot
        nickname = None

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(IRCClientFactoryFromService, IFingerService,
                               IIRCClientFactory)

    class UserStatusTree(resource.Resource):

        __implements__ = resource.IResource,

        def __init__(self, service):
            resource.Resource.__init__(self)
            self.service = service
            self.putChild('RPC2', UserStatusXR(self.service))

        def render(self, request):
            d = self.service.getUsers()
            def formatUsers(users):
                l = ['<li><a href="%s">%s</a></li>' % (user, user)
                     for user in users]
                return '<ul>'+''.join(l)+'</ul>'
            d.addCallback(formatUsers)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

        def getChild(self, path, request):
            if path=="":
                return UserStatusTree(self.service)
            else:
                return UserStatus(path, self.service)

    components.registerAdapter(UserStatusTree, IFingerService,
                               resource.IResource)

    class UserStatus(resource.Resource):

        def __init__(self, user, service):
            resource.Resource.__init__(self)
            self.user = user
            self.service = service

        def render(self, request):
            d = self.service.getUser(self.user)
            d.addCallback(cgi.escape)
            d.addCallback(lambda m:
                          '<h1>%s</h1>'%self.user+'<p>%s</p>'%m)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

    class UserStatusXR(xmlrpc.XMLRPC):

        def __init__(self, service):
            xmlrpc.XMLRPC.__init__(self)
            self.service = service

        def xmlrpc_getUser(self, user):
            return self.service.getUser(user)

    class FingerService(service.Service):

        __implements__ = IFingerService,

        def __init__(self, filename):
            self.filename = filename
            self._read()

        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, IFingerFactory(f)
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(resource.IResource(f))
                       ).setServiceParent(serviceCollection)
    i = IIRCClientFactory(f)
    i.nickname = 'fingerbot'
    internet.TCPClient('irc.freenode.org', 6667, i
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger19.py

## Advantages of Latest Version

- Readable -- each class is short
- Maintainable -- each class knows only about interfaces
- Dependencies between code parts are minimized
- Example: writing a new IFingerService is easy

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    # Advantages of latest version

    class MemoryFingerService(service.Service):

        __implements__ = IFingerService, IFingerSetterService

        def __init__(self, **kwargs):
            self.users = kwargs

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

        def setUser(self, user, status):
            self.users[user] = status

    f = MemoryFingerService(moshez='Happy and well')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(1079, IFingerSetterFactory(f), interface='127.0.0.1'
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger19a_changes.py

Full source code here:

    # Do everything properly, and componentize
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic, irc
    from twisted.python import components
    from twisted.web import resource, server, static, xmlrpc
    import cgi

    class IFingerService(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def getUsers(self):
            """Return a deferred returning a list of strings"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    def catchError(err):
        return "Internal error in server"

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value+'\n')
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class IFingerFactory(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerFactory,

        protocol = FingerProtocol

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(FingerFactoryFromService,
                               IFingerService,
                               IFingerFactory)

    class FingerSetterProtocol(basic.LineReceiver):

          def connectionMade(self):
              self.lines = []

          def lineReceived(self, line):
              self.lines.append(line)

          def connectionLost(self,reason):
              if len(self.lines) == 2:
                  self.factory.setUser(*self.lines)

    class IFingerSetterFactory(components.Interface):

        def setUser(self, user, status):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerSetterFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerSetterFactory,

        protocol = FingerSetterProtocol

        def __init__(self, service):
            self.service = service

        def setUser(self, user, status):
            self.service.setUser(user, status)

    components.registerAdapter(FingerSetterFactoryFromService,
                               IFingerSetterService,
                               IFingerSetterFactory)

    class IRCReplyBot(irc.IRCClient):

        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)
                d.addErrback(catchError)
                d.addCallback(lambda m: "Status of %s: %s" % (msg, m))
                d.addCallback(lambda m: self.msg(user, m))

    class IIRCClientFactory(components.Interface):

        '''
        @ivar nickname
        '''

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol"""

    class IRCClientFactoryFromService(protocol.ClientFactory):

        __implements__ = IIRCClientFactory,

        protocol = IRCReplyBot
        nickname = None

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(IRCClientFactoryFromService, IFingerService,
                               IIRCClientFactory)

    class UserStatusTree(resource.Resource):

        __implements__ = resource.IResource,

        def __init__(self, service):
            resource.Resource.__init__(self)
            self.service = service
            self.putChild('RPC2', UserStatusXR(self.service))

        def render(self, request):
            d = self.service.getUsers()
            def formatUsers(users):
                l = ['<li><a href="%s">%s</a></li>' % (user, user)
                     for user in users]
                return '<ul>'+''.join(l)+'</ul>'
            d.addCallback(formatUsers)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

        def getChild(self, path, request):
            if path=="":
                return UserStatusTree(self.service)
            else:
                return UserStatus(path, self.service)

    components.registerAdapter(UserStatusTree, IFingerService,
                               resource.IResource)

    class UserStatus(resource.Resource):

        def __init__(self, user, service):
            resource.Resource.__init__(self)
            self.user = user
            self.service = service

        def render(self, request):
            d = self.service.getUser(self.user)
            d.addCallback(cgi.escape)
            d.addCallback(lambda m:
                          '<h1>%s</h1>'%self.user+'<p>%s</p>'%m)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

    class UserStatusXR(xmlrpc.XMLRPC):

        def __init__(self, service):
            xmlrpc.XMLRPC.__init__(self)
            self.service = service

        def xmlrpc_getUser(self, user):
            return self.service.getUser(user)

    class FingerService(service.Service):

        __implements__ = IFingerService,

        def __init__(self, filename):
            self.filename = filename
            self._read()

        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

    # Advantages of latest version

    class MemoryFingerService(service.Service):

        __implements__ = IFingerService, IFingerSetterService

        def __init__(self, **kwargs):
            self.users = kwargs

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

        def setUser(self, user, status):
            self.users[user] = status

    application = service.Application('finger', uid=1, gid=1)
    f = MemoryFingerService(moshez='Happy and well')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, IFingerFactory(f)
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(resource.IResource(f))
                       ).setServiceParent(serviceCollection)
    i = IIRCClientFactory(f)
    i.nickname = 'fingerbot'
    internet.TCPClient('irc.freenode.org', 6667, i
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(1079, IFingerSetterFactory(f), interface='127.0.0.1'
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger19a.py

Here we show just how convenient it is to implement new back-ends when we move to a component based architecture. Note that here we also use an interface we previously wrote, FingerSetterFactory, by supporting one single method. We manage to preserve the service's ignorance of the network.

## Another Back-end

    from twisted.internet import protocol, reactor, defer, utils
    import pwd

    # Another back-end

    class LocalFingerService(service.Service):

        __implements__ = IFingerService

        def getUser(self, user):
        # need a local finger daemon running for this to work
            return utils.getProcessOutput("finger", [user])

        def getUsers(self):
            return defer.succeed([])

    f = LocalFingerService()

Source listing - listings/finger/finger19b_changes.py

Full source code here:

    # Do everything properly, and componentize
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer, utils
    from twisted.protocols import basic, irc
    from twisted.python import components
    from twisted.web import resource, server, static, xmlrpc
    import cgi
    import pwd

    class IFingerService(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def getUsers(self):
            """Return a deferred returning a list of strings"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    def catchError(err):
        return "Internal error in server"

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value+'\n')
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class IFingerFactory(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerFactory,

        protocol = FingerProtocol

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(FingerFactoryFromService,
                               IFingerService,
                               IFingerFactory)

    class FingerSetterProtocol(basic.LineReceiver):

          def connectionMade(self):
              self.lines = []

          def lineReceived(self, line):
              self.lines.append(line)

          def connectionLost(self,reason):
              if len(self.lines) == 2:
                  self.factory.setUser(*self.lines)

    class IFingerSetterFactory(components.Interface):

        def setUser(self, user, status):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerSetterFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerSetterFactory,

        protocol = FingerSetterProtocol

        def __init__(self, service):
            self.service = service

        def setUser(self, user, status):
            self.service.setUser(user, status)

    components.registerAdapter(FingerSetterFactoryFromService,
                               IFingerSetterService,
                               IFingerSetterFactory)

    class IRCReplyBot(irc.IRCClient):

        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)
                d.addErrback(catchError)
                d.addCallback(lambda m: "Status of %s: %s" % (msg, m))
                d.addCallback(lambda m: self.msg(user, m))

    class IIRCClientFactory(components.Interface):

        '''
        @ivar nickname
        '''

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol"""

    class IRCClientFactoryFromService(protocol.ClientFactory):

        __implements__ = IIRCClientFactory,

        protocol = IRCReplyBot
        nickname = None

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(IRCClientFactoryFromService, IFingerService,
                               IIRCClientFactory)

    class UserStatusTree(resource.Resource):

        __implements__ = resource.IResource,

        def __init__(self, service):
            resource.Resource.__init__(self)
            self.service = service
            self.putChild('RPC2', UserStatusXR(self.service))

        def render(self, request):
            d = self.service.getUsers()
            def formatUsers(users):
                l = ['<li><a href="%s">%s</a></li>' % (user, user)
                     for user in users]
                return '<ul>'+''.join(l)+'</ul>'
            d.addCallback(formatUsers)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

        def getChild(self, path, request):
            if path=="":
                return UserStatusTree(self.service)
            else:
                return UserStatus(path, self.service)

    components.registerAdapter(UserStatusTree, IFingerService,
                               resource.IResource)

    class UserStatus(resource.Resource):

        def __init__(self, user, service):
            resource.Resource.__init__(self)
            self.user = user
            self.service = service

        def render(self, request):
            d = self.service.getUser(self.user)
            d.addCallback(cgi.escape)
            d.addCallback(lambda m:
                          '<h1>%s</h1>'%self.user+'<p>%s</p>'%m)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

    class UserStatusXR(xmlrpc.XMLRPC):

        def __init__(self, service):
            xmlrpc.XMLRPC.__init__(self)
            self.service = service

        def xmlrpc_getUser(self, user):
            return self.service.getUser(user)

    class FingerService(service.Service):

        __implements__ = IFingerService,

        def __init__(self, filename):
            self.filename = filename
            self._read()

        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

    # Another back-end

    class LocalFingerService(service.Service):

        __implements__ = IFingerService

        def getUser(self, user):
        # need a local finger daemon running for this to work
            return utils.getProcessOutput("finger", [user])

        def getUsers(self):
            return defer.succeed([])

    application = service.Application('finger', uid=1, gid=1)
    f = LocalFingerService()
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, IFingerFactory(f)
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(resource.IResource(f))
                       ).setServiceParent(serviceCollection)
    i = IIRCClientFactory(f)
    i.nickname = 'fingerbot'
    internet.TCPClient('irc.freenode.org', 6667, i
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger19b.py

We've already written this, but now we get more for less work: the network code is completely separate from the back-end.

## Yet Another Back-end: Doing the Standard Thing

    from twisted.internet import protocol, reactor, defer, utils
    import pwd
    import os

    # Yet another back-end

    class LocalFingerService(service.Service):

        __implements__ = IFingerService

        def getUser(self, user):
            user = user.strip()
            try:
                entry = pwd.getpwnam(user)
            except KeyError:
                return defer.succeed("No such user")
            try:
                f = file(os.path.join(entry[5],'.plan'))
            except (IOError, OSError):
                return defer.succeed("No such user")
            data = f.read()
            data = data.strip()
            f.close()
            return defer.succeed(data)

        def getUsers(self):
            return defer.succeed([])

    f = LocalFingerService()

Source listing - listings/finger/finger19c_changes.py

Full source code here:

    # Do everything properly, and componentize
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer, utils
    from twisted.protocols import basic, irc
    from twisted.python import components
    from twisted.web import resource, server, static, xmlrpc
    import cgi
    import pwd
    import os

    class IFingerService(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def getUsers(self):
            """Return a deferred returning a list of strings"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    def catchError(err):
        return "Internal error in server"

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value)
                self.transport.write('\n\n')
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class IFingerFactory(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerFactory,

        protocol = FingerProtocol

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(FingerFactoryFromService,
                               IFingerService,
                               IFingerFactory)

    class FingerSetterProtocol(basic.LineReceiver):

          def connectionMade(self):
              self.lines = []

          def lineReceived(self, line):
              self.lines.append(line)

          def connectionLost(self,reason):
              if len(self.lines) == 2:
                  self.factory.setUser(*self.lines)

    class IFingerSetterFactory(components.Interface):

        def setUser(self, user, status):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerSetterFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerSetterFactory,

        protocol = FingerSetterProtocol

        def __init__(self, service):
            self.service = service

        def setUser(self, user, status):
            self.service.setUser(user, status)

    components.registerAdapter(FingerSetterFactoryFromService,
                               IFingerSetterService,
                               IFingerSetterFactory)

    class IRCReplyBot(irc.IRCClient):

        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)
                d.addErrback(catchError)
                d.addCallback(lambda m: "Status of %s: %s" % (msg, m))
                d.addCallback(lambda m: self.msg(user, m))

    class IIRCClientFactory(components.Interface):

        '''
        @ivar nickname
        '''

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol"""

    class IRCClientFactoryFromService(protocol.ClientFactory):

        __implements__ = IIRCClientFactory,

        protocol = IRCReplyBot
        nickname = None

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(IRCClientFactoryFromService, IFingerService,
                               IIRCClientFactory)

    class UserStatusTree(resource.Resource):

        __implements__ = resource.IResource,

        def __init__(self, service):
            resource.Resource.__init__(self)
            self.service = service
            self.putChild('RPC2', UserStatusXR(self.service))

        def render(self, request):
            d = self.service.getUsers()
            def formatUsers(users):
                l = ['<li><a href="%s">%s</a></li>' % (user, user)
                     for user in users]
                return '<ul>'+''.join(l)+'</ul>'
            d.addCallback(formatUsers)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

        def getChild(self, path, request):
            if path=="":
                return UserStatusTree(self.service)
            else:
                return UserStatus(path, self.service)

    components.registerAdapter(UserStatusTree, IFingerService,
                               resource.IResource)

    class UserStatus(resource.Resource):

        def __init__(self, user, service):
            resource.Resource.__init__(self)
            self.user = user
            self.service = service

        def render(self, request):
            d = self.service.getUser(self.user)
            d.addCallback(cgi.escape)
            d.addCallback(lambda m:
                          '<h1>%s</h1>'%self.user+'<p>%s</p>'%m)
            d.addCallback(request.write)
            d.addCallback(lambda _: request.finish())
            return server.NOT_DONE_YET

    class UserStatusXR(xmlrpc.XMLRPC):

        def __init__(self, service):
            xmlrpc.XMLRPC.__init__(self)
            self.service = service

        def xmlrpc_getUser(self, user):
            return self.service.getUser(user)

    class FingerService(service.Service):

        __implements__ = IFingerService,

        def __init__(self, filename):
            self.filename = filename
            self._read()

        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

    # Yet another back-end

    class LocalFingerService(service.Service):

        __implements__ = IFingerService

        def getUser(self, user):
            user = user.strip()
            try:
                entry = pwd.getpwnam(user)
            except KeyError:
                return defer.succeed("No such user")
            try:
                f = file(os.path.join(entry[5],'.plan'))
            except (IOError, OSError):
                return defer.succeed("No such user")
            data = f.read()
            data = data.strip()
            f.close()
            return defer.succeed(data)

        def getUsers(self):
            return defer.succeed([])

    application = service.Application('finger', uid=1, gid=1)
    f = LocalFingerService()
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, IFingerFactory(f)
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(resource.IResource(f))
                       ).setServiceParent(serviceCollection)
    i = IIRCClientFactory(f)
    i.nickname = 'fingerbot'
    internet.TCPClient('irc.freenode.org', 6667, i
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger19c.py

Not much to say except that now we can be churn out backends like crazy. Feel like doing a back-end for Advogato, for example? Dig out the XML-RPC client support Twisted has, and get to work!

## Aspect-Oriented Programming

At last, an example of aspect-oriented programming that isn't about logging or timing. This code is actually useful! Watch how aspect-oriented programming helps you write less code and have fewer dependencies!

## Use Woven

Here we convert to using Woven, instead of manually constructing HTML snippets. Woven is a sophisticated web templating system. Its main features are to disallow any code inside the HTML, and transparent integration with deferred results.

    # Do everything properly, and componentize
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic, irc
    from twisted.python import components
    from twisted.web import resource, server, static, xmlrpc, microdom
    from twisted.web.woven import page, model, interfaces #,widgets
    import cgi

    class IFingerService(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def getUsers(self):
            """Return a deferred returning a list of strings"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    def catchError(err):
        return "Internal error in server"

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value+"\n")
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class IFingerFactory(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerFactory,

        protocol = FingerProtocol

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(FingerFactoryFromService, IFingerService,
                               IFingerFactory)

    class FingerSetterProtocol(basic.LineReceiver):

          def connectionMade(self):
              self.lines = []

          def lineReceived(self, line):
              self.lines.append(line)

          def connectionLost(self, reason):
              if len(self.lines) == 2:
                  self.factory.setUser(*self.lines)

    class IFingerSetterFactory(components.Interface):

        def setUser(self, user, status):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerSetterFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerSetterFactory,

        protocol = FingerSetterProtocol

        def __init__(self, service):
            self.service = service

        def setUser(self, user, status):
            self.service.setUser(user, status)

    components.registerAdapter(FingerSetterFactoryFromService,
                               IFingerSetterService,
                               IFingerSetterFactory)

    class IRCReplyBot(irc.IRCClient):

        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)
                d.addErrback(catchError)
                d.addCallback(lambda m: "Status of %s: %s " % (msg, m))
                d.addCallback(lambda m: self.msg(user, m))

    class IIRCClientFactory(components.Interface):

        """
        @ivar nickname
        """

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol"""

    class IRCClientFactoryFromService(protocol.ClientFactory):

       __implements__ = IIRCClientFactory,

       protocol = IRCReplyBot
       nickname = None

       def __init__(self, service):
           self.service = service

       def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(IRCClientFactoryFromService, IFingerService,
                               IIRCClientFactory)

    class UsersModel(model.MethodModel):

        def initialize(self, *args, **kwargs):
            self.service=args[0]

        def wmfactory_users(self, request):
            return self.service.getUsers()

    components.registerAdapter(UsersModel, IFingerService, interfaces.IModel)

    class UserStatusTree(page.Page):

        template = """<html><head><title>Users</title></head><body>
        <h1>Users</h1>
        <ul model="users" view="List">
        <li pattern="listItem"><a view="Anchor" /></li>
        </ul></body></html>"""

        def initialize(self, *args, **kwargs):
            self.service=args[0]

        def getDynamicChild(self, path, request):
            return UserStatus(user=path, service=self.service)

        def wchild_RPC2 (self, request):
            return UserStatusXR(self.service)

    components.registerAdapter(UserStatusTree, IFingerService, resource.IResource)

    class UserStatus(page.Page):

        template='''<html><head><title view="Text" model="user"/></head>
        <body><h1 view="Text" model="user"/>
        <p model="status" view="Text" />
        </body></html>'''

        def initialize(self, **kwargs):
            self.user = kwargs['user']
            self.service = kwargs['service']

        def wmfactory_user(self, request):
            return self.user

        def wmfactory_status(self, request):
            return self.service.getUser(self.user)

    class UserStatusXR(xmlrpc.XMLRPC):

        def __init__(self, service):
            xmlrpc.XMLRPC.__init__(self)
            self.service = service

        def xmlrpc_getUser(self, user):
            return self.service.getUser(user)

        def xmlrpc_getUsers(self):
            return self.service.getUsers()

    class FingerService(service.Service):

        __implements__ = IFingerService,

        def __init__(self, filename):
            self.filename = filename
            self._read()

        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, IFingerFactory(f)
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000, server.Site(resource.IResource(f))
                       ).setServiceParent(serviceCollection)
    i = IIRCClientFactory(f)
    i.nickname = 'fingerbot'
    internet.TCPClient('irc.freenode.org', 6667, i
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger20.py

## Use Perspective Broker

We add support for perspective broker, Twisted's native remote object protocol. Now, Twisted clients will not have to go through XML-RPCish contortions to get information about users.

    # Do everything properly, and componentize
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic, irc
    from twisted.python import components
    from twisted.web import resource, server, static, xmlrpc, microdom
    from twisted.web.woven import page, model, interfaces
    from twisted.spread import pb
    import cgi

    class IFingerService(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def getUsers(self):
            """Return a deferred returning a list of strings"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    def catchError(err):
        return "Internal error in server"

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value)
                self.transport.write('\n\n')
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class IFingerFactory(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerFactory,

        protocol = FingerProtocol

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(FingerFactoryFromService,
                               IFingerService,
                               IFingerFactory)

    class FingerSetterProtocol(basic.LineReceiver):

        def connectionMade(self):
            self.lines = []

        def lineReceived(self, line):
            self.lines.append(line)

        def connectionLost(self, reason):
            if len(self.lines) == 2:
                self.factory.setUser(*self.lines)

    class IFingerSetterFactory(components.Interface):

        def setUser(self, user, status):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerSetterFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerSetterFactory,

        protocol = FingerSetterProtocol

        def __init__(self, service):
            self.service = service

        def setUser(self, user, status):
            self.service.setUser(user, status)

    components.registerAdapter(FingerSetterFactoryFromService,
                               IFingerSetterService,
                               IFingerSetterFactory)

    class IRCReplyBot(irc.IRCClient):

        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)
                d.addErrback(catchError)
                d.addCallback(lambda m: "Status of %s: %s" % (msg, m))
                d.addCallback(lambda m: self.msg(user, m))

    class IIRCClientFactory(components.Interface):

        """
        @ivar nickname
        """

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol"""

    class IRCClientFactoryFromService(protocol.ClientFactory):

        __implements__ = IIRCClientFactory,

        protocol = IRCReplyBot
        nickname = None

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(IRCClientFactoryFromService,
                               IFingerService,
                               IIRCClientFactory)

    class UsersModel(model.MethodModel):
        def initialize(self, *args, **kwargs):
            self.service=args[0]

        def wmfactory_users(self, request):
            return self.service.getUsers()

    components.registerAdapter(UsersModel, IFingerService, interfaces.IModel)

    class UserStatusTree(page.Page):

        template = """<html><head><title>Users</title></head><body>
        <h1>Users</h1>
        <ul model="users" view="List">
        <li pattern="listItem"><a view="Anchor" /></li>
        </ul></body></html>"""

        def initialize(self, *args, **kwargs):
            self.service=args[0]

        def getDynamicChild(self, path, request):
            return UserStatus(user=path, service=self.service)

        def wchild_RPC2 (self, request):
            return UserStatusXR(self.service)

    components.registerAdapter(UserStatusTree, IFingerService, resource.IResource)

    class UserStatus(page.Page):

        template='''<html><head><title view="Text" model="user"/></head>
        <body><h1 view="Text" model="user"/>
        <p model="status" view="Text" />
        </body></html>'''

        def initialize(self, **kwargs):
            self.user = kwargs['user']
            self.service = kwargs['service']

        def wmfactory_user(self, request):
            return self.user

        def wmfactory_status(self, request):
            return self.service.getUser(self.user)

    class UserStatusXR(xmlrpc.XMLRPC):

        def __init__(self, service):
            xmlrpc.XMLRPC.__init__(self)
            self.service = service

        def xmlrpc_getUser(self, user):
            return self.service.getUser(user)

        def xmlrpc_getUsers(self):
            return self.service.getUsers()

    class IPerspectiveFinger(components.Interface):

        def remote_getUser(self, username):
            """return a user's status"""

        def remote_getUsers(self):
            """return a user's status"""

    # need to write a little script to test this PB shit

    class PerspectiveFingerFromService(pb.Root):

        __implements__ = pb.Root.__implements__, IPerspectiveFinger

        def __init__ (self, *args):
            self.service = args[0]

        def remote_getUser(self, username):
            return self.service.getUser(username)

        def remote_getUsers(self):
            return self.service.getUsers()

    components.registerAdapter(PerspectiveFingerFromService,
                               IFingerService,
                               IPerspectiveFinger)

    class FingerService(service.Service):

        __implements__ = IFingerService,

        def __init__(self, filename):
            self.filename = filename
            self._read()

        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, IFingerFactory(f)
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8000,server.Site(resource.IResource(f))
                       ).setServiceParent(serviceCollection)
    i = IIRCClientFactory(f)
    i.nickname = 'fingerbot'
    internet.TCPClient('irc.freenode.org', 6667, i
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8889, pb.PBServerFactory(IPerspectiveFinger(f))
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger21.py

A simple client to test the perspective broker finger:

    # test the PB finger on port 8889
    # this code is essentially the same as
    # the first example in howto/pb-usage

    from twisted.spread import pb
    from twisted.internet import reactor

    def gotObject(object):
        print "got object:", object
        object.callRemote("getUser","moshez").addCallback(gotData)
    # or
    #   object.callRemote("getUsers").addCallback(gotData)

    def gotData(data):
        print 'server sent:', data
        reactor.stop()

    def gotNoObject(reason):
        print "no object:",reason
        reactor.stop()

    factory = pb.PBClientFactory()
    reactor.connectTCP("127.0.0.1",8889, factory)
    factory.getRootObject().addCallbacks(gotObject,gotNoObject)
    reactor.run()

Source listing - listings/finger/fingerPBclient.py

## Support HTTPS

All we need to do to code an HTTPS site is just write a context factory (in this case, which loads the certificate from a certain file) and then use the twisted.application.internet.SSLServer method. Note that one factory (in this case, a site) can listen on multiple ports with multiple protocols.

    # Do everything properly, and componentize
    from twisted.application import internet, service
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic, irc
    from twisted.python import components
    from twisted.web import resource, server, static, xmlrpc, microdom
    from twisted.web.woven import page, model, interfaces
    from twisted.spread import pb
    #from OpenSSL import SSL
    import cgi

    class IFingerService(components.Interface):
        def getUser(self, user):
            """Return a deferred returning a string"""

        def getUsers(self):
            """Return a deferred returning a list of strings"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    def catchError(err):
        return "Internal error in server"

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value)
                self.transport.write('\n\n')
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class IFingerFactory(components.Interface):
        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerFactoryFromService(protocol.ServerFactory):
        __implements__ = IFingerFactory,

        protocol = FingerProtocol

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(FingerFactoryFromService,
                               IFingerService,
                               IFingerFactory)

    class FingerSetterProtocol(basic.LineReceiver):

        def connectionMade(self):
            self.lines = []

        def lineReceived(self, line):
            self.lines.append(line)

        def connectionLost(self, reason):
            if len(self.lines) == 2:
                self.factory.setUser(*self.lines)

    class IFingerSetterFactory(components.Interface):

        def setUser(self, user, status):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerSetterFactoryFromService(protocol.ServerFactory):

        __implements__ = IFingerSetterFactory,

        protocol = FingerSetterProtocol

        def __init__(self, service):
            self.service = service

        def setUser(self, user, status):
            self.service.setUser(user, status)

    components.registerAdapter(FingerSetterFactoryFromService,
                               IFingerSetterService,
                               IFingerSetterFactory)

    class IRCReplyBot(irc.IRCClient):

        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)
                d.addErrback(catchError)
                d.addCallback(lambda m: "Status of %s: %s" % (msg, m))
                d.addCallback(lambda m: self.msg(user, m))

    class IIRCClientFactory(components.Interface):

        """
        @ivar nickname
        """

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol"""

    class IRCClientFactoryFromService(protocol.ClientFactory):

        __implements__ = IIRCClientFactory,

        protocol = IRCReplyBot
        nickname = None

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(IRCClientFactoryFromService,
                               IFingerService,
                               IIRCClientFactory)

    class UsersModel(model.MethodModel):

        def initialize(self, *args, **kwargs):
            self.service=args[0]

        def wmfactory_users(self, request):
            return self.service.getUsers()

    components.registerAdapter(UsersModel, IFingerService, interfaces.IModel)

    class UserStatusTree(page.Page):

        template = """<html><head><title>Users</title></head><body>
        <h1>Users</h1>
        <ul model="users" view="List">
        <li pattern="listItem"><a view="Anchor" /></li>
        </ul></body></html>"""

        def initialize(self, *args, **kwargs):
            self.service=args[0]

        def getDynamicChild(self, path, request):
            return UserStatus(user=path, service=self.service)

        def wchild_RPC2 (self, request):
            return UserStatusXR(self.service)

    components.registerAdapter(UserStatusTree, IFingerService, resource.IResource)

    class UserStatus(page.Page):

        template='''<html><head><title view="Text" model="user"/></head>
        <body><h1 view="Text" model="user"/>
        <p model="status" view="Text" />
        </body></html>'''

        def initialize(self, **kwargs):
            self.user = kwargs['user']
            self.service = kwargs['service']

        def wmfactory_user(self, request):
            return self.user

        def wmfactory_status(self, request):
            return self.service.getUser(self.user)

    class UserStatusXR(xmlrpc.XMLRPC):

        def __init__(self, service):
            xmlrpc.XMLRPC.__init__(self)
            self.service = service

        def xmlrpc_getUser(self, user):
            return self.service.getUser(user)

        def xmlrpc_getUsers(self):
            return self.service.getUsers()

    class IPerspectiveFinger(components.Interface):

        def remote_getUser(self, username):
            """return a user's status"""

        def remote_getUsers(self):
            """return a user's status"""

    class PerspectiveFingerFromService(pb.Root):

        __implements__ = pb.Root.__implements__, IPerspectiveFinger

        def __init__(self, service):
            self.service = service

        def remote_getUser(self, username):
            return self.service.getUser(username)

        def remote_getUsers(self):
            return self.service.getUsers()

    components.registerAdapter(PerspectiveFingerFromService,
                               IFingerService,
                               IPerspectiveFinger)

    class FingerService(service.Service):

        __implements__ = IFingerService,

        def __init__(self, filename):
            self.filename = filename
            self._read()

        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

    class ServerContextFactory:

        def getContext(self):
            """Create an SSL context.

            This is a sample implementation that loads a certificate from a file
            called 'server.pem'."""
            ctx = SSL.Context(SSL.SSLv23_METHOD)
            ctx.use_certificate_file('server.pem')
            ctx.use_privatekey_file('server.pem')
            return ctx

    application = service.Application('finger', uid=1, gid=1)
    f = FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, IFingerFactory(f)
                       ).setServiceParent(serviceCollection)
    site = server.Site(resource.IResource(f))
    internet.TCPServer(8000, site
                       ).setServiceParent(serviceCollection)
    internet.SSLServer(443, site, ServerContextFactory()
                       ).setServiceParent(serviceCollection)
    i = IIRCClientFactory(f)
    i.nickname = 'fingerbot'
    internet.TCPClient('irc.freenode.org', 6667, i
                       ).setServiceParent(serviceCollection)
    internet.TCPServer(8889, pb.PBServerFactory(IPerspectiveFinger(f))
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/finger22.py

## Finger Proxy

Writing new clients with Twisted is much like writing new servers. We implement the protocol, which just gathers up all the data, and give it to the factory. The factory keeps a deferred which is triggered if the connection either fails or succeeds. When we use the client, we first make sure the deferred will never fail, by producing a message in that case. Implementing a wrapper around client which just returns the deferred is a common pattern. While less flexible than using the factory directly, it's also more convenient.

    # finger proxy
    from twisted.application import internet, service
    from twisted.internet import defer, protocol, reactor
    from twisted.protocols import basic
    from twisted.python import components

    def catchError(err):
        return "Internal error in server"

    class IFingerService(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def getUsers(self):
            """Return a deferred returning a list of strings"""

    class IFingerFactory(components.Interface):

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value)
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class FingerFactoryFromService(protocol.ClientFactory):

        __implements__ = IFingerFactory,

        protocol = FingerProtocol

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(FingerFactoryFromService,
                               IFingerService,
                               IFingerFactory)

    class FingerClient(protocol.Protocol):

        def connectionMade(self):
            self.transport.write(self.factory.user+"\r\n")
            self.buf = []

        def dataReceived(self, data):
            self.buf.append(data)

        def connectionLost(self):
            self.factory.gotData(''.join(self.buf))

    class FingerClientFactory(protocol.ClientFactory):

        protocol = FingerClient

        def __init__(self, user):
            self.user = user
            self.d = defer.Deferred()

        def clientConnectionFailed(self, _, reason):
            self.d.errback(reason)

        def gotData(self, data):
            self.d.callback(data)

    def finger(user, host, port=79):
        f = FingerClientFactory(user)
        reactor.connectTCP(host, port, f)
        return f.d

    class ProxyFingerService(service.Service):
        __implements__ = IFingerService

        def getUser(self, user):
            try:
                user, host = user.split('@', 1)
            except:
                user = user.strip()
                host = '127.0.0.1'
            ret = finger(user, host)
            ret.addErrback(lambda _: "Could not connect to remote host")
            return ret

        def getUsers(self):
            return defer.succeed([])

    application = service.Application('finger', uid=1, gid=1)
    f = ProxyFingerService()
    internet.TCPServer(7779, IFingerFactory(f)).setServiceParent(
        service.IServiceCollection(application))

Source listing - listings/finger/fingerproxy.py

## Organization

Now this code, while quite modular and well-designed, isn't properly organized. Everything above the `application=` belongs in a module, and the HTML templates all belong in separate files.

We can use the templateFile and templateDirectory attributes to indicate what HTML template file to use for each Page, and where to look for it.

    # organized-finger.tac
    # eg:  twistd -ny organized-finger.tac

    import finger

    from twisted.internet import protocol, reactor, defer
    from twisted.spread import pb
    from twisted.web import resource, server
    from twisted.application import internet, service, strports
    from twisted.python import log

    application = service.Application('finger', uid=1, gid=1)
    f = finger.FingerService('/etc/users')
    serviceCollection = service.IServiceCollection(application)
    internet.TCPServer(79, finger.IFingerFactory(f)
                       ).setServiceParent(serviceCollection)

    site = server.Site(resource.IResource(f))
    internet.TCPServer(8000, site
                       ).setServiceParent(serviceCollection)

    internet.SSLServer(443, site, ServerContextFactory()
                       ).setServiceParent(serviceCollection)

    i = finger.IIRCClientFactory(f)
    i.nickname = 'fingerbot'
    internet.TCPClient('irc.freenode.org', 6667, i
                       ).setServiceParent(serviceCollection)

    internet.TCPServer(8889, pb.PBServerFactory(finger.IPerspectiveFinger(f))
                       ).setServiceParent(serviceCollection)

Source listing - listings/finger/organized-finger.tac

Note that our program is now quite separated. We have:

- Code (in the module)
- Configuration (file above)
- Presentation (templates)
- Content (/etc/users)
- Deployment (twistd)

Prototypes don't need this level of separation, so our earlier examples all bunched together. However, real applications do. Thankfully, if we write our code correctly, it is easy to achieve a good separation of parts.

## Easy Configuration

We can also supply easy configuration for common cases with a makeService method that will also help build .tap files later:

    # Easy configuration
    # makeService from finger module

    def makeService(config):
        # finger on port 79
        s = service.MultiService()
        f = FingerService(config['file'])
        h = internet.TCPServer(79, IFingerFactory(f))
        h.setServiceParent(s)

        # website on port 8000
        r = resource.IResource(f)
        r.templateDirectory = config['templates']
        site = server.Site(r)
        j = internet.TCPServer(8000, site)
        j.setServiceParent(s)

        # ssl on port 443
        if config.get('ssl'):
            k = internet.SSLServer(443, site, ServerContextFactory())
            k.setServiceParent(s)

        # irc fingerbot
        if config.has_key('ircnick'):
            i = IIRCClientFactory(f)
            i.nickname = config['ircnick']
            ircserver = config['ircserver']
            b = internet.TCPClient(ircserver, 6667, i)
            b.setServiceParent(s)

        # Pespective Broker on port 8889
        if config.has_key('pbport'):
            m = internet.TCPServer(
                int(config['pbport']),
                pb.PBServerFactory(IPerspectiveFinger(f)))
            m.setServiceParent(s)

        return s

Source listing - listings/finger/finger_config.py

And we can write simpler files now:

    # simple-finger.tac
    # eg:  twistd -ny simple-finger.tac

    from twisted.application import service

    import finger

    options = { 'file': '/etc/users',
                'templates': '/usr/share/finger/templates',
                'ircnick': 'fingerbot',
                'ircserver': 'irc.freenode.net',
                'pbport': 8889,
                'ssl': 'ssl=0' }

    ser = finger.makeService(options)
    application = service.Application('finger', uid=1, gid=1)
    ser.setServiceParent(service.IServiceCollection(application))

Source listing - listings/finger/simple-finger.tac

    % twisted -ny simple-finger.tac

Note: the finger user still has ultimate power: he can use makeService, or he can use the lower-level interface if he has specific needs (maybe an IRC server on some other port? maybe we want the non-SSL webserver to listen only locally? etc. etc.) This is an important design principle: never force a layer of abstraction: allow usage of layers of abstractions.

The pasta theory of design:

- Spaghetti: each piece of code interacts with every other piece of code [can be implemented with GOTO, functions, objects]
- Lasagna: code has carefully designed layers. Each layer is, in theory independent. However low-level layers usually cannot be used easily, and high-level layers depend on low-level layers.
- Ravioli: each part of the code is useful by itself. There is a thin layer of interfaces between various parts [the sauce]. Each part can be usefully be used elsewhere.
- ...but sometimes, the user just wants to order Ravioli, so one coarse-grain easily definable layer of abstraction on top of it all can be useful.

## Plugins

So far, the user had to be somewhat of a programmer to be able to configure stuff. Maybe we can eliminate even that? Move old code to finger/__init__.py and...

Full source code for finger module here:

    # finger.py module

    from twisted.application import internet, service, strports
    from twisted.internet import protocol, reactor, defer
    from twisted.protocols import basic, irc
    from twisted.python import components
    from twisted.web import resource, server, static, xmlrpc, microdom
    from twisted.web.woven import page, model, interfaces
    from twisted.spread import pb
    #from OpenSSL import SSL
    import cgi

    class IFingerService(components.Interface):
        def getUser(self, user):
            """Return a deferred returning a string"""

        def getUsers(self):
            """Return a deferred returning a list of strings"""

    class IFingerSetterService(components.Interface):

        def setUser(self, user, status):
            """Set the user's status to something"""

    def catchError(err):
        return "Internal error in server"

    class FingerProtocol(basic.LineReceiver):

        def lineReceived(self, user):
            d = self.factory.getUser(user)
            d.addErrback(catchError)
            def writeValue(value):
                self.transport.write(value)
                self.transport.write('\n\n')
                self.transport.loseConnection()
            d.addCallback(writeValue)

    class IFingerFactory(components.Interface):
        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerFactoryFromService(protocol.ServerFactory):
        __implements__ = protocol.ServerFactory.__implements__, IFingerFactory

        protocol = FingerProtocol

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(FingerFactoryFromService,
                               IFingerService,
                               IFingerFactory)

    class FingerSetterProtocol(basic.LineReceiver):

        def connectionMade(self):
            self.lines = []

        def lineReceived(self, line):
            self.lines.append(line)

        def connectionLost(self, reason):
            if len(self.lines) == 2:
                self.factory.setUser(*self.lines)

    class IFingerSetterFactory(components.Interface):

        def setUser(self, user, status):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol returning a string"""

    class FingerSetterFactoryFromService(protocol.ServerFactory):

        __implements__ = protocol.ServerFactory.__implements__,IFingerSetterFactory

        protocol = FingerSetterProtocol

        def __init__(self, service):
            self.service = service

        def setUser(self, user, status):
            self.service.setUser(user, status)

    components.registerAdapter(FingerSetterFactoryFromService,
                               IFingerSetterService,
                               IFingerSetterFactory)

    class IRCReplyBot(irc.IRCClient):

        def connectionMade(self):
            self.nickname = self.factory.nickname
            irc.IRCClient.connectionMade(self)

        def privmsg(self, user, channel, msg):
            user = user.split('!')[0]
            if self.nickname.lower() == channel.lower():
                d = self.factory.getUser(msg)
                d.addErrback(catchError)
                d.addCallback(lambda m: "Status of %s: %s" % (msg, m))
                d.addCallback(lambda m: self.msg(user, m))

    class IIRCClientFactory(components.Interface):

        """
        @ivar nickname
        """

        def getUser(self, user):
            """Return a deferred returning a string"""

        def buildProtocol(self, addr):
            """Return a protocol"""

    class IRCClientFactoryFromService(protocol.ClientFactory):

        __implements__ = protocol.ClientFactory.__implements__, IIRCClientFactory

        protocol = IRCReplyBot
        nickname = None

        def __init__(self, service):
            self.service = service

        def getUser(self, user):
            return self.service.getUser(user)

    components.registerAdapter(IRCClientFactoryFromService,
                               IFingerService,
                               IIRCClientFactory)

    class UsersModel(model.MethodModel):

        def initialize(self, *args, **kwargs):
            self.service=args[0]

        def wmfactory_users(self, request):
            return self.service.getUsers()

    components.registerAdapter(UsersModel, IFingerService, interfaces.IModel)

    class UserStatusTree(page.Page):

        template = """<html><head><title>Users</title></head><body>
        <h1>Users</h1>
        <ul model="users" view="List">
        <li pattern="listItem"><a view="Anchor" /></li>
        </ul></body></html>"""

        def initialize(self, *args, **kwargs):
            self.service=args[0]

        def getDynamicChild(self, path, request):
            return UserStatus(user=path, service=self.service)

        def wchild_RPC2 (self, request):
            return UserStatusXR(self.service)

    components.registerAdapter(UserStatusTree, IFingerService, resource.IResource)

    class UserStatus(page.Page):

        template='''<html><head><title view="Text" model="user"/></head>
        <body><h1 view="Text" model="user"/>
        <p model="status" view="Text" />
        </body></html>'''

        def initialize(self, **kwargs):
            self.user = kwargs['user']
            self.service = kwargs['service']

        def wmfactory_user(self, request):
            return self.user

        def wmfactory_status(self, request):
            return self.service.getUser(self.user)

    class UserStatusXR(xmlrpc.XMLRPC):

        def __init__(self, service):
            xmlrpc.XMLRPC.__init__(self)
            self.service = service

        def xmlrpc_getUser(self, user):
            return self.service.getUser(user)

        def xmlrpc_getUsers(self):
            return self.service.getUsers()

    class IPerspectiveFinger(components.Interface):

        def remote_getUser(self, username):
            """return a user's status"""

        def remote_getUsers(self):
            """return a user's status"""

    class PerspectiveFingerFromService(pb.Root):

        __implements__ = pb.Root.__implements__, IPerspectiveFinger

        def __init__(self, service):
            self.service = service

        def remote_getUser(self, username):
            return self.service.getUser(username)

        def remote_getUsers(self):
            return self.service.getUsers()

    components.registerAdapter(PerspectiveFingerFromService,
                               IFingerService,
                               IPerspectiveFinger)

    class FingerService(service.Service):

        __implements__ = service.Service.__implements__, IFingerService

        def __init__(self, filename):
            self.filename = filename
            self._read()

        def _read(self):
            self.users = {}
            for line in file(self.filename):
                user, status = line.split(':', 1)
                user = user.strip()
                status = status.strip()
                self.users[user] = status
            self.call = reactor.callLater(30, self._read)

        def getUser(self, user):
            return defer.succeed(self.users.get(user, "No such user"))

        def getUsers(self):
            return defer.succeed(self.users.keys())

    class ServerContextFactory:

        def getContext(self):
            """Create an SSL context.

            This is a sample implementation that loads a certificate from a file
            called 'server.pem'."""
            ctx = SSL.Context(SSL.SSLv23_METHOD)
            ctx.use_certificate_file('server.pem')
            ctx.use_privatekey_file('server.pem')
            return ctx

    # Easy configuration

    def makeService(config):
        # finger on port 79
        s = service.MultiService()
        f = FingerService(config['file'])
        h = internet.TCPServer(79, IFingerFactory(f))
        h.setServiceParent(s)

        # website on port 8000
        r = resource.IResource(f)
        r.templateDirectory = config['templates']
        site = server.Site(r)
        j = internet.TCPServer(8000, site)
        j.setServiceParent(s)

        # ssl on port 443
    #    if config.get('ssl'):
    #        k = internet.SSLServer(443, site, ServerContextFactory())
    #        k.setServiceParent(s)

        # irc fingerbot
        if config.has_key('ircnick'):
            i = IIRCClientFactory(f)
            i.nickname = config['ircnick']
            ircserver = config['ircserver']
            b = internet.TCPClient(ircserver, 6667, i)
            b.setServiceParent(s)

        # Pespective Broker on port 8889
        if config.has_key('pbport'):
            m = internet.TCPServer(
                int(config['pbport']),
                pb.PBServerFactory(IPerspectiveFinger(f)))
            m.setServiceParent(s)

        return s

finger module - listings/finger/finger/finger.py

    # finger/tap.py
    from twisted.application import internet, service
    from twisted.internet import interfaces
    from twisted.python import usage
    import finger

    class Options(usage.Options):

        optParameters = [
            ['file', 'f', '/etc/users'],
            ['templates', 't', '/usr/share/finger/templates'],
            ['ircnick', 'n', 'fingerbot'],
            ['ircserver', None, 'irc.freenode.net'],
            ['pbport', 'p', 8889],
            ]

        optFlags = [['ssl', 's']]

    def makeService(config):
        return finger.makeService(config)

finger/tap.py - listings/finger/finger/tap.py

And register it all:

    #finger/plugins.tml
    register('finger', 'finger.tap',
     description='Build a finger server tap',
     type='tap', tapname='finger')

finger/plugins.tml - listings/finger/finger/plugins.tml

And now, the following works

    % mktap finger --file=/etc/users --ircnick=fingerbot
    % sudo twistd -nf finger.tap

## OS Integration

If we already have the finger package installed in PYTHONPATH (e.g. we added it to site-packages), we can achieve easy integration:

### Debian

    % tap2deb --unsigned -m "Foo <foo@example.com>" --type=python finger.tac
    % sudo dpkg -i .build/*.deb

### Red Hat / Mandrake

    % tap2rpm --type=python finger.tac #[maybe other options needed]
    % sudo rpm -i .build/*.rpm

Will properly register the tap/tac, init.d scripts, etc. for the given file.

If it doesn't work on your favorite OS: patches accepted!

Index

Version: 1.1.0