# Using app.Application

## Motivation

Calling reactor methods (like `.listenTCP` and `.run`) directly, as in the examples in Writing Servers, is a good way to immediately demonstrate the use of Factories and Protocols. But you would ask for more from a fully-fledged, easy-to-run, easy-to-configure Internet Server. Twisted provides many of these features as a partnership between its Application object and the `twistd` administrator's program.

What more could we want? Well:

- starting as a daemon:

  One of the important aspects of a server program is that it can be run in the background, with output detached from the terminal, run in a chroot jail or under a different user id.

- configuration arguments:

  suppose your quote of the day ('QOTD') server behaves a bit more like the normal port 17 server and pulls a random line from `/usr/share/fortunes`. Your QOTDFactory() might take a filename to indicate where the QOTD protocols should pull these lines. It would be nice if the person installing your quote server didn't have to modify any Python code to change where this file should be found.

  Likewise, what if they want it to listen on some other port? That shouldn't require editing the code.

- basic persistence:

  If your protocol demands that you keep some state from one invocation of the server to the next, you'll need to save some information before the server shuts down, and to restore it again when you start back up.

  Suppose your protocol's purpose in life is to generate one-time keys, and that people can connect to it to retrieve a single-use key. (Don't ask me why they might want to do this. Security is such a weird big thing that chances are somebody out there will want to do something that's probably pretty dumb when you think about it carefully). The important thing is that you never give out the same key twice. So you have to remember a sequence number, and each time you give out a key, you bump up the number. Before you shut down, you save the number to a file somewhere; at start up, if the file exists you read the number from it, if it doesn't exist, you start at 0. (an example is included below)

  This kind of persistent data is a common need, and many kinds of servers require it. Hence Twisted provides an easy way to record and reload this data.

This functionality is provided by the Application class (defined in twisted/internet/app.py). You create an Application with a constructor like any other object. Then you tell the app to listen to ports (just like you told the reactor to in the previous example), providing a Factory on each one. The difference is that the App won't starting listening on those ports right away, but will wait until it starts to run.

When you're done setting up an Application object, there are three options. You can run it directly by calling the application's `.run()` method. You can use `twistd -y app.py` to run your application directly from python source code. Or you can save the Application out to a file by calling the `.save()` method. The saved application can then be started later by using `twistd -f app-start.tap`.

## Example Application

Here is a short example of the first option, running the server immediately. This example uses the pre-defined Daytime protocol, which simply sends the current time to each client.:

    #! /usr/bin/python

    from twisted.internet.app import Application
    from twisted.protocols.wire import Daytime
    from twisted.internet.protocol import Factory

    application = Application("daytimer")
    f = Factory()
    f.protocol = Daytime
    application.listenTCP(8813, f)

    if '__main__' == __name__:
        application.run()

Source listing - listings/application/app1.py

This program will start listening to port 8813 in the `app.run()` call, and won't return from that call until the server is terminated (probably when you send it SIGINT via ^C on the keyboard).

To use the second option, the same source file can be used, but using the `twistd -y` command to start the application as a daemon. If persistence is not required, then use the `--no_save` option.

    $ twistd --no_save --python=app1.py
    $ tail -f twistd.log
    04/04/2003 23:45 [-] Log opened.
    04/04/2003 23:45 [-] twistd 1.0.4alpha1 (/usr/local/bin/python 2.2.2)
    starting up
    04/04/2003 23:45 [-] license user: Nobody <>
    04/04/2003 23:45 [-] organization: No Organization
    04/04/2003 23:45 [-] reactor class: twisted.internet.default.SelectReactor
    04/04/2003 23:45 [-] Loading
    /home/cce/work/Twisted/doc/howto/listings/application/app1.py...
    04/04/2003 23:45 [-] Loaded.
    04/04/2003 23:45 [*daytimer*] twisted.internet.protocol.Factory starting on
    8813
    04/04/2003 23:45 [*daytimer*] Starting factory
    <twisted.internet.protocol.Factory instance at 0x835dcac>

To use the third option and launch the server later, just use `.save()` instead of `.run()`. The `.save()` method takes a base name for the generated `.tap` file:

    ...
    app.listenTCP(8813, f)

    app.save("start")

When you run this program, it will create a file called `daytime-start.tap`, and then exit. (The name is obtained by combining the application name with the argument to `.save()`). To start the server from the freeze-dried`.tap` file, use `twistd` (text wrapped to be more readable):

    % ./app2.py
    Saving daytimer application to daytimer-start.tap...
    Saved.
    % twistd -f daytimer-start.tap
    % tail twistd.log
    30/09/2002 01:38 [-] Log opened.
    30/09/2002 01:38 [-] twistd 0.99.2 (/usr/bin/python2.2 2.2.1) starting up
    30/09/2002 01:38 [-] license user: Nobody <>
    30/09/2002 01:38 [-] organization: No Organization
    30/09/2002 01:38 [-] reactor class: twisted.internet.default.SelectReactor
    30/09/2002 01:38 [-] Loading daytimer-start.tap...
    30/09/2002 01:38 [-] Loaded.
    30/09/2002 01:38 [*daytimer*] twisted.internet.protocol.Factory starting on 8813
    30/09/2002 01:38 [*daytimer*] Starting factory
    <twisted.internet.protocol.Factory instance at 0x81ac9fc>
    %

That will thaw out the `.tap` file, create the Application, and then run it just as if you'd invoked app.run() yourself. It forks the new server off into the background (so twistd itself completes instead of waiting for the server to die), writes the server's process ID to a file called `twistd.pid`, and directs all the server's stdout messages to a file called `twistd.log` (these file names can be changed by appropriate arguments to `twistd`: see `twistd -h` for a list).

When you try this example, be aware that `twistd` returns right away, but it takes a second or two for the server to actually start. The `twistd.pid` file won't be created until it does. Wait a moment before doing `ls` or `netstat`, or you'll think that the server failed to start. If it persists in failing, look in `twistd.log` for details. Remember that trying to bind to a reserved port will fail unless you're root, and the exception will be listed at the end of the log file.

To kill the server, just do:

    % kill `cat twistd.pid`

When the server is shut down, you'll notice that it creates a file called `daytimer-shutdown.tap` in the directory it was run from (again, the name is derived from the application name and the word shutdown). This `.tap` file is just like the `daytimer-start.tap` created by your original setup program, except that it represents the state of the Application object as it existed just before shutdown, rather than when it was freshly created by your code.

Also note that the `twistd.pid` file is automatically deleted when the application shuts down.

## Saving State Across Sessions: Adding Persistent Data

You can add persistent data (like that sequence number described above) to the protocol Factory object, and it will get saved in the `-shutdown.tap` file. Then, if you restart the server with `twistd -f daytimer-shutdown.tap`, the new server will get the data saved by the old server, and it can pick up where the old one left off, as if the server had been running continuously the whole time.

To take advantage of this, simply add the attributes you want to the Factory, or to your subclass of Service (see the docs on Perspective Broker for details about Services). When the application terminates, it simply pickles up the whole Application (and everything it references, including Factories and Services). Any attributes or objects you have added will be saved and later restored.

Here is an example:

    #! /usr/bin/python

    from twisted.internet.protocol import Protocol, Factory

    class OneTimeKey(Protocol):
        def connectionMade(self):
            key = self.factory.nextkey
            print "giving key", key
            self.factory.nextkey += 1
            self.transport.write("%d\n" % key)
            self.transport.loseConnection()

    def main():
        # namespaces are weird. if we used OneTimeKey directly, it would
        # pickle the instance as __main__.OneTimeKey, since we run this
        # module directly. So we reimport this module so the pickle refers
        # to it by its real name.
        import app3
        from twisted.internet.app import Application
        f = Factory()
        f.protocol = app3.OneTimeKey
        f.nextkey = 0
        app = Application("otk")
        app.listenTCP(8123, f)
        app.save("start")

    if __name__ == '__main__':
        main()

Source listing - listings/application/app3.py

To demonstrate this, do the following:

    % ./app3.py
    Saving otk application to otk-start.tap...
    Saved.
    % twistd -f otk-start.tap
    %
    % nc localhost 8123
    0
    % nc localhost 8123
    1
    % nc localhost 8123
    2
    %

Note that the stdout of the process is being directed into the log file, contained in `twistd.log`. Now stop the server, verify that it is no longer running, then restart it from the saved-at-shutdown `.tap` file:

    % kill `cat twistd.pid `
    % nc localhost 8123
    localhost [127.0.0.1] 8123 (?) : Connection refused
    % twistd -f otk-shutdown.tap
    % nc localhost 8123
    3
    %

Notice how the saved `.nextkey` attribute was restored, and the application picks up where it left off.

## Configuration arguments

To do this right, you'll want to follow the sequence described by the writing plugins document. Instead of writing a short program that creates a `.tap` file (by creating an Application, doing various .listenTCPs on it, then calling .save), you will write a subroutine called updateApplication(). This subroutine should take a bunch of config arguments (using the usage.Options class described in the plugins document) and use them to create Factories and feed them to .listenTCP on an existing Application instance.

With that in place, and a few files to register this new server you've created, a utility program called '`mktap`' can relieve you of the business of gathering user arguments and creating the app instance. `mktap` can use the Options subclass you define in your build-a-tap class to figure out what arguments are legal (`--port` taking a number, `--quotes` taking a filename, etc), provide `--help` with a list of valid arguments, and parse everything the user passes in `argv[]`. It creates the Application, then passes the app and the parsed options to your `updateApplication()` method, where you do the server-specific creation of a Factory and the various `listenTCP` calls. Then `mktap` saves out the `.tap` file, ready for starting by `twistd`.

The end result is that installing your new server is simplified to the following steps:

- Unpack your server module (including the classes and plugin glue files) into somewhere on your PYTHONPATH, perhaps `/usr/local/lib/python`.
- Run the standard `mktap` program, giving it the name of your module and whatever configuration arguments it requires. Watch it create a `.tap` file.
- Use `twistd` to start the server contained in the `.tap` file.

Pretty easy. At least your users will think so.

And, once your application is defined by the `.tap` file, there are other tools that can be used to configure it. `tap2deb` is a tool that creates installable Debian `.deb` packages from your `.tap` file, making installation even easier.

The Application object has some other features designed to solve common server needs:

- logging is controlled, through the log.Logger class
- the process can switch to a different uid/gid after binding reserved ports
- styles.Versioned allows old saved copies of an object to be upgraded when new versions of the class are available
- Applications have Authorizers, used to authenticate client connections
- Applications have Services, which can be accessed by PB clients

Index