# MiddleKit User's Guide

Version 0.8.1
Webware for Python 0.8.1

## Table of Contents

Synopsis
Data Types
Basic Types
Date and time types
Enumerations
Object references
Model Files
Classes.csv
Samples.csv
Configuration
Generated Python
Lists: addToBars()
Miscellaneous Topics
Database Name
Iterating over attributes
Deleting objects
Derived attributes
Model Inheritance
Related Links
Credit
Implementor's Guide
Test Cases

## Synopsis

MiddleKit provides an object-relational mapping layer that enables developers to write object-oriented, Python-centric code while enjoying the benefits of a relational database.

The benefits of Python include: generality, OOP (encapsulation, inheritance, polymorphism), a mature language, and many libraries, both standard and 3rd party.

The benefits of a relational database include: data storage, concurrent access and 3rd party tools (such as report generators).

## Data Types

All attributes in an object model must be typed. MiddleKit divides types into 4 major categories:

1. Basic types
2. Date and time types
3. Enumerations
4. Object references

### Basic Types

The basic types are easy to work with and do what you expect. They are the same as the Python types with the addition of one special case: `bool`.

While MiddleKit is designed to be generic and therefore, database agnostic, concrete examples pay good dividends. So the following table includes the type equivalents for MySQL.

Basic Types
MiddleKit | Python | MySQL | Notes
--- | --- | --- | ---
bool | int | bool | Py int = 0 or 1
int | int | int |
long | long | bigint | 64-bit int
float | float | double | 64-bit float
string | string | varchar (*) |

(*) The MySQL type is `char` if the minimum and maximum length are equal.

### Date and time types

MiddleKit supports types of date, time and datetime. With a default Python installation and MiddleKit, these are expressed as Python strings such as `'2001-02-27'`. If you have installed the mxDateTime package, then these values will be expressed as instances of DateTime and DateTimeDelta.

With a default Python installation and MiddleKit, you can pass strings

Date/Time Types
MiddleKit | Python | MySQL | Notes
--- | --- | --- | ---
datetime | string/DateTime | datetime |
date | string/DateTime | datetime |
time | string/DateTimeDelta | time |

### Enumerations

Enumerations are provided through the enum type which is directly supported in MySQL. In Python, these enumerations are kept as case sensitive strings. The object model must specify the valid values of the enumeration. While that be done via a column named "Enums", this is more frequently specified in the "Extras" column, where less common attribute specs are usually placed:

`Enums='red, green, blue'`

### Object references

There are two types of object references: a single reference and a list of references. In relational terminology, that would be "1 to 1" and "1 to many" respectively.

The type for a single reference in Python is indicated by simply naming the class of the object that can be referred to. That class must be defined in the object model and will therefore have it's own table in the database. User-defined classes are required to be capitalized (while other types are lower case). For example:

Attribute | Type
--- | ---
address | Address
billingInfo | BillingInfo

The type for a list of references is specified by `list of ClassName` and represents the 1-to-many relationship. This will be an ordinary Python list, except that some invisible MiddleKit machinery in the background will perform various services (fetch objects on demand, insert new objects, etc.). For example:

Attribute | Type
--- | ---
contacts | list of Contact
customers | list of Customer

There are two additional properties that applies to attributes whose types are object references.

For the purpose of discussion, the object containing the attribute is self while the objects being referred to are the others. Now then, the onDeleteSelf property specifies what happens to the other object(s) when the self object is deleted:

1. deny - do not allow self to be deleted
2. cascade - also delete other when self is deleted
3. detach - allow self to be deleted with no effect on other (this is the default)

There is a similar property onDeleteOther which specifies what happens to the self object when the other object(s) is deleted:

1. deny - do not allow other to be deleted (this is the default)
2. cascade - also delete self when other is deleted
3. detach - allow other to be deleted, and set the reference attribute in self that referred to other to None

The default value of onDeleteSelf is detach, and the default value of onDeleteOther is deny. In other words, by default, you can delete an object which references other objects, but you can't delete an object which is referenced by other objects. An example specification would be `onDeleteOther=cascade`.

Note: onDeleteSelf can also be specified for "list of reference" attributes, where it has the same effect as it does when applied to reference attributes.

## Model Files

### Classes.csv

This is the object model where classes and attributes are defined. See the Quick Start for an example.

### Samples.csv

This is an optional file containing sample data for the model. See the Quick Start for an example.

Note that a blank field in the samples will be substituted with the default value of the attribute (as specified in the object model, e.g., Classes.csv). To force a None value (NULL in SQL), use 'none' (without the quotes).

### Configuration

An MK model can have configuration files inside it that affect things like code generation.

Settings.config is the primary configuration file.

The Package setting can be used to declare the package that your set of middle objects are contained by. This is useful for keeping your middle objects packaged away from other parts of your programs, thereby reducing the chances of a name conflict. This is the recommended way of using MK.

An example Settings.config:

    {
        'Package': 'Middle',
    }

Your code would then import classes like so:

    from Middle.Foo import Foo

Don't forget to put an __init__.py in the directory so that Python recognizes it as a package.

The SQLLog setting can be used to get MiddleKit to echo all SQL statements to 'stdout', 'stderr' or a filename. For filenames, an optional 'Mode' setting inside SQLLog can be used to write over or append to an existing file. The default is write. Here are some examples:

    {
        'SQLLog': { 'File': 'stdout' },
    }

    {
        'SQLLog': { 'File': 'middlekit.sql' },
    }

    {
        'SQLLog': { 'File': 'middlekit.sql', 'Mode': 'append' },
    }

The Database setting overrides the database name, which is otherwise assumed to be same name as the model. This is particularly useful if you are running two instances of the same application on one host.

    {
        'Database': 'foobar',
    }

The DeleteBehavior setting can be used to change what MiddleKit does when you delete objects. The default behavior is "delete" which means that objects are deleted from the SQL database when they are deleted from the MiddleKit object store. But setting DeleteBehavior to "mark" causes an extra SQL datetime column called "deleted" to be added to each SQL table, and records that are deleted from the object store in MiddleKit are kept in SQL tables with the deleted field set to the date/time when the object was deleted. This setting has no effect on the visible behavior of MiddleKit; it only changes what happens behind the scenes in the SQL store.

    {
        'DeleteBehavior': 'mark',
    }

The SQLConnectionPoolSize setting is used to create a MiscUtils.DBPool instance for use by the store. For DB API modules with a threadsafety of only 1 (such as MySQLdb), this is particularly useful (in one benchmark, the speed up was 15 - 20%). Simply set the size of the pool in order to have one created and used:

    {
    	'SQLConnectionPoolSize': 20,
    }

The UsePickledClassesCache setting defaults to 1 and causes MiddleKit to cache the Classes.csv text file as a binary pickle file named Classes.pickle.cache. This reduces subsequent load times by about 40%. The cache will be ignored if it can't be read, is older than the CSV file, has a different Python version, etc. You don't normally even need to think about this, but if for some reason you would like to turn off the use of the cache, you can do so through this setting.

SQLGenerator.config has one setting: DropStatements whose potential values are:

- database - This causes a DROP DATABASE to be used in Create.sql. This is the cleanest way to handle models and also the default.
- tables - This assumes the database already exists and does a DROP TABLE for each table in the model. There is a small danger here that if you remove a class from a model, then its old SQL table will remain in the database. The main use for this setting is for database servers that are slow creating new databases. One user reported a 1 minute lag for Microsoft SQL Server 2000, which is too much of a lag when running the test suite.

An example SQLGenerator.config:

    {
        'DropStatements': 'database',  # database, tables
    }

## Generated Python

### Attributes: foo() and setFoo()

For each attribute, foo, MiddleKit stores its value in the attribute _foo, returns it in the accessor method foo() and allows you to set it with setFoo(). You should always use foo() to get the value of an attribute, as there could be some logic there behind the scenes.

### Lists: addToBars()

Given an attribute of type list, with the name "bars", MK will generate a Python method named addToBars() that will make it easy for you to add a new object to the list:

    newBar = Bar()
    newBar.setXY(1, 2)
    foo.addToBars(newBar)

This method actually does a lot more for you, ensuring that you're not adding an object of the wrong type, adding the same object twice, etc. Here is a complete list:

- assert newBar is not None
- assert newBar inherits from Bar
- assert newBar is not already in bars
- add newBar to bars
- add newBar to foo's object store, if needed
- set newBar's foo-reference to foo

You don't have to remember the details since this behavior is both supplied and what you would expect. Just remember to use the various addToBars() methods.

Not setBars() method is provided for list typed attributes.

## Miscellaneous Topics

### Database Name

MiddleKit uses the name of the store as the name of the database. This works well most of the time. However, if you need to use a different database name, there are two techniques available:

1. You can specify the 'Database' setting in Settings.config. See Configuration for an example.

2. You can pass the database name via the object store's constructor arguments, which are then passed on to the DB API module. This technique overrides both the default model name and the model settings. For example:

    store = MySQLObjectStore(db='foobar', user='prog', passwd='asdklfj')

### Iterating over attributes

Every once in a while, you might get a hankering for iterating over the attributes of an MK object. You can do so like this:

    for attr in obj.klass().allAttrs():
    	print attr.name()

The klass() method seen above, returns the object's MiddleKit Klass, which is the class specification that came from the object model you created. The allAttrs() method returns a klass' list of attributes.

The attributes are instances of MiddleKit.Core.Attr (or one of its subclasses such as ObjRefAttr) which inherits from UserDict and acquires additional methods from mix-ins located in MiddleKit.Design and MiddleKit.Run. Since attributes are essentially dictionaries, you can treat them like so, although if you modify them you are asking for serious trouble.

    for attr in obj.klass().allAttrs():
    	keys = attr.keys()
    	keys.sort()
    	print '%s: %s' % (attr.name(), keys)

If you had asked the klass for its attrs() instead of allAttrs(), you would have missed out on attributes that were inherited.

If you want to get a dictionary of all the attribute values for a particular object, don't roll your own code. You can already ask your middle objects for allAttrs(), in which case you get values instead of definitions (which is what Klass returns for allAttrs()).

### Deleting objects

If you need to delete an object from the object store, you can do so like this:

    store.deleteObject(object)

As with other changes, the deletion is is not committed until you perform store.saveChanges().

This may raise one of these two exceptions defined in MiddleKit.Run.ObjectStore:

- DeleteReferencedException - This is raised when you try to delete an object that is referenced by other objects. To avoid this exception, either make sure that there are no other objects referencing an object before you try to delete it, or change the onDeleteOther property of the reference attribute to detach or cascade.
- DeleteObjectWithReferencesError - This is raised when you try to delete an object that references other objects, with a reference attribute that has the property onDeleteSelf=deny. To avoid this exception, either set the reference to None before deleting, or switch to a different setting for onDeleteSelf on the reference attribute.

See Object references for the specifications of onDeleteSelf and onDeleteOther.

### Derived attributes

Sometimes it can be convenient to define an attribute in MiddleKit that does not exist in the SQL database back end. Perhaps you want to compute the value from other attributes, or store the value somewhere else outside of the SQL database. Yet you still want to be able to iterate over the attribute using the allAttrs() method provided in MiddleKit.

To do this, simply set the property isDerived on the attribute in the model file. You will have to write your own setter and getter methods for the attribute.

## Model Inheritance

Model inheritance is an advanced feature for developers who wish to reuse models in other projects that are also model based. In Settings.config, you can specify other models to inhert class definitions from, which are termed parent models:

    {
        'Inherits': ['/usr/lib/mkmodels/News', 'Users'],
    }

Note that the .mkmodel extension is assumed. Also, relative filenames are relative to the path of the model inheriting them.

The essential effect is that the classes found in parent models are available to instantiate, subclass and create sample data from, and are termed inherited classes. You can also redefine an inherited class before using it in other class declarations. Classes are identified strictly by name.

The resolution order for finding a class in a model that has parent classes is the same as the basic method resolution order in Python 2.2, although don't take that mean that MiddleKit requires Python 2.2 (it requires 2.0 or greater).

Model inheritance does not affect the files found in the parent model directories. Also, settings and sample data are not inherited from parents; only class definitions.

In MiddleKit.Core.Model, the methods of interest that relate to model inheritance are klass(), which will traverse the parent model hierarchy if necessary, and allKlassesInOrder() and allKlassesByName(). See the doc strings for more info.

Caveats:

- Suppose model B inherits model A and that A has a class Base. If B wants to redefine Base and classes that inherit from it, the redefinition should come first:

      	Class Attr Type
      	Base
      	      b    int
      	Sub(Base)
      	      c    int

  If instead, B declares Sub first, then it will erroneously pick up the Base from A.

- A model cannot define a class such as "Foo(Foo)" where a given class is overridden, but inherited by the override. This could be useful for adding additional attribute definitions to an existing class.
- Although the "diamond of inheritance" situation has been accounted for, using ideas from Python 2.2, it is not yet covered by the test suite.
- "Infinite inheritance", where there is a cycle in inheritance, is not specifically caught.

## Related Links

The topic of object-relational mapping (ORM) is an old one. Here are some related links if you wish to explore the topic further:

Scott Ambler has written some papers on the topic of ORM and also maintains a set of ORM related links:

- Mapping Objects to Relational Databases
- The Design of a Robust Persistence Layer for Relational Databases

Apple has a very mature (and perhaps complex) ORM framework named Enterprise Objects Framework, or EOF, available in both Java and Objective-C. All the docs are online at the WebObjects page:

- WebObjects Developer Documentation

The only other Python ORM that we're aware of is dbObj by Boudewijn Rempt:

- dbObj

Here's a Perl ORM that someone recommended as interesting:

- Tangram

## Credit

Authors: Chuck Esterbrook

## Implementor's Guide

@@ 2000-10-28 ce: This should be a separate guide.

## Test Cases

In the Tests directory of MiddleKit you will find several test case object models.

@@ 2001-02-13 ce: complete this