# Fundamental Specifications of Kyoto Cabinet Version 1

Copyright (C) 2009-2010 Mikio Hirabayashi
Last Update: Sat, 17 Apr 2010 01:28:52 +0900

## Table of Contents

1. Introduction
2. Features
3. Installation
4. Tutorial
5. Command Line Utilities
6. Tips and Hacks
7. License

## Introduction

Kyoto Cabinet is a library of routines for managing a database. The database is a simple data file containing records, each is a pair of a key and a value. Every key and value is serial bytes with variable length. Both binary data and character string can be used as a key and a value. Each key must be unique within a database. There is neither concept of data tables nor data types. Records are organized in hash table or B+ tree.

The following access methods are provided to the database: storing a record with a key and a value, deleting a record by a key, retrieving a record by a key. Moreover, traversal access to every key are provided. These access methods are similar to ones of DBM (or its followers: NDBM and GDBM) library defined in the UNIX standard. Kyoto Cabinet is an alternative for DBM because of its higher performance.

Each operation of hash database has the time complexity of "O(1)". Therefore, in theory, the performance is constant regardless of the scale of the database. In practice, the performance is determined by the speed of the main memory or the storage device. If the size of the database is less than the capacity of the main memory, the performance will seem on-memory speed which is faster than std::map of STL. Of course, the database size can be greater than the capacity of the main memory and the upper limit is 8 exabytes. Even in that case, each operation needs only one or two seeking of the storage device.

Each operation of B+ tree has the time complexity of "O(log N)". Therefore, in theory, the performance is logarithmic to the scale of the database. Although the performance of random access of B+ tree is slower than that of hash database, B+ tree supports sequential access in order of the keys, which realizes forward matching search for strings and range search for integers. The performance of sequential access is much faster than that of random access.

As the API is based on object-oriented design, hash database and B+ tree database have same methods which inherited from the upper abstract class. Prototype database by containers of STL and cache database with LRU deletion algorithm are also provided under the same base class. All databases have practical utility methods related to transaction and cursor. Programs for command line interface are also included in the package.

Kyoto Cabinet runs very fast. For example, elapsed time to store one million records is 0.9 seconds for hash database, and 1.1 seconds for B+ tree database. Moreover, the size of database of Kyoto Cabinet is very small. For example, overhead for a record is 16 bytes for hash database, and 4 bytes for B+ tree database. Furthermore, scalability of Kyoto Cabinet is great. The database size can be up to 8EB (9.22e18 bytes).

Kyoto Cabinet is written in the C++ language, and provided as API of C++, C, Java, Python, Ruby, and Perl. Kyoto Cabinet is available on platforms which have API conforming to C++03 with the TR1 library extensions. Kyoto Cabinet is a free software licensed under the GNU General Public License.

## Features

This section describes the features of Kyoto Cabinet.

### Genealogy

The original DBM was developed by Kenneth Thompson as a part of the original AT&T UNIX. After that, a lot of followers developed such DBM-like products as NDBM, SDBM, GDBM, TDB, and BerkeleyDB. In 2003, I developed QDBM to replace GDBM for performance reason.

In 2007, Tokyo Cabinet was developed as the successor of QDBM on the following purposes. They were achieved and Tokyo Cabinet could replace conventional DBM products.

- improves space efficiency : smaller size of database file.
- improves time efficiency : faster processing speed.
- improves parallelism : higher performance in multi-thread environment.
- improves usability : simplified API.
- improves robustness : database file is not corrupted even under catastrophic situation.
- supports 64-bit architecture : enormous memory space and database file are available.

In 2009, Kyoto Cabinet was developed as another successor of QDBM. Compared with the sibling product (Tokyo Cabinet), the following advantages were pursued. However, the performance of Tokyo Cabinet is higher than Kyoto Cabinet, at least in single thread operations.

- improves space efficiency : smaller size of database file.
- improves parallelism : higher performance in multi-thread environment.
- improves portability : abstraction of the lower layer to support non-POSIX systems.
- improves usability : simplified API, object-oriented design.
- improves robustness : database file is not corrupted even under catastrophic situation.

I'll maintain the both of Tokyo Cabinet and Kyoto Cabinet because their values are different.

### Effective Implementation of Hash Database

Kyoto Cabinet uses hash algorithm to retrieve records. If a bucket array has sufficient number of elements, the time complexity of retrieval is "O(1)". That is, time required for retrieving a record is constant, regardless of the scale of a database. It is also the same about storing and deleting. Collision of hash values is managed by separate chaining. Data structure of the chains is binary search tree. Even if a bucket array has unusually scarce elements, the time complexity of retrieval is "O(log n)".

Kyoto Cabinet attains improvement in retrieval by loading RAM with the whole of the bucket array. If the bucket array is on RAM, it is possible to access a region of a target record by about one set of file operations such as `lseek', `read', and `write'. The bucket array saved in a file is not read into RAM with the `read' call but directly mapped to RAM with the `mmap' call. Therefore, preparation time on connecting to a database is very short, and two or more processes can share the same memory map.

The hash function used for hash table is MurMurHash 2.0. If the number of elements of the bucket array is about half of records stored within a database, although it depends on characteristic of the input, the probability of collision of hash values is about 55.3% (35.5% if the same, 20.4% if twice, 11.0% if four times, 5.7% if eight times). In that case, it is possible to retrieve a record by two or less sets of file operations. If it is made into a performance index, in order to handle a database containing one million of records, a bucket array with half a million of elements is required. The size of each element is 4 bytes. That is, if 2M bytes of RAM is available, a database containing one million records can be handled.

When overwriting a record with a value whose size is greater than the existing one, it is necessary to move the region to another position of the file. Because the time complexity of the operation depends on the size of the region of a record, extending values successively is inefficient. However, Kyoto Cabinet deal with this problem by alignment. If increment can be put in padding, it is not necessary to move the region.

Generally speaking, while succession of updating, fragmentation of available regions occurs, and the size of a database grows rapidly. Kyoto Cabinet deals with this problem by the free block pool and the auto defragmentation mechanism. If a record is removed or shifted to another position, the region will be treated as a free block. The free block pool manages free blocks and reuses the best fit region for a new record. The auto defragmentation is to shift records and free blocks separately. Successive free blocks coalesce into one.

### Useful Implementation of B+ Tree Database

Although B+ tree database is slower than hash database, it features ordering access to each record. The order can be assigned by users. Records of B+ tree are sorted and arranged in logical pages. Sparse index organized in B tree that is multiway balanced tree are maintained for each page. Thus, the time complexity of retrieval and so on is "O(log n)". Cursor is provided to access each record in order. The cursor can jump to a position specified by a key and can step forward or backward from the current position. Because each page is arranged as double linked list, the time complexity of stepping cursor is "O(1)".

B+ tree database is implemented, based on the above hash database. Because each page of B+ tree is stored as each record of hash database, B+ tree database inherits efficiency of storage management of hash database. Because the header of each record is smaller and alignment of each page is adjusted according to the page size, in most cases, the size of database file is cut by half compared to one of hash database.

Although operations of many pages are required to update B+ tree, Kyoto Cabinet expedites the process by the page cache and reducing file operations. The page cache is implemented with double layered LRU list, which realizes that frequently accessed pages are cached in the "hot" list and recently accessed pages are cached in the "warm" LRU list. If the page cache works efficiently and the whole of the sparse index is cached on memory, it is possible to retrieve a record by one or less set of file operations.

Each page of B+ tree can be stored with compressed. The default compression method is "Deflate" by ZLIB. Because records in a page has similar patterns, high efficiency of compression is expected due to the Lempel-Ziv algorithm. In case of handling text data, the size of a database is reduced to about 50% or less. If the scale of a database is large and disk I/O is the bottleneck, featuring compression makes the processing speed improved to a large extent. Moreover, you can specify such external compression algorithms as LZO and LZMA.

### Practical Functionality

Kyoto Cabinet features transaction mechanisms. It is possible to commit a series of operations between the beginning and the end of the transaction in a lump, or to abort the transaction and perform rollback to the state before the transaction. Two isolation levels are supported; "serializable" and "read uncommitted". Durability is secured by write ahead logging and shadow paging.

Auto transaction and auto recovery mechanisms are also supported. If the auto transaction option is specified when opening the database, every updating operation is guarded by transaction which is committed implicitly. Therefore, durability can be assured without explicit transaction operations. The auto recovery mechanism works after the database is crashed outside transaction. If inconsistency of the database is detected when opening the database, all regions are scanned as with "fsck" and the database is reconstructed with surviving records implicitly.

Kyoto Cabinet provides two modes to connect to a database: "reader" and "writer". A reader can perform retrieving but neither storing nor deleting. A writer can perform all access methods. Exclusion control between processes is performed when connecting to a database by file locking. While a writer is connected to a database, neither readers nor writers can be connected. While a reader is connected to a database, other readers can be connect, but writers can not. According to this mechanism, data consistency is guaranteed with simultaneous connections in multitasking environment.

Functions of API are reentrant and available in multi-thread environment. Different database objects can be operated in parallel entirely. For simultaneous operations against the same database object, rwlock (reader-writer lock) is used for exclusion control. That is, while a writing thread is operating an object, other reading threads and writing threads are blocked. However, while a reading thread is operating an object, reading threads are not blocked. Locking granularity depends on data structures. Hash database uses record locking. B+ tree database uses page locking.

In order to improve performance and concurrency, Kyoto Cabinet uses such atomic operations built in popular CPUs as atomic-increment and CAS (compare-and-swap). Lock primitives provided by the native environment such as the POSIX thread package are alternated by own primitives using CAS.

### Simple but Flexible Interfaces

Kyoto Cabinet provides simple APIs based on object-oriented design. Every operation for database is encapsulated and published as lucid methods as `open', `close', `set', `remove', `get', and so on. The classes of hash database and B+ tree database are derived class of the common abstract class which defines the interface. Porting an application from one database to another is easy. Moreover, the polymorphic database API is provided to assign a database in run-time.

Kyoto Cabinet supports the "visitor" pattern. You can define arbitrary database operations with call back functions. The visitor class encapsulates that call back functions and their state data. The database class has the "accept" method, which accepts an instance of the visitor class and calls its functions with a record data. The return value of the call back function is reflected as the new state of the record.

While the core API is provided for C++, bindings for other languages such as C, Java, Python, Ruby, and Perl are also provided. Command line interfaces are also provided corresponding to each API. They are useful for prototyping, test, and debugging.

## Installation

This section describes how to install Kyoto Cabinet with the source package. As for a binary package, see its installation manual.

### Preparation

Kyoto Cabinet is available on UNIX-like systems. At least, the following environments are supported.

- Linux 2.6 and later (i386/x86-64/PowerPC/Alpha/SPARC)
- FreeBSD 7.1 and later (i386/x86-64)
- Solaris 10 and later (i386/x86-64/SPARC)
- Mac OS X 10.5 and later (x86-64)
- Windows XP and later (i386/x86-64)

`gcc` (GNU Compiler Collection) 4.2 or later and `make` (GNU Make) are required to install Kyoto Cabinet with the source package. They are installed by default on Linux, FreeBSD and so on.

As Kyoto Cabinet depends on the following libraries, install them beforehand.

- ZLIB : for loss-less data compression. 1.2.3 or later is required.

### Installation

When an archive file of Kyoto Cabinet is extracted, change the current working directory to the generated directory and perform installation.

Run the configuration script.

    $ ./configure

Build programs.

    $ make

Perform self-diagnostic test.

    $ make check

Install programs. This operation must be carried out by the `root` user.

    # make install

### Result

When a series of work finishes, the following files will be installed.

    /usr/local/include/kccommon.h
    /usr/local/include/kcutil.h
    /usr/local/include/kcdb.h
    /usr/local/include/kcthread.h
    /usr/local/include/kcfile.h
    /usr/local/include/kccompress.h
    /usr/local/include/kccompare.h
    /usr/local/include/kcmap.h
    /usr/local/include/kcprotodb.h
    /usr/local/include/kccachedb.h
    /usr/local/include/kchashdb.h
    /usr/local/include/kctreedb.h
    /usr/local/include/kcpolydb.h
    /usr/local/include/kclangc.h
    /usr/local/lib/libkyotocabinet.a
    /usr/local/lib/libkyotocabinet.so.x.y.z
    /usr/local/lib/libkyotocabinet.so.x
    /usr/local/lib/libkyotocabinet.so
    /usr/local/lib/pkgconfig/kyotocabinet.pc
    /usr/local/bin/kcutiltest
    /usr/local/bin/kcutilcodec
    /usr/local/bin/kcprototest
    /usr/local/bin/kccachetest
    /usr/local/bin/kchashtest
    /usr/local/bin/kchashmgr
    /usr/local/bin/kctreetest
    /usr/local/bin/kctreemgr
    /usr/local/bin/kcpolytest
    /usr/local/bin/kcpolymgr
    /usr/local/bin/kclanctest
    /usr/local/share/kyotocabinet/...
    /usr/local/man/man1/...

### Options of Configure

The following options can be specified with ``./configure`'.

- `--enable-debug` : build for debugging. Enable debugging symbols, do not perform optimization, and perform static linking.
- `--enable-devel` : build for development. Enable debugging symbols, perform optimization, and perform dynamic linking.
- `--enable-profile` : build for profiling. Enable profiling symbols, perform optimization, and perform dynamic linking.
- `--enable-static` : build by static linking.
- `--disable-shared` : avoid to build shared libraries.
- `--disable-zlib` : build without ZLIB compression.
- `--disable-atomic` : build without atomic operations.

``--prefix`' and other options are also available as with usual UNIX software packages. If you want to install Kyoto Cabinet under ``/usr`' not ``/usr/local`', specify ``--prefix=/usr`'. As well, the library search path does not include ``/usr/local/lib`', it is necessary to set the environment variable ``LD_LIBRARY_PATH`' to include ``/usr/local/lib`' before running applications of Kyoto Cabinet.

### How to Use the Library

Kyoto Cabinet provides API of the C++ language and it is available by programs conforming to the C++03 standard. As the header files of Kyoto Cabinet are provided as ``kcutil.h`', ``kchashdb.h`', and so on, applications should include one or more of them accordingly to use the API. As the library is provided as ``libkyotocabinet.a`' and ``libkyotocabinet.so`' and they depends ``libz.so`', ``libstdc++.so`', ``librt.so`', ``libpthread.so`', ``libm.so`', and ``libc.so`', linker options corresponding to them are required by the build command. The typical build command is the following.

    $ g++ -I/usr/local/include example.cc -o example \
      -L/usr/local/lib -lkyotocabinet -lz -lstdc++ -lrt -lpthread -lm -lc

### For Windows

Microsoft Visual Studio (Visual C++) is required to build Kyoto Cabinet on Windows. The building configuration is described in the file ``VCmakefile`', which should be edited according to your environment. Then, perform the following command in a command prompt window.

    > nmake -f VCmakefile

If you want, perform self-diagnostic test.

    > nmake -f VCmakefile check

If all building processes finish successfully, the static library ``kyotocabinet.lib`' and some executable files are generated. As for now, neither DLL nor installation tool is provided. Please, install the header files, the library file, and the executable files by yourself.

If the header files and the library file are in the current directory, you can build an application program by the following command.

    > cl /I. example.cc kyotocabinet.lib

By default, the library is built with linking to ``LIBCMT.LIB`' by the ``/MT`' option. If you want to use ``MSVCRT.LIB`', which is required by the MFC, rebuild the library with the ``/MD`' option and set the same option when building applications. If your environment is 64-bit version, add the ``/D_SYS_WIN64_`' option to compiler options to improve performance.

## Tutorial

This section describes how to use Kyoto Cabinet with the command line utilities and some sample application programs.

### Command Line Utility for the File Hash Database

To begin with, let's build a file hash database with the command line utility ``kchashmgr`'. The database stores a list of staffs of a company. Each record is composed of the key and the value. The key is the staff ID and the value is person's name.

The database must be created just one time before any database operation. Let's create a database file "casket.kch" with the default configuration.

    $ kchashmgr create staffs.kch

Register some staffs into the database.

    $ kchashmgr set staffs.kch 1001 "George Washington"
    $ kchashmgr set staffs.kch 1002 "John Adams"
    $ kchashmgr set staffs.kch 1003 "Thomas Jefferson"
    $ kchashmgr set staffs.kch 1004 "James Madison"

Check the current contents.

    $ kchashmgr list -pv staffs.kch
    1001    George Washington
    1002    John Adams
    1003    Thomas Jefferson
    1004    James Madison

To retrieve the value of a record, search the database with the key.

    $ kchashmgr get staffs.kch 1003
    Thomas Jefferson

Of course, you can remove a record with the key.

    $ kchashmgr remove staffs.kch 1003

That's all for the fundamental operations. The DBM family have been improving performance thanks to discarding the functionality.

### Sample Application of the File Hash Database

Next, let's write a sample application program handling a file hash database. See the following source code.

    #include <kchashdb.h>

    using namespace std;
    using namespace kyotocabinet;

    // main routine
    int main(int argc, char** argv) {

      // create the database object
      HashDB db;

      // open the database
      if (!db.open("casket.kch", HashDB::OWRITER | HashDB::OCREATE)) {
        cerr << "open error: " << db.error().name() << endl;
      }

      // store records
      if (!db.set("foo", "hop") ||
          !db.set("bar", "step") ||
          !db.set("baz", "jump")) {
        cerr << "set error: " << db.error().name() << endl;
      }

      // retrieve a record
      string* value = db.get("foo");
      if (value) {
        cout << *value << endl;
        delete value;
      } else {
        cerr << "get error: " << db.error().name() << endl;
      }

      // traverse records
      DB::Cursor* cur = db.cursor();
      cur->jump();
      pair<string, string>* rec;
      while ((rec = cur->get_pair(true)) != NULL) {
        cout << rec->first << ":" << rec->second << endl;
        delete rec;
      }
      delete cur;

      // close the database
      if (!db.close()) {
        cerr << "close error: " << db.error().name() << endl;
      }

      return 0;
    }

Save the above code as a file "example.cc". Then, perform the following command line. The command ``kcutilcodec conf`' prints the building configuration.

    $ g++ `kcutilcodec conf -i` -o example example.cc `kcutilcodec conf -l`

Execute the application program built by the above.

    $ ./example
    hop
    foo:hop
    bar:step
    baz:jump

The API of the file hash database is defined in the header ``kchash.h`'. So, include the header near the front of a source file. All symbols of Kyoto Cabinet are packaged in the name space ``kyotocabinet`'. You can use them without any prefix by importing the name space.

    #include <kchashdb.h>
    using namespace kyotocabinet;

The class ``HashDB`' contains all functionality of the file hash database and each instance expresses a file hash database file.

    HashDB db;

Each database file must be opened by the ``open`' method before any database operation. The flag ``HashDB::OWRITER`' means the process will update the database. The flag ``HashDB::OCREATE`' means a database file will be created if it does not exist yet.

    db.open("casket.kch", HashDB::OWRITER | HashDB::OCREATE);

Every opened database must be closed by the ``close`' method when it is no longer in use. Closing the database is very important to avoid data corruption and memory leak.

    db.close();

To store a record, use the ``set`' method with the key and the value.

    db.put("foo", "hop");

To retrieve the value of a record, use the ``get`' method with the key. The return value is NULL if no record corresponds to the key. On success, the return value is the pointer to a dynamic object and it should be deleted explicitly after the use.

    string* value = db.get("foo", "hop");
    if (value) {
      cout << *value << endl;
      delete value;
    }

Except for ``set`' and ``get`', there are other methods; ``add`', ``append`', ``remove`', ``increment`', and ``cas`'. Each method has two versions; for ``std::string`' parameters and for ``char*`' and ``size_t`' parameters.

Traversing records is a bit complicated task. It needs a cursor object, which expresses the current position in the sequence of all records in the database. Each cursor is created by the ``cursor`' method of the database object. Each cursor should be initialized by ``jump`' method before actual record operations.

    DB::Cursor* cur = db.cursor();
    cur->jump();

The cursor class has such methods against the record at the current position as ``set_value`', ``remove`', ``get_key`', ``get_value`', and ``get_pair`'. Most methods have an optional stepping parameter to shift the current position to the next record atomically. Therefore, iterating such methods with the stepping parameter results in that all records are visited.

    pair<string, string>* rec;
    while ((rec = cur.get_pair(true)) != NULL) {
      cout << rec->first << ":" << rec->second << endl;
      delete rec;
    }

### More Complex Example Using the Visitor Pattern

Every operation against a record can be abstracted by the "visitor" pattern. A visitor object specifies call back methods which receives the state of a record and returns the new state. Let's see the next sample using the visitor pattern.

    #include <kchashdb.h>

    using namespace std;
    using namespace kyotocabinet;

    // main routine
    int main(int argc, char** argv) {

      // create the database object
      HashDB db;

      // open the database
      if (!db.open("casket.kch", HashDB::OREADER)) {
        cerr << "open error: " << db.error().name() << endl;
      }

      // define the visitor
      class VisitorImpl : public DB::Visitor {
        // call back function for an existing record
        const char* visit_full(const char* kbuf, size_t ksiz,
                               const char* vbuf, size_t vsiz, size_t *sp) {
          cout << string(kbuf, ksiz) << ":" << string(vbuf, vsiz) << endl;
          return NOP;
        }
        // call back function for an empty record space
        const char* visit_empty(const char* kbuf, size_t ksiz, size_t *sp) {
          cerr << string(kbuf, ksiz) << " is missing" << endl;
          return NOP;
        }
      } visitor;

      // retrieve a record with visitor
      if (!db.accept("foo", 3, &visitor, false) ||
          !db.accept("dummy", 5, &visitor, false)) {
        cerr << "accept error: " << db.error().name() << endl;
      }

      // traverse records with visitor
      if (!db.iterate(&visitor, false)) {
        cerr << "iterate error: " << db.error().name() << endl;
      }

      // close the database
      if (!db.close()) {
        cerr << "close error: " << db.error().name() << endl;
      }

      return 0;
    }

The methods ``accept`' and ``iterate`' receive visitor objects. Each visitor object must be implement the interface ``DB::Visitor`'. The method ``visit_full`' is called for an existing record. The parameters specify the pointer to key buffer, the length of the key buffer, the pointer to the value buffer, the length of the value buffer, and the pointer to the variable to notice the length of the return value. The return value is ``NOP`' for no update, ``REMOVE`' to remove the record, or otherwise the pointer to the buffer contains arbitrary byte pattern to update the value. The method ``visit_empty`' is called for an empty record space. The specification is the same to ``visit_full`' except that it does not receive the record value.

As it is, almost all built-in operations like ``set`', ``remove`', and ``get`' are implemented with the ``accept`' method. The following code is to store a record.

    class Setter : public DB::Visitor {
    public:
      Setter(const char* vbuf, size_t vsiz) : vbuf_(vbuf), vsiz_(vsiz) {}
    private:
      const char* visit_full(const char* kbuf, size_t ksiz,
                             const char* vbuf, size_t vsiz, size_t* sp) {
        *sp = vsiz_;
        return vbuf_;
      }
      const char* visit_empty(const char* kbuf, size_t ksiz, size_t* sp) {
        *sp = vsiz_;
        return vbuf_;
      }
      const char* vbuf_;
      size_t vsiz_;
    };
    Setter setter("foo", 3);
    accept(kbuf, ksiz, &setter, true);

### Features of Various Database Classes

As Kyoto Cabinet provides various database classes, they have the common base class, which defines the common interface of all database classes. That is, you can use the following classes in the same way.

class | file | description
--- | --- | ---
`ProtoHashDB` | kcprotodb.h | prototype hash database. on-memory database implemented with std::unorderd_map of STL.
`ProtoTreeDB` | kcprotodb.h | prototype tree database. on-memory database implemented with std::map of STL.
`CacheDB` | kccachedb.h | cache database. on-memory database featuring LRU deletion.
`HashDB` | kchashdb.h | file hash database. file database of hash table; typical DBM.
`TreeDB` | kctreedb.h | file tree database. file database of B+ tree; DBM with order.
`PolyDB` | kcpolydb.h | polymorphic database. dynamic binding of the above databases.

Each database has different features in persistence, algorithm, time complexity, record sequence, and lock model for concurrency.

| `ProtoHashDB` | `ProtoTreeDB` | `CacheDB` | `HashDB` | `TreeDB`
persistence | volatile | volatile | volatile | persistent | persistent
algorithm | hash table | red black tree | hash table | hash table | B+ tree
complexity | O(1) | O(log N) | O(1) | O(1) | O(log N)
sequence | undefined | lexical order | undefined | undefined | custom order
lock unit | whole (rwlock) | whole (rwlock) | record (mutex) | record (rwlock) | page (rwlock)

The C language binding is also provided as a wrapper of the polymorphic database API. Include the header file ``kclangc.h`' and use the pointer to ``KCDB`' as a database object.

Please see the the API documents for detail. Writing your own sample application is the best way to learn this library.

## Command Line Utilities

This section describes how to use command line utilities. They are useful to manage database contents and to test the library and its applications.

### kcutiltest

The command ``kcutiltest`' is a utility for facility test and performance test of the utility functions. This command is used in the following format. `rnum' specifies the number of iterations. `path' specifies the path of a file.

`kcutiltest mutex [-th num] [-iv num] rnum`
  Performs test of lock primitives.
`kcutiltest file [-th num] [-rnd] [-msiz num] path rnum`
  Performs test of the filesystem abstraction.
`kcutiltest map [-rnd] rnum`
  Performs test of data mapping structures.
`kcutiltest misc rnum`
  Performs test of miscellaneous mechanisms.

Options feature the following.

- `-th num` : specifies the number of worker threads.
- `-iv num` : specifies the interval between iterations.
- `-rnd` : performs random test.
- `-msiz num` : specifies the size of the memory-mapped region.

This command returns 0 on success, another on failure.

### kcutilcodec

The command ``kcutilcodec`' is a tool to use encoding and decoding features, or to show the configuration. This command is used in the following format.

`kcutilcodec conf [-v|-i|-l|-p]`
  Shows the configuration of Kyoto Cabinet.

Options feature the following.

- `-v` : show the version number of Kyoto Cabinet.
- `-i` : show options to include the headers of Tokyo Cabinet.
- `-l` : show options to link the library of Tokyo Cabinet.
- `-p` : show the directory path of the commands.

This command returns 0 on success, another on failure.

### kcprototest

The command ``kcprototest`' is a utility for facility test and performance test of the prototype database. This command is used in the following format. `rnum' specifies the number of iterations.

`kcprototest order [-tree] [-th num] [-rnd] [-etc] [-tran] rnum`
  Performs in-order tests.
`kcprototest queue [-tree] [-th num] [-it num] [-rnd] rnum`
  Performs queuing operations.
`kcprototest wicked [-tree] [-th num] [-it num] rnum`
  Performs mixed operations selected at random.
`kcprototest tran [-tree] [-th num] [-it num] rnum`
  Performs test of transaction.

Options feature the following.

- `-tree` : test the prototype tree database instead of the prototype hash database.
- `-th num` : specifies the number of worker threads.
- `-rnd` : performs random test.
- `-etc` : performs miscellaneous operations.
- `-tran` : performs transaction.
- `-it num` : specifies the number of repetition.

This command returns 0 on success, another on failure.

### kccachetest

The command ``kccachetest`' is a utility for facility test and performance test of the cache database. This command is used in the following format. `rnum' specifies the number of iterations.

`kccachetest order [-th num] [-rnd] [-etc] [-tran] [-bnum num] [-capcnt num] [-capsiz num] rnum`
  Performs in-order tests.
`kccachetest queue [-th num] [-it num] [-rnd] [-bnum num] [-capcnt num] [-capsiz num] rnum`
  Performs queuing operations.
`kccachetest wicked [-th num] [-it num] [-bnum num] [-capcnt num] [-capsiz num] rnum`
  Performs mixed operations selected at random.
`kccachetest tran [-th num] [-it num] [-bnum num] [-capcnt num] [-capsiz num] rnum`
  Performs test of transaction.

Options feature the following.

- `-th num` : specifies the number of worker threads.
- `-rnd` : performs random test.
- `-etc` : performs miscellaneous operations.
- `-tran` : performs transaction.
- `-bnum num` : specifies the number of buckets of the hash table.
- `-capcnt num` : specifies the maximum number of records.
- `-capsiz num` : specifies the maximum size of memory usage.
- `-it num` : specifies the number of repetition.

This command returns 0 on success, another on failure.

### kchashtest

The command ``kchashtest`' is a utility for facility test and performance test of the file hash database. This command is used in the following format. `path' specifies the path of a database file. `rnum' specifies the number of iterations.

`kchashtest order [-th num] [-rnd] [-set|-get|-getw|-rem|-etc] [-tran] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-msiz num] [-dfunit num] [-erv] path rnum`
  Performs in-order tests.
`kchashtest queue [-th num] [-it num] [-rnd] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-msiz num] [-dfunit num] [-erv] path rnum`
  Performs queuing operations.
`kchashtest wicked [-th num] [-it num] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-msiz num] [-dfunit num] [-erv] path rnum`
  Performs mixed operations selected at random.
`kchashtest tran [-th num] [-it num] [-hard] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-msiz num] [-dfunit num] [-erv] path rnum`
  Performs test of transaction.

Options feature the following.

- `-th num` : specifies the number of worker threads.
- `-rnd` : performs random test.
- `-set` : performs setting operation only.
- `-get` : performs getting operation only.
- `-getw` : performs getting with a buffer operation only.
- `-rem` : performs removing operation only.
- `-etc` : performs miscellaneous operations.
- `-tran` : performs transaction.
- `-oat` : opens the database with the auto transaction option.
- `-oas` : opens the database with the auto synchronization option.
- `-onl` : opens the database with the no locking option.
- `-otl` : opens the database with the try locking option.
- `-onr` : opens the database with the no auto repair option.
- `-apow num` : specifies the power of the alignment of record size.
- `-fpow num` : specifies the power of the capacity of the free block pool.
- `-ts` : tunes the database with the small option.
- `-tl` : tunes the database with the linear option.
- `-tc` : tunes the database with the compression option.
- `-bnum num` : specifies the number of buckets of the hash table.
- `-msiz num` : specifies the size of the memory-mapped region.
- `-dfunit num` : specifies the unit step number of auto defragmentation.
- `-erv` : reports all errors.
- `-it num` : specifies the number of repetition.
- `-hard` : performs physical synchronization.

This command returns 0 on success, another on failure.

### kchashmgr

The command ``kchashmgr`' is a utility for test and debugging of the file hash database and its applications. `path' specifies the path of a database file. `key' specifies the key of a record. `value' specifies the value of a record. `file' specifies the input file.

`kchashmgr create [-otr] [-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] path`
  Creates a database file.
`kchashmgr inform [-onl|-otl|-onr] [-st] path`
  Prints status information.
`kchashmgr set [-onl|-otl|-onr] [-add|-app|-inci|-incd] [-sx] path key value`
  Stores a record.
`kchashmgr remove [-onl|-otl|-onr] [-sx] path key`
  Removes a record.
`kchashmgr get [-onl|-otl|-onr] [-sx] [-px] [-pz] path key`
  Prints the value of a record.
`kchashmgr list [-onl|-otl|-onr] [-max num] [-sx] [-pv] [-px] path [key]`
  Prints keys of all records, separated by line feeds.
`kchashmgr import [-onl|-otl|-onr] [-sx] path [file]`
  Imports records from a TSV file.
`kchashmgr dump [-onl|-otl|-onr] path [file]`
  Dump records into a snapshot file.
`kchashmgr load [-otr] [-onl|-otl|-onr] path [file]`
  Load records from a snapshot file.
`kchashmgr defrag [-onl|-otl|-onr] path`
  Performs defragmentation.
`kchashmgr check [-onl|-otl|-onr] path`
  Checks consistency.

Options feature the following.

- `-otr` : opens the database with the truncation option.
- `-onl` : opens the database with the no locking option.
- `-otl` : opens the database with the try locking option.
- `-onr` : opens the database with the no auto repair option.
- `-apow num` : specifies the power of the alignment of record size.
- `-fpow num` : specifies the power of the capacity of the free block pool.
- `-ts` : tunes the database with the small option.
- `-tl` : tunes the database with the linear option.
- `-tc` : tunes the database with the compression option.
- `-bnum num` : specifies the number of buckets of the hash table.
- `-st` : prints miscellaneous information.
- `-add` : performs adding operation.
- `-app` : performs appending operation.
- `-inci` : performs integer increment operation.
- `-incd` : performs real number increment operation.
- `-sx` : the input data is evaluated as a hexadecimal data string.
- `-px` : the output data is converted into a hexadecimal data string.
- `-pz` : does not append line feed at the end of the output.
- `-max` : specifies the maximum number of shown records.
- `-pv` : prints values of records also.

This command returns 0 on success, another on failure.

### kctreetest

The command ``kctreetest`' is a utility for facility test and performance test of the file tree database. This command is used in the following format. `path' specifies the path of a database file. `rnum' specifies the number of iterations.

`kctreetest order [-th num] [-rnd] [-set|-get|-getw|-rem|-etc] [-tran] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-msiz num] [-dfunit num] [-pccap num] [-rcd] [-erv] path rnum`
  Performs in-order tests.
`kctreetest queue [-th num] [-it num] [-rnd] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-msiz num] [-dfunit num] [-pccap num] [-rcd] [-erv] path rnum`
  Performs queuing operations.
`kctreetest wicked [-th num] [-it num] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-msiz num] [-dfunit num] [-pccap num] [-rcd] [-erv] path rnum`
  Performs mixed operations selected at random.
`kctreetest tran [-th num] [-it num] [-hard] [-oat|-onl|-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-msiz num] [-dfunit num] [-pccap num] [-rcd] [-erv] path rnum`
  Performs test of transaction.

Options feature the following.

- `-th num` : specifies the number of worker threads.
- `-rnd` : performs random test.
- `-set` : performs setting operation only.
- `-get` : performs getting operation only.
- `-getw` : performs getting with a buffer operation only.
- `-rem` : performs removing operation only.
- `-etc` : performs miscellaneous operations.
- `-tran` : performs transaction.
- `-oat` : opens the database with the auto transaction option.
- `-oas` : opens the database with the auto synchronization option.
- `-onl` : opens the database with the no locking option.
- `-otl` : opens the database with the try locking option.
- `-onr` : opens the database with the no auto repair option.
- `-apow num` : specifies the power of the alignment of record size.
- `-fpow num` : specifies the power of the capacity of the free block pool.
- `-ts` : tunes the database with the small option.
- `-tl` : tunes the database with the linear option.
- `-tc` : tunes the database with the compression option.
- `-bnum num` : specifies the number of buckets of the hash table.
- `-psiz num` : specifies the size of each page.
- `-msiz num` : specifies the size of the memory-mapped region.
- `-dfunit num` : specifies the unit step number of auto defragmentation.
- `-pccap num` : specifies the capacity size of the page cache.
- `-rcd` : use the decimal comparator instead of the lexical one.
- `-erv` : reports all errors.
- `-it num` : specifies the number of repetition.
- `-hard` : performs physical synchronization.

This command returns 0 on success, another on failure.

### kctreemgr

The command ``kctreemgr`' is a utility for test and debugging of the file tree database and its applications. `path' specifies the path of a database file. `key' specifies the key of a record. `value' specifies the value of a record. `file' specifies the input file.

`kctreemgr create [-otr] [-onl|-otl|-onr] [-apow num] [-fpow num] [-ts] [-tl] [-tc] [-bnum num] [-psiz num] [-rcd] path`
  Creates a database file.
`kctreemgr inform [-onl|-otl|-onr] [-st] path`
  Prints status information.
`kctreemgr set [-onl|-otl|-onr] [-add|-app|-inci|-incd] [-sx] path key value`
  Stores a record.
`kctreemgr remove [-onl|-otl|-onr] [-sx] path key`
  Removes a record.
`kctreemgr get [-onl|-otl|-onr] [-sx] [-px] [-pz] path key`
  Prints the value of a record.
`kctreemgr list [-onl|-otl|-onr] [-max num] [-sx] [-pv] [-px] path [key]`
  Prints keys of all records, separated by line feeds.
`kctreemgr import [-onl|-otl|-onr] [-sx] path [file]`
  Imports records from a TSV file.
`kctreemgr dump [-onl|-otl|-onr] path [file]`
  Dump records into a snapshot file.
`kctreemgr load [-otr] [-onl|-otl|-onr] path [file]`
  Load records from a snapshot file.
`kctreemgr defrag [-onl|-otl|-onr] path`
  Performs defragmentation.
`kctreemgr check [-onl|-otl|-onr] path`
  Checks consistency.

Options feature the following.

- `-otr` : opens the database with the truncation option.
- `-onl` : opens the database with the no locking option.
- `-otl` : opens the database with the try locking option.
- `-onr` : opens the database with the no auto repair option.
- `-apow num` : specifies the power of the alignment of record size.
- `-fpow num` : specifies the power of the capacity of the free block pool.
- `-ts` : tunes the database with the small option.
- `-tl` : tunes the database with the linear option.
- `-tc` : tunes the database with the compression option.
- `-bnum num` : specifies the number of buckets of the hash table.
- `-psiz num` : specifies the size of each page.
- `-rcd` : use the decimal comparator instead of the lexical one.
- `-st` : prints miscellaneous information.
- `-add` : performs adding operation.
- `-app` : performs appending operation.
- `-inci` : performs integer increment operation.
- `-incd` : performs real number increment operation.
- `-sx` : the input data is evaluated as a hexadecimal data string.
- `-px` : the output data is converted into a hexadecimal data string.
- `-pz` : does not append line feed at the end of the output.
- `-max` : specifies the maximum number of shown records.
- `-pv` : prints values of records also.

This command returns 0 on success, another on failure.

### kcpolytest

The command ``kcpolytest`' is a utility for facility test and performance test of the polymorphic database. This command is used in the following format. `path' specifies the path of a database file. `rnum' specifies the number of iterations.

`kcpolytest order [-th num] [-rnd] [-set|-get|-getw|-rem|-etc] [-tran] [-oat|-onl|-onl|-otl|-onr] path rnum`
  Performs in-order tests.
`kcpolytest queue [-th num] [-it num] [-rnd] [-oat|-onl|-onl|-otl|-onr] path rnum`
  Performs queuing operations.
`kcpolytest wicked [-th num] [-it num] [-oat|-onl|-onl|-otl|-onr] path rnum`
  Performs mixed operations selected at random.
`kcpolytest tran [-th num] [-it num] [-hard] [-oat|-onl|-onl|-otl|-onr] path rnum`
  Performs test of transaction.
`kcpolytest misc path`
  Performs miscellaneous tests.

Options feature the following.

- `-th num` : specifies the number of worker threads.
- `-rnd` : performs random test.
- `-set` : performs setting operation only.
- `-get` : performs getting operation only.
- `-getw` : performs getting with a buffer operation only.
- `-rem` : performs removing operation only.
- `-etc` : performs miscellaneous operations.
- `-tran` : performs transaction.
- `-oat` : opens the database with the auto transaction option.
- `-oas` : opens the database with the auto synchronization option.
- `-onl` : opens the database with the no locking option.
- `-otl` : opens the database with the try locking option.
- `-onr` : opens the database with the no auto repair option.
- `-it num` : specifies the number of repetition.
- `-hard` : performs physical synchronization.

This command returns 0 on success, another on failure.

### kcpolymgr

The command ``kcpolymgr`' is a utility for test and debugging of the file hash database and its applications. `path' specifies the path of a database file. `key' specifies the key of a record. `value' specifies the value of a record. `file' specifies the input file.

`kcpolymgr create [-otr] [-onl|-otl|-onr] path`
  Creates a database file.
`kcpolymgr inform [-onl|-otl|-onr] [-st] path`
  Prints status information.
`kcpolymgr set [-onl|-otl|-onr] [-add|-app|-inci|-incd] [-sx] path key value`
  Stores a record.
`kcpolymgr remove [-onl|-otl|-onr] [-sx] path key`
  Removes a record.
`kcpolymgr get [-onl|-otl|-onr] [-sx] [-px] [-pz] path key`
  Prints the value of a record.
`kcpolymgr list [-onl|-otl|-onr] [-max num] [-sx] [-pv] [-px] path [key]`
  Prints keys of all records, separated by line feeds.
`kcpolymgr import [-onl|-otl|-onr] [-sx] path [file]`
  Imports records from a TSV file.
`kcpolymgr dump [-onl|-otl|-onr] path [file]`
  Dump records into a snapshot file.
`kcpolymgr load [-otr] [-onl|-otl|-onr] path [file]`
  Load records from a snapshot file.
`kcpolymgr defrag [-onl|-otl|-onr] path`
  Performs defragmentation.
`kcpolymgr check [-onl|-otl|-onr] path`
  Checks consistency.

Options feature the following.

- `-otr` : opens the database with the truncation option.
- `-onl` : opens the database with the no locking option.
- `-otl` : opens the database with the try locking option.
- `-onr` : opens the database with the no auto repair option.
- `-st` : prints miscellaneous information.
- `-add` : performs adding operation.
- `-app` : performs appending operation.
- `-inci` : performs integer increment operation.
- `-incd` : performs real number increment operation.
- `-sx` : the input data is evaluated as a hexadecimal data string.
- `-px` : the output data is converted into a hexadecimal data string.
- `-pz` : does not append line feed at the end of the output.
- `-max` : specifies the maximum number of shown records.
- `-pv` : prints values of records also.

### kclangctest

The command ``kclangctest`' is a utility for facility test and performance test of the C language binding. This command is used in the following format. `path' specifies the path of a database file. `rnum' specifies the number of iterations.

`kcpolytest order [-rnd] [-etc] [-tran] [-oat|-onl|-onl|-otl|-onr] path rnum`
  Performs in-order tests.

Options feature the following.

- `-rnd` : performs random test.
- `-etc` : performs miscellaneous operations.
- `-tran` : performs transaction.
- `-oat` : opens the database with the auto transaction option.
- `-oas` : opens the database with the auto synchronization option.
- `-onl` : opens the database with the no locking option.
- `-otl` : opens the database with the try locking option.
- `-onr` : opens the database with the no auto repair option.

This command returns 0 on success, another on failure.

## Tips and Hacks

This section describes tips and hacks to use Kyoto Cabinet.

### Tuning the Cache Database

The cache database is on-memory database featuring LRU deletion. The following tuning methods are provided.

- `tune_buckets` : sets the number of buckets of the hash table.
- `cap_count` : sets the capacity by record number.
- `cap_size` : sets the capacity by memory usage.

By default, the cache database maintains all records on memory and no record is expired. If you want to expire old records to keep the memory usage constant, call ``cap_count`' and/or ``cap_size`' to limit the capacity.

The default tuning of the bucket number is about one million. If you intend to store more records, call ``tune_buckets`' to set the bucket number. The suggested ratio of the bucket number is the same to the total number of records and it is okay from 50% to 400%. If the ratio decreases smaller than 50%, the time efficiency will decrease gradually.

If you want to cache ten millions of records and keep the memory usage less than 8GB, the following tuning is suggested for example.

    db.tune_buckets(10LL * 1000 * 1000);
    db.cap_count(10LL * 1000 * 1000);
    db.cap_size(8LL << 30);
    db.open(...);

All tuning methods must be called before the database is opened.

### Tuning the File Hash Database

The file hash database is file database of hash table. The following tuning methods are provided.

- `tune_alignment` : sets the power of the alignment of record size.
- `tune_fbp` : sets the power of the capacity of the free block pool.
- `tune_options` : sets the optional features.
- `tune_buckets` : sets the number of buckets of the hash table.
- `tune_map` : sets the size of the internal memory-mapped region.
- `tune_defrag` : sets the unit step number of auto defragmentation.
- `tune_compressor` : set the data compressor.

The default alignment power is 3, which means the address of each record is aligned to a multiple of 8 (1<<3) bytes. If you trust that the database is constructed at a time and not updated often, call ``tune_alignment`' to set the alignment power 0, which means 1 (1<<0) byte. If the typical size of each record is expected to be larger than 1KB, tune the alignment 8 or more.

The tuning of the free block pool by ``tune_fbp`' does not have to be modified in most cases. The default is 10, which means the capacity of the free block pool is 1024 (1<<10).

The optional features by ``tune_options`' is useful to reduce the size of the database fine at the expense of scalability or time efficiency. If ``HashDB::TSMALL`' is specified, the width of record addressing is reduced from 6 bytes to 4 bytes. As the result, the footprint for each record is reduced from 16 bytes to 12 bytes. However, it limits the maximum size of the database file up to 16GB (2GB multiplied by the alignment). If ``HashDB::TLINEAR`' is specified, the data structure of the collision chain of hash table is changed from binary tree to linear linked list. In that case, the footprint of each record is reduced from 16 bytes to 10 bytes although the time efficiency becomes sensitive to the number of the hash buckets. If ``HashDB::TCOMPRESS`' is specified, the value of each record is compressed implicitly when stored in the file. If the value is bigger than 1KB or more, compression is effective.

The default tuning of the bucket number is about one million. If you intend to store more records, call ``tune_buckets`' to set the bucket number. The suggested ratio of the bucket number is the twice of the total number of records and it is okay from 100% to 400%. If the ratio decreases smaller than 100%, the time efficiency will decrease gradually.

The default tuning of the size of the internal memory-mapped region is 64MB. If the database size is expected to be larger than 64MB, call ``tune_map` to set the map size larger than the expected size of the database. Although the capacity of the RAM on the machine limits the map size, increasing the map size is effective to improve performance.

By default, auto defragmentation is disabled. If the existing records in the database are modified (removed or modified with varying the size), fragmentation of available regions proceeds gradually. In that case, call ``tune_defrag`' to enable auto defragmentation and set the unit step number. The suggested unit step number is 8, which means that a set of defragmentation operations is performed each 8 updating operations. The more the unit is, space efficiency becomes higher but time efficiency becomes lower.

The default compression algorithm of the ``HashDB::TLINEAR`' option is "Deflate" by ZLIB. If you want to use another algorithm, call ``tune_compressor`' to set a functor which implements compression and decompression functions.

If you intend to store ten thousands of records and reduce the database size as possible, the following tuning is suggested for example.

    db.tune_alignment(0);
    db.tune_buckets(10LL * 1000);
    db.tune_options(HashDB::TSMALL | HashDB::TLINEAR);
    db.tune_defrag(8);
    db.open(...);

If you have a monster machine and intend to store ten billion records and improve time efficiency as possible, the following tuning is suggested for example.

    db.tune_buckets(20LL * 1000 * 1000 * 1000);
    db.tune_map(300LL << 30);
    db.open(...);

All tuning methods must be called before the database is opened. Because the settings of ``tune_alignment`', ``tune_fbp`', ``tune_options`', and ``tune_buckets`' are recorded as the meta data of the database, the methods are called before the database is created and they can not be modified afterward. Because other tuning parameters are not recorded in the database, they should be specified before every time opening the database.

### Tuning the File Tree Database

The file tree database is file database of B+ tree. Because each node of B+ tree is serialized as a page buffer and stored as a record of the file hash database, all tuning parameters of the file hash database is inherited to the file tree database. Moreover, the following tuning methods are added.

- `tune_page` : sets the size of each page.
- `tune_page_cache` : sets the capacity size of the page cache.
- `tune_comparator` : sets the record comparator.

The tuning of the page size by ``tune_page`' does not have to be modified in most cases. The default is 8192, which is the twice of the typical page size of popular environments. If the size of each node exceeds the parameter, the node is divided into two.

The default tuning of the capacity size of the page cache is 64MB. If your machine has abundant RAM, call ``tune_page`' to load all nodes on the page cache. If the RAM is not abundant, it is better to keep the default page cache size and assign the RAM for the internal memory-mapped region by ``tune_map`'.

The default record comparator is the lexical ordering function. That is, records in the B+ tree are placed in the lexical order of each key. If you want to use another ordering, call ``tune_comparator`' to set a functor which implements the ordering function.

The default alignment of the file tree database is 256 (1<<8). The default bucket number of the file tree database is about 65536. Other default tuning parameters are the same to the file hash database. Note that the bucket number should be calculated by the number of pages. The suggested ratio of the bucket number is about 10% of the number of records. If the compression option is specified, all records in each page are compressed at once. Therefore, compression is more effective for the file tree database rather than for the file hash database.

If you intend to store ten thousands of records and reduce the database size as possible, the following tuning is suggested for example.

    db.tune_buckets(1LL * 1000);
    db.tune_options(TreeDB::TCCOMPESS);
    db.tune_defrag(8);
    db.tune_page(32768);
    db.open(...);

If you have a monster machine and intend to store ten billion records and improve time efficiency as possible, the following tuning is suggested for example.

    db.tune_buckets(1LL * 1000 * 1000 * 1000);
    db.tune_map(300LL << 30);
    db.tune_page_cache(2LL << 30);
    db.open(...);

All tuning methods must be called before the database is opened. Because the setting of ``tune_page`' is recorded as the meta data of the database, the methods are called before the database is created and it can not be modified afterward. Because other tuning parameters are not recorded in the database, they should be specified before every time opening the database.

### Transaction

If an application process which opened a database terminated without closing the database, it involves risks that some records may be missing and the database may be broken. By default, durability is settled when the database is closed properly and it is not settled for each updating operation. Kyoto Cabinet deal with the problem by transaction mechanism based on WAL (write ahead logging). Transaction is begun and committed by application explicitly. Durability during transaction is settled for each updating operation. Transaction can be aborted by application. In that case, all update operations during transaction are voided and the content of the database is rollbacked. Although transaction is very useful like this, throughput of updating operation decreases to about 50% of the default manner due to overhead of writing WAL.

    db.begin_transaction();
    db.set("japan", "tokyo");
    db.set("korea", "seoul");
    db.end_transaction();

If you can't be bothered to begin and commit transaction explicitly, use auto transaction mechanism by specifying the ``FileDB::AUTOTRAN`' option when opening the database. Auto transaction is begun and committed implicitly for each operation. Overhead of auto transaction is lighter than explicit transaction.

    db.open("casket.kch", HashDB::OWRITER | HashDB::OCREATE | HashDB::OAUTOTRAN);
    db.set("japan", "tokyo");
    db.set("china", "beijing");

After all, it is important to choose the usage according to the requirements of your application.

default
  risk on process crash: Some records may be missing.
  risk on system crash: Some records may be missing.
  performance penalty: none
  remark: Auto recovery after crash will take time in proportion of the database size.

transaction
  implicit usage: open(..., FileDB::OAUTOTRAN);
  explicit usage: begin_transaction(false); ...; end_transaction(true);
  risk on process crash: none
  risk on system crash: Some records may be missing.
  performance penalty: Throughput will be down to about 50% or less.

transaction + synchronize
  implicit usage: open(..., FileDB::OAUTOTRAN | FileDB::OAUTOSYNC);
  explicit usage: begin_transaction(true); ...; end_transaction(true);
  risk on process crash: none
  risk on system crash: none
  performance penalty: Throughput will be down to about 1% or less.

### Backup

Any hardware will break down suddenly. Especially such storage devices as HDD and SSD are fragile. Therefore, making backup files of your database file periodically is very important even if you use transaction. You can copy a database file by such popular commands as ``cp`' and ``tar`' when the database is not being updated by another process.

If an application uses multi threads and you want to make a backup file of the database in safety, use the ``FileDB::copy`' method, which synchronizes the database status with the database file and makes a copy file. During the copying operation, it is assured that the database file is not updated.

    db.copy("backup.kch");

You may want "hot backup", which means that other threads are not blocked while a thread is creating a backup file. In that case, use the ``File::synchronize`' method which synchronizes the database file and calls a function defined arbitrary. The call back function can execute a "snapshot" command provided by the operating system.

    class BackupImpl : public FileProcessor {
      bool process(const std::string& path, int64_t size, int64_t count) {
        char cmd[1024];
        sprintf(cmd, "snapshot.sh %s", path.c_str());
        return system(cmd) == 0;
      }
    } proc;
    db.synchronize(&proc);

### Pseudo-snapshot

Chiefly for the cache database, the pseudo-snapshot mechanism is provided. The ``FileDB::dump_snapshot`' dumps all records into a stream or a file. The ``FileDB::load_snapshot`' method loads records from a stream or a file. Although the operations are performed atomically, they don't finish momentarily but take time in proportion of the database size with blocking the other threads. Because the format of pseudo-snapshot data is common among the all database classes, it is useful to migrate records for each other.

    db.dump_snapshot("backup.kcss");
    db.load_snapshot("backup.kcss");

If you don't want to let the other threads be blocked. Use the cursor mechanism and save/load records by yourself.

## License

Kyoto Cabinet is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version.

Kyoto Cabinet is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see ``http://www.gnu.org/licenses/`'.

Kyoto Cabinet was written by Mikio Hirabayashi. You can contact the author by e-mail to ``hirarin@gmail.com`'.