Ace
Jones

    acejones@users.sourceforge.net

2005-06-01
0.7

# Importing and Export

Tony
Bloomfield

    tonybloom@users.sourceforge.net

## GnuCash Importer

### GnuCash Files

The &kappname; GnuCash importer handles direct reading of standard (XML) files as produced by GnuCash versions 1.8 and 2.0. The following are not supported

- import of database (Postgres) data

- import of 'multi-book' files

- import into an existing &kappname; file

- import of small-business specific features (Employees, Invoices, etc.)

- export to GnuCash files.

The import will probably only work correctly if presented with a valid file. It is recommended that the GnuCash 'Check & Repair All' function (in the Actions menu) be run before attempting to import.

Files can be opened by specifying the file name on the command line (`kmymoney <filepath>`), or by means of the &kappname; Ctrl-O File Open or File Import menu items.

The similarity between the two products means that much day-to-day data can be imported in a straightforward fashion. However, there are some areas where differences arise, and various options are provided to deal with these. The following sections will describe some of these differences; understanding them should lead to a smoother importation.

### Similarities, Differences and Terminology

#### Small Business Usage

It should be noted that &kappname; is a personal finance manager, and as such, does not directly support any of the business features of GnuCash, such as tax tables, payroll, and tracking of lots. Any Accounts Payable or Receivable accounts found in a file will be imported as Liability or Asset accounts respectively.

#### Accounts

##### Account types

For both products, the highest level of structure in the file is the account. &kappname; supports 5 main types of account, Asset, Liability, Income, Expense and Equity, each of which may have various subtypes, e.g. Checking, Credit Card, etc. &kappname; includes a 'standard' account for each of these five types, and all other accounts are held subordinate to one of these. &kappname; enforces more consistency (or less flexibility, depending on your point of view) between account types than does GnuCash, and the importer will correct any inconsistencies it detects. This may result in a slightly different account structure, though this can, within reason, be amended after the import is complete.

##### Categories

&kappname; uses the term Category to denote an account of an Income or Expense type. Unlike GnuCash, these are not considered as 'ledger' accounts and entry of transactions directly is not currently supported, all postings being made by splits (see below) from transactions entered in other account types.

##### Structure and Placeholders

GnuCash supports the use of Placeholder accounts. In effect, these are just read-only accounts into which no transactions can be entered, but which function in an analogous fashion to folders in a directory structure, as a holder for other accounts. Though &kappname; does not support this feature as such, it does provide a parent/child account relationship, so the importer simulates placeholders by creating empty accounts.

##### Account Type map

###### Equivalence of GnuCash and &kappname; account types

GnuCash type | &kappname; type
--- | ---
BANK | Checking
CHECKING | Checking
SAVINGS | Savings
ASSET | Asset
CASH | Cash
CURRENCY | Cash
MONEYMRKT | MoneyMarket
STOCK | Stock
MUTUAL | Stock
EQUITY | Equity
LIABILITY | Liability
CREDIT | CreditCard
INCOME | Income
EXPENSE | Expense
RECEIVABLE | Asset
PAYABLE | Liability

#### Transactions and Splits

##### Balanced transactions

As with GnuCash, data is entered in the form of transactions, each generally consisting of 2 or more split entries. In fact, valid GnuCash transactions will always contain at least 2 splits, and to conform to GnuCash's double-entry bookkeeping standard, these must be in monetary balance (i.e. must balance out to zero). &kappname; encourages, but does not enforce, this standard.

##### Payees

&kappname; prefers that all transactions have a Payee (a generic term that encompasses both payees and payers), and unlike GnuCash, a list of these payees is maintained. Payee names are generated by the importer from the GnuCash transaction's Description field.

##### Transfers

&kappname; uses the term Transfer to describe a transaction which does not involve a Category, but only transfers money between Asset and/or Liability accounts.

##### Reconcile

&kappname; provides an account reconciliation function similar to that of GnuCash, and the corresponding transaction status will be imported.

#### Commodities

GnuCash uses the term Commodity to cover both currencies and non-currency assets. These are treated separately in &kappname;.

##### Currencies

&kappname; has inbuilt support for all foreign currency types. &kappname; also requires that the user specify a base currency, this being the default currency for new accounts. The importer will attempt to determine the most likely base currency, though this choice may be rejected in favour of an alternative.

(NOTE: &kappname; does not currently support accounts denominated in 'defunct' currencies (except those replaced by the Euro). At present, it will be necessary to remove any such accounts from your GnuCash file before importing. We hope to improve on this situation in a future release.)

##### Securities and Investments

Non-currency assets (normally stocks and bonds) are called Securities by &kappname;, and represent the main difference between the two products, in that &kappname; requires any account denominated in a security to be subordinate to an Investment Account. This is described in more detail in the chapter on Investments. Though users may have implemented such a relationship, GnuCash imposes no defined structure on it, so the importer is unable to detect it and perform an automatic conversion. Three options are therefore made available:

- Create a separate Investment account for each security, with the same name as the security

- Create a single Investment account which will act as 'parent' for all security accounts

- Create several Investment accounts, and assign securities to them as directed by the user.

It depends entirely on user requirements which of these options is relevant in each situation, and in some cases, manual restructuring of accounts after importation may be necessary.

##### Prices and currency rates

Security prices and currency exchange rates as displayed in the GnuCash Price Editor will be imported. In addition, price and rate entries will be generated from all transactions involving securities and multiple currencies.

##### Online Quotes

For obtaining online price and currency rate quotations, GnuCash uses a package called Finance::Quote. &kappname; does not support this at present, and has its own methods, covered in more detail in online quotes. Obtaining a quote requires a currency or stock (ticker) symbol, and these are imported unchanged into &kappname;. The source from which the quote is to be obtained, however, cannot be imported. Instead the following dialog will allow selection of a 'native' &kappname; price source, or a user-defined source, for each account for which online quotes are required.

#### Scheduled Transactions

&kappname; does not retain the separation made in GnuCash between template transactions and their frequency of occurrence. Transaction data will be duplicated if the same template is used in different schedules, but this is not likely to be of great significance.

##### Schedule types

&kappname; classifies all schedules as one of three types, Bills, Deposits and Transfers. Since GnuCash does not make such a distinction, the importer attempts to determine the classification from the accounts and direction of money movements. It may be that in some cases incorrect assumptions are made, and these will need manual correction.

##### Suspect Schedules

Some features of GnuCash scheduled transactions are not available in &kappname;, so the importer tries in each case to reach a reasonable compromise in converting the data. These transactions will be flagged as suspect, and the user will be given the option of editing them directly during the import process. Examples of situations which may cause this are:

- some frequency intervals supported in GnuCash are not currently available in &kappname;

- &kappname; does not support the use of formulae and variables in amount fields

- complex cases which have not yet been identified for import.

Despite best efforts, it is possible that, due to the many options involved, a scheduled transaction may cause a fatal error within &kappname;. If this sort of problem seems to be occurring, the importer offers the option to drop all suspect schedules.

#### Reports

&kappname; provides a comprehensive selection of configurable reports, described in more detail in Reports. These will not necessarily, however, match precisely those reports available in GnuCash.

### Selecting Importer Options

#### Investment Handling

See "Securities and Investments" above.

#### Scheduled Transactions

See "Scheduled Transactions" above.

#### Decoding Options

If your native language is written in letters or symbols which are different from those used in the 'Latin' (i.e., generally Western European) languages, these will be represented in a special fashion ('encoded') in your GnuCash file. If these letters are not displayed correctly on your screen, then they must be decoded. Currently, it is often not possible to detect accurately which form of decoding must be used, so you may need to set this option and select an entry from the list. In general, the first item in the list will be that which is considered appropriate for your locale (i.e. the country and language which was selected as native when your operating system was installed), so this should be tried first. Since the import process does not overwrite your GnuCash file, you are free to experiment with any of these selections.

#### Debug Options

These need only be used in the event of import problems, which in any case should be reported to the &kappname; developer list &devlist;. Note that the traces produced by these options may contain data of a confidential nature, and the Anonymize option should be used if they are to be made publicly available.

### Import Report

At the end of processing, the importer produces a report showing the number of different entities processed, and any errors or anomalies encountered. This report will be displayed on screen, and may be saved to a file for later review. A full report may contain the following sections:

- Record counts

- Inconsistencies in account types and actions taken

- Details of suspect schedules

## QIF Importer

### QIF format considered harmful

Generally speaking, the QIF format should be avoided wherever possible. It is a poor choice for transporting financial data. Among other thing, QIF suffers from these problems:

- Lack of standardized format. Different versions of the same program will impart different meanings to the same element.

- Lack of transaction identifier. Because there is no ID number associated with each transaction, matching duplicates is haphazard at best.

- Lack of expressiveness. The grammar is really simple, and cannot portray the depth of financial information found in today's financial environment.

This is generally why Intuit stopped supporting QIF input at all with Quicken 2005. If you have the option of getting data some other way, like OFX, always choose that option.

### How to import a QIF file

To import a QIF file, first ensure you have a valid &kappname; file open. Then select Import Qif from the File menu.

The resulting dialog prompts for the QIF filename allowing you to locate the file by clicking on the Browse button.

If you want &kappname; to automatically enter new payees check the appropriate option otherwise &kappname; will prompt you each time a new payee is found.

The QIF Profile is very important, because all QIF files are different. Please be sure it is set up correctly before continuing. See the next section for more details on that.

Click on Import to import the QIF file.

The first thing the importer will do is ask you which account to import the transactions into. If there are transactions from multiple accounts in your file, you will be asked this question multiple times.

QIF import report QIF import report

After importing, some of your transactions may be flashing red in the ledger. This is because they need to be assigned a category. The importer was not able to automatically assign a category based on your past transaction history.

### Confirming the imported transactions

When &kappname; has finished importing the QIF transactions the account will be shown with the imported transactions listed in Yellow.

QIF import verification QIF import verification

Some of your transactions may be flashing red in the ledger. This is because they need to be assigned a category. The importer was not able to automatically assign a category based on your past transaction history.

Transaction data can be edited or even deleted if needed. To edit a transaction simply double click on the entry or hit enter when the entry is highlighted. Once finished click on OK to accept the imported transactions or Cancel to remove the imported transactions.

### Importing Investments

Please note that if you are importing a file with investment transactions, those investments must first exist in your &kappname; file. The trading symbol is used to match, so please ensure that the symbol in &kappname; is exactly the same as the one in the file you're importing.

### Setting up a QIF profile

Because there is no universally standard format for the QIF file, different vendors have taken liberties with the format, and introduced their own nuances. Therefore, you must create a QIF Profile before you can import a QIF file. This profile tells &kappname; about the peculiarities of your file. To edit an existing QIF Profile, or to create a new one, press the "New" button on the QIF Import dialog, near the profile selector.

QIF Profile Editor

The most commonly changed thing between QIF implementations is the date format. So if this is the first time you're importing a QIF file, spend a few moments to figure out what format the dates are in, and set the QIF Profile accordingly. See the discussion below on apostrophe format for more details.

QIF Profile Date

### Apostrophe format

Many common QIF writers use a 2-digit representation for the year. This is ambiguous, because the importer cannot know which century the date belongs in. To make things even more complicated, QIF files will often used an apostrophe as a year separator to indicate that the date belongs in the OTHER century from the default.

For example, if the default century is 1900-1999, the date 12/31/95 would mean 1995. The date 12/31'05 would mean 2005.

Because the QIF format is not standardized, it's impossible to know which century is desired. This is why you have to explicitly state it in the QIF profile. You do this by specifying which century is intended when an apostrophe is found. In the example above, you would set the Apostrophe Format to "2000-2099", so dates with an apostrophe will be interpreted as being > year 2000. In this case, dates without an apostrophe will be treated as being in the 1900's.

### Transaction matching

As noted previously, one of the major drawbacks of the QIF format is the lack of a unique identifier for each transaction. Thus, if you import a QIF file and some of the transactions are already in your ledger, you may get duplicates. KMM attempts to get around this by looking for transactions that look similar to those you already have. If it finds something that looks like the same transaction, it will discard the apparent duplicate.

This can be a problem if you have transactions that look too similar but are actually different. In this case, you can try turning off the "Attempt to match similar transactions" option in the QIF profile.

### Writing an import filter

Sometimes you may have data in a custom format, like comma-separated-values (CSV), or something else unique to your situation. You can still import that file into &kappname; using a QIF Import Filter. A filter is a custom program you write which takes your special file as input, and produces a QIF file as output. This can be a shell script, a perl script, a compiled program written in C/C++, or anything else you can dream of, as long as the system can run it.

To use it, edit your favorite QIF Profile, and select the Filter tab. Enter the location of your filter program where prompted. Then, whenever you do a QIF import using this profile, the file you select for importing will be run through your filter first.

A common problem is to convert a list of comma-separated-values into a QIF file. This is a textbook case for the awk tool. Create a script called csv2qif.awk, with the following two lines as contents:

    	BEGIN { FS=","; print "!Type:Bank" }

    	{ print "D"$1; print "T"$2; print "N"$3; print "P"$4; print "M"$5; print "^" }

Then, change the QIF keys (D,T,N,P,M) to match the order of your csv data. Set the input filter to `awk -f csv2qif.awk`.

Another problem sometimes arises in the encoding of QIF files. &kappname; expects files to be UTF8 encoded. If your file is encoded in something else, it can be useful to convert it to UTF8. For example to convert it from iso-8859-1, you would set the input filter to `recode iso-8859-1..utf-8`.

### Special &kappname; QIF extensions

As noted previously, one of the major drawbacks of the QIF format is the lack of a unique identifier for each transaction. If you are writing your own QIF file creator (or filter, as described above), you can overcome this problem. &kappname; supports the '#' field. The importer will interpret this as a unique transaction ID, and disregard it if the transaction is already in the system.

## QIF Exporter

To export one of your accounts to a QIF file, choose the "File | Export | Qif..." menu option. You will be prompted for which single account to export, what file to export it to, and what QIF Profile to use.

QIF Export

## OFX Importer Plugin

### Getting the plugin

&kappname; will import OFX files painlessly. However, this functionality is not built into the core program. You must obtain and install the OFX Importer Plugin. Once that is installed, the command to import OFX files will automatically show up under the "File | Import" menu.

If you have installed from RPM, the OFX Imported Plugin is contained within the kmymoney-ofx RPM. It should be available from whatever source you got the base &kappname; RPM. If you have built from sources, all you need to do is have the libOFX 0.8.1 development headers & libraries installed on your system. The &kappname; build process will detect these and compile the plugin.

Should you run into trouble trying to compile &kappname;, and you are certain you have the correct version of libOFX installed, please contact the developers list &devlist; for assistance. Include a copy of your config.log file, compressed first via gzip.

### File | Import | OFX...

The most basic way to import an OFX file is to choose the importer from the menu bar. From the File menu, choose Import, and then OFX. If OFX does not show up under Import, you do not have the OFX Importer Plugin installed correctly. Please see the previous section.

The first thing the importer will do is ask you which account to import the transactions into. If there are transactions from multiple accounts in your file, you will be asked this question multiple times.

After importing, some of your transactions may be flashing red in the ledger. This is because they need to be assigned a category. The importer was not able to automatically assign a category based on your past transaction history. You can edit each flashing transaction in the ledger to assign a category, and it will then stop flashing red.

Please note that this section describes the "native" OFX importer. OFX files may also be imported using the AqBanking Importer Plugin if you have installed that. Note that the two importers do behave slightly differently, and they are written and supported by two different developers.

### Importing Investments

Please note that if you are importing a file with investment transactions, those investments must first exist in your &kappname; file. The trading symbol is used to match, so please ensure that the symbol in &kappname; is exactly the same as the one in the file you're importing.

### Web Connect

The easiest way to import an OFX file is to set up Web Connect. Visit your bank's web site, and click on a link to download an OFX file. Your browser should ask you what program you would like to use to open the program. Point your browser to &kappname;. It will then import the downloaded OFX file into the &kappname; file you most recently had open.

If you need to import the OFX file into some other &kappname; file, load up that file in &kappname; first, and then visit your bank.

### Direct Connect

This version of &kappname; does not support the ability to contact your bank directly to obtain statements or make payments. However, Martin Preuss has created a program called AqBanking to do this for you. It includes a plugin which integrates seamlessly with &kappname;. Please visit the AqBanking site for more details.

### Exporting an OFX file

It is not possible to export your data as an OFX file currently. If you're interested to contribute in this area, please contact the libofx development team for details.

## Writing Importer Plugins

&kappname; contains explicit support for importer plugins. If you have a custom format, and you would like to write an importer plugin, we would value your contribution. To do so, you'll need to compile the program from source. Then use the OFX Importer Plugin as an example.