Merging Reports
Prev | Chapter 3. Running Lire | Next

## Merging Reports

Since june 2002, Lire supports the merging of reports: one can combine two reports into one bigger report. This can be used to generate e.g. a weekly report from 7 daily reports, or generate a site-wide report from reports about the behaviour of each server on a site.

We describe some of the issues involved here. More in-depth information can be found in the lr_xml_merge manpage.

### Manual Merging

We give an example.

Example 3.5. Merging Reports

To process two BIND v9 logfiles, and merge the reports, one would run:

    $ zcat /var/log/named.2.gz | lr_run lr_log2report \
       -o xml bind9_query > $XMLDIR/20020622.xml
    $ zcat /var/log/named.1.gz | lr_run lr_log2report \
       -o xml bind9_query > $XMLDIR/20020623.xml
    $  lr_run lr_xml2report -U dns -S bind9_query \
       $XMLDIR/20020622.xml $XMLDIR/20020623.xml > \
       $ASCIIDIR/20020622-20020623.txt

Of course, refer to the lr_log2report and lr_xml2report manpages for more detailed information.

### Automating Merging using cron: some ideas

Suppose you're running a daily cronjob which feeds your log to lr_log2report (possibly via lr_cron). If you'd like to generate weekly reports, you should first make sure `LR_ARCHIVE` is set to '1'. This way, each daily XML report will get saved in files like e.g. `$XMLDIR/hostname/20020602085048-20020603221309.xml.gz` where `XMLDIR` is `/var/lib/lire/data/report/xml/dns/bind9` (the raw log files, as well as the intermediate DLF files, will get saved too, btw; see also the section called “The Lire Archive and Temporary Files”). Now you can add a weekly cronjob, which unzips the last 7 files from this directory to 7 temporary files, and runs:

    lr_run lr_xml2mail -U yoursuperservice -S yourservice \
     uncompressed-xml-file-nr-1 uncompressed-xml-file-nr-2 ... \
     uncompressed-xml-file-nr-7 joe@example.com

Be sure to clean up the 7 uncompressed files afterwards. Refer to the lr_xml2mail manpage for more details.

### Gotchas

The merging functionality is very powerful, and allows you to shoot yourself in the foot. We document some pitfalls.

#### Holes in your reporting period

When merging XML report files xml.3 (2002-06-02 08:50:48 CEST - 2002-06-09 08:05:06 CEST) and xml.1 (2002-06-16 08:18:40 CEST - 2002-06-21 22:13:09 CEST) , the generated report will gladly display "Reporting on period: 2002-06-02 08:50:48 CEST - 2002-06-21 22:13:09 CEST": There is no saveguard against forgetting in-between report files.

#### Take care when changing report configuration file parameters

In some cases, changing the report configuration file (``superservice`.cfg`) just before merging might lead to bogus data in your report.

Consider this case: Our firewall.cfg file looks like:

    =section Denied Packets Reports
    |select-action action_match="denied"
    top-pkt-by-src              ips_to_show=10

We process some firewall logs, and archive the XML reports. Now we set ips_to_show to 1000, and merge the XML reports. This could incorrectly omit some IPs! You've got no guarantee the exact top 1000 IPs are shown. This is due to the fact the XML reports do not contain all information from the log: they're reports, after all.

Due to these issues, the merging is implemented with some heuristics: we keep more data than what's requested by the user in the XML report, to be able to handle most after-the-fact merging requests. We've tested the algorithm with a pretty broad range of real-life log files, and found out generally, the merged reports do give a good reflection of what actually has happened on the network: the heuristic is pretty well choosen. However, if you really need guaranteed 100% accurate data, generate your report directly from the raw logs. If you just want a quick overview, the merging is more suitable. Just make sure you're not cranking the limit parameters up too high in this case.

See also the Report Generating chapter in the Lire Architecture part of the Lire Developer's Manual.

Prev | Up | Next
Generating A Report From A Log File | Home | Sending Anonymized Log Files To A Responder