$Id: BUGS,v 1.265 2008/11/19 13:32:36 vanbaal Exp $

Lire BUGS list.

(

severities are as used by the debian bugtracking system:

critical 
     makes unrelated software on the system (or the whole system) break, or 
     causes serious data loss, or introduces a security hole on systems
     where you install Lire.
grave 
     makes Lire unuseable or mostly so, or causes data loss, or introduces a 
     security hole allowing access to the accounts of users who use the 
     package. 
serious 
     makes the package unsuitable for release. 
important 
     a bug which has a major effect on the usability of Lire, without 
     rendering it completely unusable to everyone. 
normal 
     the default value, applicable to most bugs. 
minor 
     a problem which doesn't affect Lire's usefulness, and is presumably
     trivial to fix. 
wishlist 
     for any feature request, and also for any bugs that are very difficult to
     fix due to major design considerations. 

)

- serious: announce pkgsrc port on announcement list, once stable release is
  available.

- serious: Lire requires GNU Make.  Specifically all/po/Makefile.am does.
  This is tedious for Solaris users (and BSD users, as well).  See
    Date: Sat, 30 Oct 2004 23:13:33 +0200
    From: Joost van Baal
    To: "Francis J. Lacoste", Wolfgang Sourdeau
    Cc: LogReport Questions List
    Subject: Lire requires GNU Make (was: Re: the workaround ...)
    Message-ID: <20041030211333.GO9648@nagy.mdcc.cx>
  It was reported too in 
    Date: Thu, 3 Aug 2006 10:35:34 +0200
    From: Joost van Baal
    To: lhc
    Cc: LogReport Development List
    Subject: Re: Unable to compile lire-2.0.2 under SunOS 5.8
    Message-ID: <20060803083534.GJ3428@nagy.mdcc.cx>
  Either update http://download.logreport.org/pub/current/doc/user-manual/ch02s02.html#sect:standalone-install-req
  or fix the Makefile.am.

- important: Lire fails to build with a non-GNU Make.  See
  http://logreport.org/contact/lists/questions/msg00559.php .  Either fix this
  or document GNU Make in INSTALL as a requirement.  This especially hurts the
  Lire on Solaris community.
                                     JvB, 30 October 2004

- important: It seems DBD::SQLite2 v 0.33 (released 09-Aug-2004), which uses
  SQLite 2.8.12 is unusable on Solaris: bus error.  See
  http://logreport.org/contact/lists/questions/msg00561.php .  This seems to
  make Lire unusable on Solaris.  Should we drop Solaris support?
                                     JvB, 30 October 2004

- normal: get rid of Debian-centric .3pm extension for manpages: make it
  configurable.  Thanks to Peter Bex for reporting this.
                                     JvB, 9 March 2008

- normal: Add support for "Message forwarded from hostname:" tag in 
  Lire::Syslog. Reported by Edward Quick.
                                     FLJ, 20 December 2002

- normal: dubious bug in lr_anonymize : Malformed domainnames do 
  not get anonymized. (See also one of the wishlist items below).
  It can be argued that this really is not a bug, because the domain name
  is not conform the RFC. However, lr_anonymize already deals with
  other non-conformant domain names. (where to draw the line...)

- normal: sendmail2dlf chokes on some wacky sendmail logs.  see the BUGS
  section in the sendmail2dlf manpage.  Reported by Edward Eldred in
  <Pine.A41.4.21L1.0205241026120.28780-100000@login5.isis.unc.edu>
                                    JvB, 26 July 2002

- normal: Table shouldn't size themselves according to the
  longest possible entry in the table. It should try to fit *most*
  entries, long one should be cropped in PDF.
                                    FLJ, Jul  8 2003

- normal: all DLF converters should get migrated to the Perl module style
  setup.  Migrated ones are the ones in superservice/lib/fooDlfConverter.pm,
  they have a supporting /etc/lire/plugins/*_init file.  Unmigrated ones
  live as a script in lib/lire/convertors/*2dlf.
                                    JvB, 21 March 2004

- normal: On OpenBSD, running

   lr_log2report -o html ipfilter < /etc/motd

  (Lire 2.0) fails with:

   Formatting report as html in -...
   tar: Failed open to write on /dev/rst0: Permission denied

  This is likely due to differences in BSD tar and GNU tar; it might fail
  with Solaris tar too.
                                    JvB, 4 Sept 2004

- normal: Fix bug in lr_log2report's html commandline, see bugs section
  in manpage.
                                    JvB, 4 Oct 2006

- minor: contents of TODO should get merged in this file.
                                    JvB, 22 June 2002

- minor: when specifying `0' as parameter in a top-report, the title shows
  up as `Foo by Bar, Top 0', while it actually no longer is a top-report,
  but shows all values.
                                    JvB/FLJ, 22 June 2002

- minor: documentation needs to get reviewed and completed:

  - The list of supported applications/log files in the user manual is
    obsolete/incomplete (though it does mention how to generate an authoritive
    list)                           JvB, 7 may 2003

- minor: Handle warning: line in qmail log files nicely instead of rejecting 
  them: 

	Oct  7 10:52:37 hibou lire: email qmail lr_tag-20021007105232-19195 qmail2dlf warning skipping line '1033731266.302474 warning: trouble opening remote/15/1079428; will try again later': first field should be one of 'new', 'info', 'starting', 'delivery', 'end', 'bounce' or 'status:', not 'warning:'

						     FLJ, 10 Oct 2002

- minor: subst-configvars should be replaced by calls to AC_DEFINE_DIR from
  the Autoconf Macro Archive ( http://www.gnu.org/software/ac-archive/ ):
  the problem we're trying to solve has been solved by others before!
                                                     JvB, 27 Apr 2003

- wishlist: drop lr_scale_ configuration variable and make the unit to
  use fixed so that numbers are comparable in the table. As suggested
  by Edwin Groothuis make that unit overridable in the report
  specifications (a unit attribute).
				    FLJ, Nov 12 2002

- wishlist: the fw1_lea2dlf DLF converter requires the Date::Manip module
  which isn't a standard component. It also requires the use of a
  helper program available from
  http://www.fellhauer-web.de/projects/fw1-loggrabber.html
                                    FLJ, May 17 2003

- wishlist: the spamassassin2dlf convertor should document exact expected
  log file format.  Bug found during discussion with Tom Northeast.
                                    JvB, Aug 24 2005

- wishlist: get reports generated via merging display from which data they
  where generated.  E.g. "Weekly top-m report, generated from daily top-n
  reports".  This in order to give the reader a clue on accuracy.
                                                     JvB, 23 June 2002

- wishlist: make it possible to specify default output format in configure:
  ./configure --with-default-output=XXX
  Make configure fails if the needed requirements can't be found. Asked by
  Rob Dinoff.

- wishlist: make it possible to build Lire on a system which lacks
  XML::Parser.  This is useful for package builders.  (The Debian package
  would no longer have libxml-parser-perl in its Build-Depends-Indep.)
  configure.in would have to check for some environment variable before
  failing.
                                                      JvB, 21 Jul 2002

- normal: Get the Firewall-1 firewall DLF converter properly integrated
  and tested.  Solve the LEA issues.  Requested by Oscar Castaneda.
                                                      JvB, 10 Apr 2003

- wishlist: Adapt WELF firewall convertor to fully support Netasq (
  www.netasq.com ).  See 

   Date: Sat, 6 Apr 2002 13:01:29 -0500
   From: "Francis J. Lacoste" <flacoste@logreport.org>
   To: Van-Hau TRAN
   Subject: Re: hibou Online Responder Report for 04/05/02
   Message-ID: <20020406180129.GD613@Contre.COM>

            JvB, May 10 2002

- wishlist: Implement a Watchguard Soho ( 
  http://www.watchguard.com/products/fireboxsoho.asp ) firewall DLF convertor.
  See

   From: Ben el Cherb
   To: logreport@logreport.org
   Cc: flacoste@logreport.org
   Subject: Re: hibou Online Responder Report for 04/05/02
   Date: Sun, 07 Apr 2002 04:01:06 
   Message-ID: <F54HyM14MCYMPSO3ZZ8000055e0@hotmail.com>

            JvB, May 10 2002

- wishlist: add a convertor for Netscreen ScreenOS 3.0.1's native syslog logs.
  (We now only support the somewhat broken WELF-style logs from this box.)
  suggested in a discussion between Francis and Mark D. Nagel, see 
  <03fe01c22ed4$e2b98a30$cc6fa8c0@BLEU> (not publicly archived.)
            JvB, July 30 2002

- wishlist: Add a "sort" attribute to the records group operation.

- wishlist: Improve email reports :
    - Better error reports (Longer error messages, maybe with failed email
      split errors by to-relay and from-relay).  See also

       Subject: Re: email reports
       From: "Francis J. Lacoste"
       To: development
       Message-Id: <1028124202.32487.29.camel@Arendt>

    - Add a summary (with received, sent, forwarded subcategories).
    - SMTPD reports (connections by hour).
    - Add total (to_domain and from_domain) domain statistics 
      (requested by Matthias Jung)
    - Add domain-by-period reports (requested by Matthias Jung).

- wishlist: handle postfix/pipe log messages. This is encountered, for example,
    on hosts that have postfix with the Amavis virus scanner installed, as
    well as hosts using  the cyrus message transport.  See also

     Message-ID: <3B2BE13C.7040101@tbcinc.net>
     Date: Sat, 16 Jun 2001 17:44:12 -0500
     From: Justin Mecham <XXXXXXX>
     To: LogReport <logreport@logreport.org>
     Subject: Re: your log file to log@postfix.logreport.org

    (Inform Justin once we've fixed this.)

- wishlist: handle 'reject:' log line in the postfix converter:

   Dec 30 07:12:21 smtp postfix/smtpd[680]: reject: RCPT from \
   unknown[212.96.116.2]: 554 Service unavailable; [212.96.116.2] blocked \
   using relays.ordb.org, reason: This mail was handled by an open relay \
   - please visit <http://ORDB.org/lookup/?host=212.96.116.2>; \
   from=<oxygen9690@eudoramail.com> to=<xyz@mooreheadcomm.com>

  Requested by Andy Thompson on Dec 29
                                                     FLJ, 30 Dec 2002

- wishlist: the email dlf format could better have a dlf record for _all_
  message-queued events.  We now only record the final action taken on the
  message.  Only when the logfile doesn't contain the last action taken on the
  message at hand, a stat=queued events makes it into the dlf now.  Queueing
  events migth very well be interesting information, and might be needed for
  some useful reports.  Thanks Mark Huizer for reminding me about this.
                                                     JvB, 31 May 2002

  Even better: we should probably redesign the email DLF format to be more in
  line with what actual email logs contain, that is records for
  receiving, queued, sending, forwarding, bouncing, etc. events. This could be
  modelled after the exim logging model that is very clean (compared to all
  the others
  we support now). This would simplify the email DLF converters a lot,
  since the flow analysis can be moved to generic modules instead of
  having to be reimplemented in each of DLF converter. This would make all
  sorts of reports possible (bounce, forwarding, anomalies).

  It is possible to implement this in the current "single schema model"
  by using something like we did in the firewall superservice where some
  field are only relevant to IDS type of event and others only to packet
  accounting type of events.
                                                      FLJ, 31 May 2002

- wishlist: we should map http requests with escaped url's to their unescaped
  equivalent in some reports, eg map '%7E' to '~'.  See

   From: "E.L. Willighagen" <egonw@sci.kun.nl>
   Date: Tue, 30 Oct 2001 17:20:43 +0100 (MET)
   Message-Id: <200110301620.f9UGKhh16160@studs3.sci.kun.nl>
   To: bugs@logreport.org, edwin@mavetju.org 
   Subject: re: hex-encoded pathnames

  Define a new derived DLF format to support this.  The destinction needs to
  be made in the default dlf format, since apache will give a 404 when given
  a request for a cgi script with an escaped '&' in the url.  See also
  Debian Bug #291063.

  Related to this: when writing an XML report, we are not guarded against a
  buggy report schema writing non-utf8 stuff to the XML.
  Lire::Report::write_report() assumes all subreports are in UTF-8 format, but
  does not check for this.  It should.  If a schema turns out to be buggy in this
  way, Lire dies with a message like: "not well-formed (invalid token) at line
  1004, column 46, byte 53763 at /usr/lib/perl5/XML/Parser.pm line 187".
                                                   updated by JvB, 16 jul 2006

- wishlist: Get rid of commands in the Subject: field of mails to a
  responder (currently we support 'anon').  Use subemailaddresses for this, 
  e.g.: log-anon-html@<service>.logreport.org .

- wishlist: Get lr_log2mail support multipart/alternative emails, so that a
  responder could send both ascii and html reports in one response.
                                                   JvB, 27 may 2002

- wishlist: define a DlfConverter that could parse log files based on field
  specifications.

- wishlist: Make it possible to auto-detect log file types. This could
  be done by trying all the DlfConverters on a small sample of lines and
  compare errors/success statistics. 

- wishlist: implement a mechanism to translate IP address to FQDN.
  Two possibilities:
    1) A lr_getaddrinfo filter which works on the DLF
    2) A lr_xml2resolver filter which works on the XML report.
  Useful for DNS, firewall, www superservice (at least).  See

   Date: Wed, 20 Oct 2004 12:53:19 +0200
   From: Joost van Baal
   To: Jeffeny Hoogervorst
   Cc: LogReport Questions List
   Subject: Re: Question about logreport (resolving hostnames)
   Message-ID: <20041020105319.GM8544@nagy.mdcc.cx>

  archived at http://logreport.org/contact/lists/questions/msg00545.php .

- wishlist: improve ftp reports:
  1: Sort by file. It would display each time a particular file was accessed by
     a particular person and it would list the time.
  2: Sort by user and file. It would display each time a particular file was
     accessed by a particular person.
  3: Sort by user. It would show all files and times accessed by a particular
     user.
  reported by Brett Simpson on Thu, 15 Nov 2001 09:46:59 -0500, in
  Message-Id: <sbf38f1b.032@GroupWise>, Subject: FTP improvement questions

  Furthermore:
  1: Show the number of failed logins.
  2: Show who failed to login and at what time.
  3: Show the number of successful logins.
  4: Show who was successful to login and at what time.
  5: Differentiate between downloads and uploads.
  reported by Brett Simpson on Wed, 21 Nov 2001 09:59:08 -0500 in Message-Id: 
  <sbfb7aef.077@GroupWise>, Subject: Another idea for FTP reporting.

  This might better be handled in a new 'login' super service, however.  JvB.


- wishlist: improve www reports.  See a comparison with our competitors,
  mentioned in

   Date: Tue, 11 Mar 2003 10:26:05 +0100
   From: Wytze van der Raay
   To: people
   Cc: Stichting LogReport <bod>
   Subject: FYI: comparison of web access log analyzers
   Message-ID: <20030311102605.A24779@navarro.bebop.nlnet.nl>

  .
                                                   JvB, 2003-11-20

- wishlist: document this:
   - why do you advise to create a dedicated user account? why not 'nobody'
    or 'daemon'?

- wishlist: Define a cross-superservice report for virtual hosts.
  Requested by Jens Ott, 
	       Lars Magerkohl.	                       FLJ, 24 Mar 2002
							     4 Jun 2002

- wishlist: Make Lire conforms to the NIST standard and uses GiB, MiB
  and KiB for size units and G, M and B for non-size fields. See
  http://kerneltrap.com/article.php?sid=434&target=new. Egonw Dec 27 2001

- wishlist: New email service: Exchange 2000. Requested by 
  Andy Van den Heede.				       FLJ, Apr 06 2002

- wishlist: Support for IPv6 addresses in Lire.  (E.g. the bind9 service and
  the lr_anonymize script.)
  Maybe we should put in a Lire::LogRegex module common regular expressions
  that could be used across service to parse IP addresses and other common
  regular expressions.  Kimura Fuyuki's patch for Regexp::Common adds IPv6
  support to this CPAN module.  See <86elfmjy7r.wl@sz.homedns.org> .

  Requested by Kimura Fuyuki.			       FLJ, Jun 04 2002

- wishlist: Better layout in Excel output format (uses border around
  surreports, contrasting row colours, etc.)

- wishlist: Define a login and daemon schema that could
  probably be used by lot of services.

  See message <1028130934.32593.80.camel@Arendt> to the development
  mailing list.
				
							FLJ, July 31 2002

- wishlist: Define a new user feedback interface that could be used
  to inform the users about the progress of the importation, analysis, report
  generation etc. Backend for syslog, stderr and Lire::UI could be made.
  							 FLJ, Aug 31 2004

- wishlist: Add menu items to import a log file, run an analysers and
  generate a report in Lire::UI.
  							 FLJ, Aug 31 2004

- wishlist: Add a store argument to the lr_log2mail, lr_log2report instead
  of always using a temporary one.
  							 FLJ, Aug 31 2004

- wishlist: Add a lr_import_log command that could import a log file 
 into an existing DlfStore.
  							 FLJ, Aug 31 2004

- wishlist: rename the `dns' superservice to `dnsquery': more descriptive
  name.
                                                        JvB, 16 Aug 2002

- wishlist: make bind8_query, bind9_query and qmail DLF converters
  support more log formats. These log files may be encapsulated in
  syslog log files. qmail can use other timestamp formats.
							FLJ, 16 Aug 2002


- wishlist: get rid of the term `address': it is used at some places, to
  refer to an online responders' address.  That'd better be called `service',
  since these belong to exactly the same namespace.  The file address.cf could
  better be named service_map.cf.
                                                         JvB, 18 Aug 2002


- wishlist: qmail2dlf should support more timestamp formats and syslog
  encoded log files.
							 FLJ, Sep 01 2002

- wishlist: provide configuration hooks to normalize usernames (for
  case-insensitive systems for example). Suggested by Brett Hales.

							 FLJ, Nov 12 2002

- wishlist: support for Microsoft Internet Connection Firewall log files. 
  Suggested by Diego Quintela.
							 FLJ, Nov 23 2002

- wishlist: support for Real Server logfile format. Spec is available from
  http://service.real.com/help/library/guides/g270/htmfiles/report.htm#44459
  Requested by Frank Elsner on questions mailing list.
							 FLJ, Nov 29 2002

- wishlist: support for bind query log files logged through syslog.
  							 FLJ, Dec 03 2002

- wishlist: update argomail parser to support latest log files (version 
  1.8.1.9 log file) break because of different greeting message format and
  use of EHLO.

                                                         FLJ, Jan 16 2003

- wishlist: add possibility to generate a cover sheets from a bunch of
  reports. Suggested by Stewart James (see http://news.its.vu.edu.au/reports/
  for a cover sheet sample).

                                                         FLJ, Feb 21 2003

- wishlist: Support the 'show-timezone' PIX option which adds a timezone
  string to the log. An example can be found in the email sent
  by Amit Sood to our bugs alias.

                                                         FLJ, Feb 28 2003

- wishlist: Add support for OpenBSD pf firewall logs. Requested by
  Paul Weissmann.
                                                         FLJ, Feb 28 2003.

- wishlist: Add an option to display the timestamps in another timezone
  than the one of the log file. Requested by Kevin Krumwiede.
  This could be an additional propertiy on ImportJob and ReportJob.

                                                         FLJ, Apr 11 2003.

- wishlist: in Lire::UI, display the store manipulation utility through a
  tabbed interface so as to reduce the amount of widgets displayed. This
  would enable an easier handling of the widgets as well as a reduction
  of contraints related to the screen estate.

                                                         WAS, Aug 31 2004.

- wishlist: if a integer range contains only one integer, it should be
  listed as "n", not as ["n"-"n+1">.  E.g. in

   User Sessions by Their Recurrence

   Visit #                                                 Sessions % Total
   ------------------------------------------------------- -------- -------
   [1-2>                                                        190    68.3
   [2-3>                                                         35    12.6
   [3-4>                                                         15     5.4

  . Reported in

   From: "E.L. Willighagen"
   To: LogReport Support
   Subject: Re: logreport.com on hibou 20041024
   Date: Mon, 25 Oct 2004 08:47:20 +0200
   Cc: people
   Message-Id: <200410250847.21559.e.willighagen@science.ru.nl>

                                                           EW, Oct 25 2004

- wishlist: make Lire easier extensible.  E.g., it'd be nice if DLF files
  could be exported in their ascii representation from a DLF store.  And
  it'd be nice if we had a CSV report format: that's machine parsable for
  people who don't know XML and/or Perl.  Some people _are_ using old-style
  Lire ways to do these kind of things, e.g. in situations where just a
  part of Lire is used.
                                                          JvB, Oct 13, 2004

  See www/script/dlf2combined (and all/script/lr_call2CLFtime.in) for a first
  shot at a way to use just the dlf-stuff of lire.
                                                          JvB, Nov 19, 2008

- wishlist: some suggestion items on what could be
  part of a 2.1 release.  (In no particular order)
  a. Convert remaining pre-1.3 DLF converters to the new API.
  b. Add reports for the lire_import_log and lire_import_stats
     superservice.
  c. Write new responder using new APIs.
  d. Interactive performance improvement, Lire takes much time
     to start because of all the XML files it has to parse. It would
     be easy to add a persistent cache to speed that part up.
  e. Add more information on where the data came from to the XML report
     (DLF sources used, reports merged)
  f. Report generation performance improvements.
                                                          FJL, Oct 12, 2004

