RWFLOWPACK(1)

NAME
rwflowpack - Collects flow data and stores it in binary SiLK Flow files

SYNOPSIS
  rwflowpack --sensor-configuration=FILE_PATH
        { --log-destination=DESTINATION
          | --log-directory=DIR_PATH [--log-basename=BASENAME]
          | --log-pathname=FILE_PATH }
        [--byte-order=ENDIAN] [--compression-method=COMP_METHOD]
        [--site-config-file=FILENAME] [--flush-timeout=VAL]
        [--pack-interfaces] [--no-file-locking] [--log-level=LEVEL]
        [--log-sysfacility=NUMBER] [--pidfile=FILE_PATH] [--no-daemon]
        [--input-mode=MODE] [--output-mode=MODE]
        MODE_SPECIFIC_SWITCHES

To collect NetFlow v5 or IPFIX data from the network (default):

  rwflowpack ... [--input-mode=stream] [--sensor-name=SENSOR] ...

To collect from files containing NetFlow v5 PDUs:

  rwflowpack ... --input-mode=file --netflow-file=FILE_PATH
        [--sensor-name=SENSOR] [--archive-directory=DIR_PATH] ...

To collect from a remote flowcap(8) process:

  rwflowpack ... --input-mode=flowcap [--flowcap-port=NUMBER]
        --flowcap-address=IP_ADDR[:NUMBER][,IP_ADDR[:NUMBER]]
        --work-directory=DIR_PATH --valid-directory=DIR_PATH
        [--archive-directory=DIR_PATH] ...

To collect from local files containing files created by flowcap:

  rwflowpack ... --input-mode=fcfiles --incoming-directory=DIR_PATH
        [--polling-interval=NUMBER] [--archive-directory=DIR_PATH] ...

To store the SiLK Flow files on the local machine (default):

  rwflowpack ... [--output-mode=local-storage]
        --root-directory=DIR_PATH ...

To forward the SiLK Flow files to a remote machine:

  rwflowpack ... --output-mode=sending --sender-directory=DIR_PATH
        --incremental-directory=DIR_PATH
        ...

DESCRIPTION
rwflowpack is a daemon that collects NetFlow-like data, converts the data to the SiLK Flow record format, categorizes each flow (e.g., as incoming or outgoing), and stores the data in binary flat files within a directory tree, with one file per hour-category-sensor tuple.

See the SiLK Installation Handbook for an explanation of how SiLK categorizes flows and converts data to the SiLK format.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

  General Configuration
The following switch is required:

--sensor-configuration=FILE_PATH
    Give the path to the configuration file that rwflowpack will consult to determine whether a record represents an incoming or outgoing flow. The complete syntax of the configuration file is described in the sensor.conf(5) manual page; see also the SiLK Installation Handbook.

One of the following switches is required:

--log-destination=DESTINATION
    Specify the destination where logging messages are written. When DESTINATION begins with a slash "/", it is treated as a file system path and all log messages are written to that file; there is no log rotation. When DESTINATION does not begin with "/", it must be one of the following strings:

    "none"
        Messages are not written anywhere.

    "stdout"
        Messages are written to the standard output.

    "stderr"
        Messages are written to the standard error.

    "syslog"
        Messages are written using the syslog(3) facility.

    "both"
        Messages are written to the syslog facility and to the standard error (this option is not available on all platforms).

--log-directory=DIR_PATH
    Use DIR_PATH as the directory where the log files are written. DIR_PATH must be a complete directory path. The log files have the form

      DIR_PATH/LOG_BASENAME-YYYYMMDD.log

    where YYYYMMDD is the current date and LOG_BASENAME is the application name or the value passed to the --log-basename switch when provided. The log files will be rotated: at midnight local time a new log will be opened and the previous day's log file will be compressed using gzip(1). (Old log files are not removed by rwflowpack; the administrator should use another tool to remove them.) When this switch is provided, a process-ID file (PID) will also be written in this directory unless the --pidfile switch is provided.

    NOTE: Prior to SiLK 0.10.0, when the --sensor switch was given, the sensor name was automatically included as part of the log file name. As of SiLK 0.10.0, the --log-basename must be used to include this information in the log file name.

--log-pathname=FILE_PATH
    Use FILE_PATH as the complete path to the log file. The log file will not be rotated.

The following set of switches is optional:

--byte-order=ENDIAN
    Set the byte order for newly created SiLK Flow files. When appending records to an existing file, the byte order of the file is maintained. The argument is one of the following:

    "native"
        Use the byte order of the machine where rwflowpack is running. This is the default.

    "big"
        Use network byte order (big endian) for the flow files.

    "little"
        Write the flow files in little endian format.

--compression-method=COMP_METHOD
    Set the compression method for newly created SiLK Flow files to COMP_METHOD. When appending records to an existing file, the compression method of the file is maintained.

    In addition to the packing (shrinking) of the flow records that SiLK normally does, rwflowpack can use an external library to further reduce the size of the records on disk. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support the following:

    none
        Do not compress the SiLK Flow records using an external library.

    zlib
        Use the zlib(3) library for compressing the flow records.

    lzo1x
        Use the lzo1x algorithm from the LZO real-time compression library for compressing the flow records.

    best
        Use whichever available method gives the "best" compression in general, though not necessarily the "best" for this particular file.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the $SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of $SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the $SILK_DATA_ROOTDIR environment variable; the data root directory that is compliled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/, $SILK_PATH/share/, or $SILK_PATH; or in the share/silk/ and share/ directories parallel to the application's directory.

--flush-timeout=VAL
    Set the timeout for flushing any in-memory records to disk to VAL seconds. If not specified, the default is 5 minutes (600 seconds). When using local storage mode, this value specifies how often the files are flushed to disk to ensure that any records in memory are written to disk. When using sending output mode with a stream input mode, this value specifies how often to move the files from the incremental-directory to the sender-directory.

--pack-interfaces
    Allow one to override the default file output format of the packed SiLK Flow files that rwflowpack writes. When this switch is present, rwflowpack writes additional information into the packed files: the router's SNMP input and output interfaces and the next-hop IP address. The extra data produced by this switch is useful for determining why traffic is being stored in certain files.

    Note that this switch will only affect newly created files. New records will always be appended to an existing file in the file's current output format to maintain file integrity.

--no-file-locking
    Do not use advisory write locks. Normally, rwflowpack will attempt to obtain a write lock on the data files prior to writing records to them; these locks prevent two instances of rwflowpack from writing to the same data file. However, not all file systems support advisory write locks, and this switch must be used when writing data to such a file system.

--log-level=LEVEL
    Set the severity of messages that will be logged. The levels from most severe to least are: "emerg", "alert", "crit", "err", "warning", "notice", "info", "debug". The default is "info".

--log-sysfacility=NUMBER
    Set the facility that syslog(3) uses for logging messages. This switch takes a number as an argument. The default is a value that corresponds to "LOG_USER" on the system where rwflowpack is running. This switch produces an error unless --log-destination=syslog is specified.

--log-basename=LOG_BASENAME
    Use LOG_BASENAME in place of the application name for the files in the log directory. See the description of the --log-directory switch.

--pidfile=FILE_PATH
    Set the complete path to the file in which rwflowpack writes its process ID (PID) when it is running as a daemon. No PID file is written when --no-daemon is given. When this switch is not present, no PID file is written unless the --log-directory switch is specified, in which case the PID is written to LOGPATH/rwflowpack.pid>.

    NOTE: Prior to SiLK 0.10.0, when the --sensor switch was given, the sensor name was automatically included as part of the PID-file name. As of SiLK 0.10.0, the --pidfile switch must be used to include this information in the PID-file name.

--no-daemon
    Force rwflowpack to stay in the foreground---it does not become a daemon. Useful for debugging.

--reader-function=NUMBER
    This switch is deprecated and ignored. It is only present to maintain backward compatibility.

  Input and Output Mode
rwflowpack supports multiple ways of getting (the input mode) storing (the output mode) data.

--input-mode=MODE
    Determine how rwflowpack will gather data. The default input MODE is "stream". The available modes are

    "stream"
        rwflowpack opens a port for every network-listening probe specified in the sensor configuration file. These ports expect to receive NetFlow v5 PDUs as generated by a router.

    "file"
        rwflowpack reads NetFlow v5 PDUs from a file. The file's size must be an integer multiple of 1464, where each 1464 byte chunk contains a 24 byte NetFlow v5 header and space for thirty 48 byte NetFlow records. The number of valid records per chunk is specified in the header.

    "flowcap"
        rwflowpack connects over a TCP socket to a machine running flowcap(8), and transfers specially compressed NetFlow v5 files (called flowcap files) from the remote location to the local disk for processing. Note that "flowcap" mode may not be available depending on how rwflowpack was built.

--output-mode=MODE
    Determines what rwflowpack will do with the data as it is packed into SiLK binary files. The default output MODE is "local-storage". The available modes are

"local-storage"
    rwflowpack writes the data on the local machine into a directory tree with a specific structure.

"sending"
    rwflowpack writes the data into a temporary location on the local disk. A separate program, rwsender(8), moves the data from the local machine to remote machines where rwreceiver(8) working in concert with the rwflowappend(8) will write the data into a directory tree with a specific structure. Note that rwflowpack may have been built without support for "sending" mode.

  Stream Collection Switches (--input-mode=stream)
When the --input-mode switch is set to "stream" or when the switch is not provided, rwflowpack expects to receive stream(s) of NetFlow data over the network. rwflowpack will open a UDP port for every probe given in the configuration file that has a "listen-on-port" attribute.

The following switch is optional:

--sensor-name=SENSOR
    Cause rwflowpack to ignore all probes in the sensor configuration file except the probes for SENSOR. Only data for this sensor will be collected. This allows a common configuration file to be used by multiple rwflowpack invocations, yet also allow each rwflowpack instance only collect for a single sensor. There must be a sensor-probe definition for SENSOR in the configuration file. When this switch is not present, rwflowpack will collect and pack data for all sensors.

  PDU File Collection Switches (--input-mode=file)
Instead of reading flows from the network, rwflowpack can process files containing NetFlow data. These files can be in one of two forms: either files generated by a NetFlow collector, or files generated by another SiLK program called flowcap.

To make rwflowpack process a file generated by a NetFlow collector, pass it the --input-mode="file" switch. The file must have a particular format: The file's length should be an integer multiple of 1464 bytes, where 1464 is the maximum length of the NetFlow v5 PDU. Each 1464 block should contain the 24-byte NetFlow v5 header and space for 30 48-byte flow records, even if data for only one NetFlow record is valid.

Although rwflowpack can get the names of the NetFlow files from the "read-from-file" attributes in the sensor configuration file, it is more common to set "read-from-file" to /dev/null, and to pass the name of the NetFlow file on the command line with the --netflow-file switch. This simplifies scripting; otherwise, the sensor configuration file would have to be rewritten for each run.

In this mode, rwflowpack will not become a daemon; it will remain in the foreground, process the NetFlow file(s), and exit.

The following switches are all optional:

--sensor-name=SENSOR
    Cause rwflowpack to ignore all probes in the sensor configuration file except the probes for SENSOR. See above for a full description of this switch.

--netflow-file=FILE_PATH
    Name the full path of the file from which rwflowpack reads NetFlow v5 PDUs.

--archive-directory=DIR_PATH
    Nameds the full path of the directory to which NetFlow files will be moved after they have been processed by rwflowpack. If this switch is not provided, the original NetFlow source files are not modified, moved, or deleted. Removing files from the archive-directory is not the job of rwflowpack; the system administrator should implement a separate cron job to clean this directory.

  Flowcap Collection Switches (--input-mode=flowcap)
When the --input-mode=flowcap switch is provided, rwflowpack will process files created by another SiLK daemon called flowcap. flowcap collects NetFlow records near the router generating the NetFlow records, compresses the records, and stores them on the local disk. rwflowpack will contact the machine where flowcap is running and transfer the files via TCP to its local disk; rwflowpack then processes the files to generate SiLK Flow files.

When operating in flowcap input mode, the first four of the following switches are required:

--flowcap-address=IP_ADDR[:NUMBER][,IP_ADDR[:NUMBER]] ...
    Specify the host addresses of the flowcap servers. rwflowpack will attempt to contact these machines on the ports given. If no port is specified for an address, it will use the port specified by the --flowcap-port switch.

--flowcap-port=NUMBER
    Specify the default port on which rwflowpack will attempt to contact to flowcap servers.

--work-directory=DIR_PATH
    Name the full path of the directory used to store files as they are being received from flowcap. The files in this directory are incomplete; any files in this directory will be removed when rwflowpack is started. Once complete, files are moved from this directory to the valid-directory. The work-directory, valid-directory, and archive-directory must be on the same file system mount point.

--valid-directory=DIR_PATH
    Name the full path of the directory used to store files that have been successfully received from flowcap but which have not yet been processed by rwflowpack. Once processed by rwflowpack, files are moved from this directory to the archive-directory, if it has been specified. The work-directory, valid-directory, and archive-directory must be on the same file system mount point.

--archive-directory=DIR_PATH
    Name the full path of the directory used to store flowcap files after rwflowpack has processed them. If this switch is not provided, the flowcap files are deleted. Removing files from the archive-directory is not the job of rwflowpack; the system administrator should implement a separate cron job to clean this directory. The work-directory, valid-directory, and archive-directory must be on the same file system mount point.

--fc-address=IP_ADDR
    Deprecated alias for --flowcap-address.

--fc-port=NUMBER
    Deprecated alias for --flowcap-port.

  Flowcap Files Collection Switches (--input-mode=fcfiles)
When the --input-mode=fcfiles switch is provided, rwflowpack will process files created by another SiLK daemon called flowcap. flowcap collects NetFlow records near the router that is generating the NetFlow records, compresses the records, and stores them on the local disk. rwflowpack will will poll for flowcap files to be inserted in its incoming directory. rwflowpack then processes the files to generate SiLK Flow files.

When operating in flowcap files input mode, the first of the following switches are required:

--incoming-directory=DIR_PATH
    Name the full path of the directory which rwflowpack will monitor for files created by flowcap. Once processed by rwflowpack, files are moved from this directory to the archive-directory, if it has been specified. The incoming-directory and archive-directory must be on the same file system mount point.

--polling-interval=NUMBER
    Insert the number of seconds between which rwflowpack will poll the incoming-directory for new files created by flowcap. This defaults to 15 seconds.

--archive-directory=DIR_PATH
    Name the full path of the directory used to store flowcap files after rwflowpack has processed them. If this switch is not provided, the flowcap files are deleted. Removing files from the archive-directory is not the job of rwflowpack; the system administrator should implement a separate cron job to clean this directory. The incoming-directory and archive-directory must be on the same file system mount point.

  Local Storage Mode Switches (--output-mode=local-storage)
Once rwflowpack has collected data, categorized it, and written it into files, it can do one of two things with the files:

1   Store the files on the local disk in a well-defined location.

2   Transfer the files to another machine and store them in a well defined location.

(The data files must be stored in a well-defined location so that rwfilter(1) can find them. To see rwfilter's idea of the well-defined location, run rwfilter --help and read the text after the explanation of the --data-rootdir switch.)

The default output-mode is to store the files on the local disk. When operating in this mode, the following switch is required:

--root-directory=DIR_PATH
    Name the full path of the directory under which the files containing the packed SiLK Flow records will be stored. rwflowpack will create subdirectories below DIR_PATH based on the data received.

  Sending Mode Storage Switches (--output-mode=sending)
To transfer the packed SiLK Flow files to another machine, specify the --output-mode=sending switch and invoke the rwsender(8) to transfer the files. When rwflowpack is interoperating with rwsender, the following three switches must be provided:

--incremental-directory=DIR_PATH
    Name the full path of the directory under which packed SiLK files will initially be created. Files in this directory are considered to be incomplete; any files in this directory will be removed when rwflowpack is started. Once complete, files are moved from this directory to the sender-directory.

--sender-directory=DIR_PATH
    Name the full path of the directory under which completed "incremental" files are stored while awaiting action by rwsender. The rwsender is responsible for removing files from this directory.

FILES
The root of the directory tree that contains the packed, binary SiLK Flow files is set by the --root-directory switch; this directory is called the SILK_DATA_ROOTDIR. Immediately underneath it are subdirectories corresponding to the traffic categories (directions) discussed above. Under these are directories representing the year, month, and day in YYYY/MM/DD format. That is

  $SILK_DATA_ROOTDIR/in/{$YEAR}/{$MONTH}/{$DAY}/*
  $SILK_DATA_ROOTDIR/inweb/{$YEAR}/{$MONTH}/{$DAY}/*
  $SILK_DATA_ROOTDIR/innull/{$YEAR}/{$MONTH}/{$DAY}/*
  $SILK_DATA_ROOTDIR/out/{$YEAR}/{$MONTH}/{$DAY}/*
  $SILK_DATA_ROOTDIR/outweb/{$YEAR}/{$MONTH}/{$DAY}/*
  $SILK_DATA_ROOTDIR/outnull/{$YEAR}/{$MONTH}/{$DAY}/*

For example, output web files for October 4th, 2003 are recorded in $SILK_DATA_ROOTDIR/outweb/2003/10/04/

The names of the files in these directories include all of this information, and are written in the form:

  I<flowType>-I<sensorName>_YYYYMMDD.HH

where flowType encodes the category and sensorName is the sensor on which the flow was collected.

SEE ALSO
SiLK Installation Handbook, sensor.conf(5), flowcap(8), rwfilter(1), rwflowappend(8), rwreceiver(8), rwsender(8), syslog(3)