RWFILTER(1)

NAME
rwfilter - Choose which SiLK Flow records to process

SYNOPSIS
  rwfilter [--dynamic-library=DYNLIB_FILENAME]
        [--pass-destination=PASS_PATH]
        [--fail-destination=FAIL_PATH] [--all-destination=ALL_PATH]
        [--input-pipe=INPUT_PATH] [--xargs=INPUT_STREAM]
        { [--print-statistics] | [--print-volume-statistics] }
        [--print-filenames] [--dry-run] [--num-records=N]
        [--compression-method=COMP_METHOD]
        [--start-date=YYYY/MM/DD[:HH] [--end-date=YYYY/MM/DD[:HH]]]
        [--class=CLASS] [--type={all | TYPE[,TYPE ...]}]
        [--sensors=n1[,n2 ...]]
        [--data-rootdir=PATH] [--site-config-file=FILENAME]
        [--print-missing-filenames]
        [--stime=DATE_RANGE] [--etime=DATE_RANGE]
        [--active-time=DATE_RANGE] [--duration=INTEGER_RANGE]
        [--sport=INTEGER_LIST] [--dport=INTEGER_LIST]
        [--aport=INTEGER_LIST] [--protocol=INTEGER_LIST]
        [--icmp-type=INTEGER_LIST] [--icmp-code=INTEGER_LIST]
        [--bytes=INTEGER_RANGE] [--packets=INTEGER_RANGE]
        [--bytes-per-packet=DECIMAL_RANGE]
        [--saddress=IP_ADDR_MASK] [--not-saddress=IP_ADDR_MASK]
        [--daddress=IP_ADDR_MASK] [--not-daddress=IP_ADDR_MASK]
        [--any-address=IP_ADDR_MASK] [--not-any-address=IP_ADDR_MASK]
        [--next-hop-id=IP_ADDR_MASK] [--not-next-hop-id=IP_ADDR_MASK]
        [--sipset=IP_SET_FILENAME] [--not-sipset=IP_SET_FILENAME]
        [--dipset=IP_SET_FILENAME] [--not-dipset=IP_SET_FILENAME]
        [--anyset=IP_SET_FILENAME] [--not-anyset=IP_SET_FILENAME]
        [--nhipset=IP_SET_FILENAME] [--not-nhipset=IP_SET_FILENAME]
        [--input-index=INTEGER_LIST] [--output-index=INTEGER_LIST]
        [--tcp-flags=TCP_FLAGS] [--flags-all=HIGH_MASK_FLAGS]
        [--fin-flag=SCALAR] [--syn-flag=SCALAR] [--rst-flag=SCALAR]
        [--psh-flag=SCALAR] [--ack-flag=SCALAR] [--urg-flag=SCALAR]
        [--ece-flag=SCALAR] [--cwr-flag=SCALAR]
        [--flags-initial=HIGH_MASK_FLAGS]
        [--flags-session=HIGH_MASK_FLAGS]
        [--attributes=ATTRIBUTES] [--application=INTEGER_LIST]
        [--scc=COUNTRY_CODE_LIST] [--dcc=COUNTRY_CODE_LIST]
        [--stype=SCALAR] [--dtype=SCALAR]

DESCRIPTION
rwfilter serves two purposes: (1) It acts as an interface to the data store to select which SiLK Flow records to process, and (2) it partitions those records into one or more pass and/or fail streams.

The selection switches let one choose records by where the flow was collected (its sensor), the date of collection, and the flow's direction.

The partitioning switches describe various types of traffic behavior (e.g., TCP traffic, or all traffic going to port 80). rwfilter identifies records matching or violating the behavior(s), and partitions them into appropriate output streams (i.e., files) as specified.

These output streams from rwfilter are always binary. The output must be passed through another tool in the SiLK Tool Suite for further processing to get human-readable output.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

  Output Switches
At least one of the following output switches must be provided:

--pass-destination=PASS_PATH
    PASS_PATH refers to a non-existent file, a named pipe, or "stdout". The pass-destination will output records which have passed ALL of the partitioning predicates.

--fail-destination=FAIL_PATH
    FAIL_PATH refers to a non-existent file, a named pipe, or "stdout". The fail-destination will output records which failed ANY of the partitioning predicates.

  Additional I/O Switches

--all-destination=ALL_PATH
    ALL_PATH refers to a file, a named pipe, or stdout. This output will output all records read by rwfilter.

--input-pipe=INPUT_PATH
    INPUT_PATH is a named pipe or the string "stdin". This refers to another source of rwfilter records. Note that rwfilter will not read from the standard input by default, to get this behavior, you must use --input-pipe=stdin.

--xargs=INPUT_PATH
    Causes rwfilter to read file names from INPUT_PATH; the input should have one file name per line. rwfilter will open each file in turn and read records from it.

--print-statistics
--print-statistics=PATH
    Prints out the statistics on files read - the number of records which passed, the number which failed and the total read. If a PATH is provided, the statistics will be printed there; otherwise they are printed to the standard error.

--print-volume-statistics
--print-volume-statistics=PATH
    An enhanced version of --print-statistics, in that the statistics include the number of records, packets, and bytes that passed and failed the filter.

--print-filenames
    Print the names of input files as they are read

--dry-run
    Perform a sanity check on the input arguments to check that the arguments are acceptable.

--max-pass-records=N
    Stop reading input after N records have been written to the pass-destination.

--compression-method=COMP_METHOD
    Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

    none
        Do not compress the output using an external library

    zlib
        Use the zlib(3) library for compressing the output

    lzo1x
        Use the lzo1x algorithm from the LZO real time compression library for compression

    best
        Use whichever available method gives the "best" compression in general, though not necessarily the "best" for this particular output.

  Selection Options
The following options determine which files are read from the data store to provide the records.

--start-date=YYYY/MM/DD[:HH]
--end-date=YYYY/MM/DD[:HH]
    The date predicates indicate which time to start and end the search; these predicates are expressed in "YYYY/MM/DD:HH" format. In all cases, express values less than 10 with a zero, so 09 for 9, 08 for 8, and so on.

    For example, "2003/01/18:00" represents the first hour of January 18th, 2003, while "2002/10/01:22" corresponds to 22:00 GMT on October 1st, 2002.

    When the hour of the start-date is given and end-date is not specified, files for that single hour are processed.

    When the hour of the start-date is not given, the hour of the end-date is ignored, and files for all dates between midnight on start-date and 23:59 on end-date are processed.

    When --start-date is not given, rwfilter processes all files for the current day.

--class=CLASS
    CLASS is used to select groups of data. Currently only a single class may be selected. If the --class option is not given, a class is selected by default. Use the --help option to see the list of available classes and the default class.

--type={"all" | TYPE[,TYPE ...]}
    The --type predicate is used to further specify data by specifying the TYPE of traffic using the scheme for your deployment. TYPEs typically refer to the direction of the flow; TYPEs depend on the class and on the site where SiLK is installed. The switch takes a comma-separated list of types or the keyword "all" which specifies all types for the specified class. If the --type switch is not given, a list of default types is used. Use the --help option to get the list of available types for each class.

--sensors=n1[,n2 ...]
    Sensor is used to select data files from specific sensors. This is a comma separated list of sensor names and/or sensor IDs (integers) that will depend on your installation. If not given, the default is all sensors.

--data-rootdir=PATH
    This option causes rwfilter to use PATH as the root of the data store directory, which overrides the location given in the $SILK_DATA_ROOTDIR environment variable, which overrides the location that was compiled into rwfilter. The default data store directory is available via the --help option.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the $SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of $SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the $SILK_DATA_ROOTDIR environment variable; the data root directory that is compliled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/, $SILK_PATH/share/, or $SILK_PATH; or in the share/silk/ and share/ directories parallel to the application's directory.

--print-missing-files
    This option prints to the standard error file names that the selection engine expected to find but did not. This list can be misleading, so use it judiciously.

  Partitioning Switches
rwfilter supports the following partitioning switches, at least one of which must be specified. The switches are AND'ed together; i.e., to pass the filter, the record must pass the test implied by each switch. Any record that does not pass will be sent to the fail-destination(s), if specified.

SWITCH PARAMETERS

The forms of the parameters to these partitioning switches are:

•   DATE_RANGE is a range of two dates, start-range and end-range, each in the form "YYYY/MM/DD[:HH[:MM[:SS[.ssssss]]]]", for example "2003/01/31:23:45:00.000-2003/01/31:23:59:59.999" represents the last fifteen minutes of Jan 31, 2003. The start-range and end-range must be set to at least day precision. For the start-range, unspecified hour, minute, second, and millisecond values set to 0; for the end-range, those values are set to 23, 59, 59, and 999 respectively. Thus "2003/01/31:23-2003/01/31:23" will become "2003/01/31:23:00:00.000-2003/01/31:23:59:59.999". If an end-range is not given, it is set to the start-range, giving a range of a single millisecond.

•   SCALAR is a single integer; for example 4.

•   INTEGER_RANGE is a range of two positive integers: MIN-MAX; for example "1-500". For many options, the upper limit of the range may be omitted, such as "1-".

•   INTEGER_LIST is a comma separated list of SCALARs and INTEGER_RANGEs; for example, "1,2,3,5-10,99-103".

•   DECIMAL_RANGE is a range of decimal values with accuracy up to 10^-4 expressed as MIN-MAX; for example, "5.0-10.031".

•   IP_ADDR_MASK are expressed in one of two forms. As CIDR blocks (192.168.0.0/16) or as four INTEGER_LISTs joined by dot ".". The character "x" can be used as an abbreviation for 0-255. For example, "10.10,16-31.x.x" represents the following CIDR blocks:

      10.10.0.0/16
      10.16.0.0/20

•   TCP_FLAGS is any combination of the letters "F","S","R","P","A","U","E","C", where "F"=FIN flag; "S"=SYN; "R"=RST; "P"=PSH; "A"=ACK; "U"=URG; "E"=ECE; "C"=CWR

•   HIGH_MASK_FLAGS is a pair of TCP_FLAGS strings separated by a slash ("/"). Flags to the right of the slash are the mask; any flag not listed in the mask may have any value in the flow. Flags to the left of the slash are the expected high flags; they must be set in the flow. Thus, flags listed in mask but not in high must be off for all packets in the flow. It is an error if a flag is listed in high but not in mask. Some examples:

    "AS/ASFR"
        means ACK,SYN must be high, FIN,RST must be low, and the other flags (PSH, URG, ECE, CWR) may have any value.

    "A/A"
        means the ACK packet must be SET. All other flags may have any value.

    "/F"
        means the FIN packet must be OFF. All other flags may have any value.

    "F/S"
        is an error; use "F/FS" instead, which means FIN must be high, SYN must be low, and other flags can have any value.

•   IP_SET_FILENAME is the name of a file containing a binary IPset. Binary IPsets are created from rwfilter output with the rwset tool, or from text input with the rwsetbuild tool.

•   COUNTRY_CODE_LIST a comma separated list of lowercase two-letter country codes, as well as the following special codes:

    "--"
        N/A (e.g. private and experimental reserved addresses)

    "a1"
        anonymous proxy

    "a2"
        satellite provider

    "o1"
        other

    An example: "cx,uk,kr,jp,--"

•   ATTRIBUTES is any combination of the letters "F","T","C", where

    "F" collector saw additional packets in the flow following a packet with a FIN flag (excluding ACK)

    "T" collector prematurely created a record for a long-running connection due to a timeout

    "C" this flow is a continuation of flow that the collector was forced to close due to a timeout

SWITCHES

The switches are:

--stime=DATE_RANGE
    Pass the record if its starting time is in this DATE_RANGE.

--etime=DATE_RANGE
    As --stime for the ending time.

--active-time=DATE_RANGE
    Pass the record if the record was active at ANY time during this DATE_RANGE. If a single time is specified, pass the record if it was active at that instant.

--duration=INTEGER_RANGE
    Pass the record if its duration (eTime-sTime) is in this INTEGER_RANGE.

--sport=INTEGER_LIST
    Pass the record if its source port is in this INTEGER_LIST, possible values are 0-65535.

--dport=INTEGER_LIST
    Pass the record if its destination port is in this INTEGER_LIST, possible values are 0-65535

--aport=INTEGER_LIST
    Pass the record if its source port and/or its destination port is in this INTEGER_LIST, possible values are 0-65535. For example, use --aport=25 to see all SMTP conversions regardless or where they originated.

--protocol=INTEGER_LIST
    Pass the record if its IP Suite Protocol is in this INTEGER_LIST, possible values are 0-255.

--icmp-type=INTEGER_LIST
    Pass the record if its ICMP type is in this INTEGER_LIST; possible values 0-255. This switch will act as if "--protocol=1" has been specified; it is an error to specify any other values for the protocol.

--icmp-code=INTEGER_LIST
    Pass the record if its ICMP code is in this INTEGER_LIST; possible values 0-255. This switch will act as if "--protocol=1" has been specified; it is an error to specify any other values for the protocol.

--bytes=INTEGER_RANGE
    Pass the record if its byte count is in this INTEGER_RANGE.

--packets=INTEGER_RANGE
    Pass the record if its packet count is in this INTEGER_RANGE.

--bytes-per-packet=DECIMAL_RANGE
    Pass the record if its average bytes per packet count (bytes/packet) is in this DECIMAL_RANGE.

--saddress=IP_ADDR_MASK
    Pass the record if its source IP address is matched by this IP_ADDR_MASK. To match on multiple IPs, use an IPset (see --sipset).

--daddress=IP_ADDR_MASK
    Pass the record if its destination IP address is matched by this IP_ADDR_MASK (see also --dipset).

--any-address=IP_ADDR_MASK
    Pass the record if either its source or its destination IP address is matched by this IP_ADDR_MASK (see also --anyset). Does not consider the next-hop IP address.

--not-saddress=IP_ADDR_MASK
    Pass the record if its source IP address is not matched by this IP_ADDR_MASK (see also --not-sipset).

--not-daddress=IP_ADDR_MASK
    Pass the record if its destination IP address is not matched by this IP_ADDR_MASK (see also --not-dipset).

--not-any-address=IP_ADDR_MASK
    Pass the record if neither its source nor its destination IP address is matched by this IP_ADDR_MASK (see also --not-anyset). Does not consider the next-hop IP address.

--sipset=IP_SET_FILENAME
    Pass the record if its source IP address is in the list of IPs contained in the binary set file IP_SET_FILENAME

--dipset=IP_SET_FILENAME
    As --sipset for the destination IP address.

--anyset=IP_SET_FILENAME
    Pass the record if either its source IP address or its destination IP address is in the list of IPs contained in the binary set file IP_SET_FILENAME. Does not consider the next-hop IP.

--nhipset=IP_SET_FILENAME
    As --sipset for the next-hop IP address.

--not-sipset=IP_SET_FILENAME
    Pass the record if its source IP address is not in the list of IPs contained in the binary set file IP_SET_FILENAME

--not-dipset=IP_SET_FILENAME
    As --not-sipset for the destination IP address.

--not-anyset=IP_SET_FILENAME
    Pass the record if neither its source IP address nor its destination IP address is in the list of IPs contained in the binary set file IP_SET_FILENAME. Does not consider the next-hop IP.

--not-nhipset=IP_SET_FILENAME
    As --not-sipset for the next-hop IP address.

--tcp-flags=TCP_FLAGS
    Pass the record if, for any one of its packets, any of the specified TCP_FLAGS was on.

--flags-all=HIGH_MASK_FLAGS
    HIGH_MASK_FLAGS is a set of HIGH_FLAGS/MASK_FLAGS; HIGH_FLAGS must be a subset of MASK_FLAGS. Pass the record if the flags listed in HIGH_FLAGS are set and the flags listed in MASK_FLAGS but not listed in HIGH_FLAGS are not-set. This switch may be repeated up to eight times, so that "--flags-all=S/S --flags-all=A/A" will pass flows that have either only-SYN high or only-ACK high.

--fin-flag=SCALAR
    Set to 0, only passes records where the FIN Flag is Low, Set to 1, only passes records where the FIN Flag is high.

--syn-flag=SCALAR
    As --fin-flag except for the SYN Flag

--rst-flag=SCALAR
    As --fin-flag except for the RST Flag

--psh-flag=SCALAR
    As --fin-flag except for the PSH Flag

--ack-flag=SCALAR
    As --fin-flag except for the ACK Flag

--urg-flag=SCALAR
    As --fin-flag except for the URG Flag

--ece-flag=SCALAR
    As --fin-flag except for the ECE Flag

--cwr-flag=SCALAR
    As --fin-flag except for the CWR Flag

--dynamic-library=DYNLIB_FILENAME
    Augment the partitioning switches by using run-time loading of the plug-in (shared object) whose path is "DYNLIB_FILENAME". The creation of thes plug-ins is beyond the scope of this manual page; the process is described in Analysts' Handbook: Using SiLK for Network Traffic Analysis.

SiLK can store flows generated by enhanced collection software that provides more information than NetFlow v5. These flows may support some or all of these additional switches; for flows without this additional information, the field's value is always 0.

--flags-initial=HIGH_MASK_FLAGS
    As --flags-all, except this switch considers only the initial packet in the flow.

--flags-session=HIGH_MASK_FLAGS
    As --flags-all, except this switch ignores the initial packet in the flow.

--attributes=HIGH_ATTRIBUTES/CARE_ATTRIBUTES
    Passes the flow if the attribute of the flow matches this ATTRIBUTE. Attributes are "F","T","C"; see above for a description of these values.

--application=INTEGER_LIST
    Passes the flow if the application that the flow collection software assigned to the flow is in the specified INTEGER_LIST. Some flow generation software will guess the application based on the contents of the packets that make up the flow. This will be the standard port that application; for example, HTTP traffic on non-standard ports will have an application of 80.

For the following three filter tests, some file formats do not store these values, in which case the value is always 0:

--next-hop-id=IP_ADDR_MASK
    Pass the record if its next hop IP address is matched by this IP_ADDR_MASK.

--not-next-hop-id=IP_ADDR_MASK
    Pass the record if its next hop IP address is not matched by this IP_ADDR_MASK.

--input-index=INTEGER_LIST
    Pass the record if its incoming SNMP interface is in this INTEGER_LIST.

--output-index=INTEGER_LIST
    Pass the record if its outgoing SNMP interface is in this INTEGER_LIST.

Additional filtering switches are provided by run-time loading of plug-ins (shared object files or dynamic libraries) when the plug-in is available. rwfilter automatically looks for the following plug-ins:

ADDRESS TYPE (libaddrtype.so)

--stype=SCALAR
    When SCALAR is 0, pass the record if its source IP address is non-routable. When 1, pass if internal. When 2, pass if external (i.e., routable but not internal). When 3, pass if not internal (non-routable or external). See libaddrtype(3).

--dtype=SCALAR
    As --stype for the destination IP address.

COUNTRY CODE (libccfilter.so)

--scc=COUNTRY_CODE_LIST
    Pass the record if the country code of its source IP address is in the specified COUNTRY_CODE_LIST. See libccfilter(3).

--dcc=COUNTRY_CODE_LIST
    As --scc for the destination IP address.

PREFIX MAP (libpmapfilter.so)

--pmap-file=FILENAME
    FILENAME refers to a prefixmap file generated using rwpmapbuild(1). This switch must precede all other --pmap-* switches. See libpmapfilter(3).

--pmap-saddress=LABELS
    For an IP prefix map, pass the record if the source IP address maps to a label contained in the list of labels in LABELS.

--pmap-daddress=LABELS
    As --pmap-saddress for the destination IP address.

--pmap-sport-proto=LABELS
    For a port/protocol map, pass the record if the source port and protocol combination maps to a label contained in the list of labels in LABELS.

--pmap-dport-proto=LABELS
    As --pmap-sport-proto for destination port and protocol.

IPPORT (libipport.so)

This plug-in provides support for filtering by IP-port pairs. To use this plug-in, one creates a text file where each line contains a single IP address (either in dotted-decimal notation or as an integer), white space, and a list of ports of interest for that IP. The port list can be a single number (80), a range of numbers ("6000-6100"), or comma-separated list of numbers and ranges ("6000-6100,80"). The file may also contain blank lines and comments; comments begin with the "#" character and continue to the end of the line.

--ipport-any=ipport_filename
    Pass the record if either the source IP and port or the destination IP and port are listed in the text file ipport_filename.

EXAMPLES
The most basic filtering involves looking at specific traffic over a specific time. For example:

  rwfilter --start-date=2003/02/19:00 --end-date=2003/02/19:23 \
        --pass=alltcp.rwf --proto=6

will create a file, alltcp.rwf containing all TCP traffic. This filter file will not be readable with cat, but can be examined using rwcut(1).

Please note that the output file described above could be extremely large.

Once a file is written, rwfilter can filter the file again, for example:

  rwfilter --aport=80 alltcp.rwf --pass=allweb.rwf

will generate allweb.rwf. This progressive filtering can also be done at the command line, but the interim files can be examined with rwcut, rwuniq and other tools.

Multiple filters can be chained at the command line using pipes:

  rwfilter --start-date=2003/02/19:00 --end-date=2003/02/19:23 \
        --proto=6 --pass=stdout | \
        rwfilter --input-pipe=stdin --aport=80 --packets=1-5 \
        --pass=smallweb.rwf

ENVIRONMENT

SILK_DATA_ROOTDIR
    When set, overrides the compiled-in value for the location of the directory tree containing the files of SiLK Flow records collected and stored by the packing system (rwflowpack(8)).

SILK_PATH
    When set, rwfilter looks for plug-ins in $SILK_PATH/share/lib and $SILK_PATH/lib, and it looks for configuration files in $SILK_PATH/share/silk, $SILK_PATH/share, and $SILK_PATH.

SILK_DYNLIB_DEBUG
    When set to 1, rwfilter prints status messages to the standard error as it tries to open each of its plug-ins.

NOTES
rwfilter is the most commonly used application in the suite. It provides access to the data files and performs all the basic queries.

rwfilter supports a variety of I/O options - in addition to reading from the data store, rwfilter results can be chained together with named pipes to output results to multiple files simultaneously. An introduction to named pipes is outside the scope of this document, however.

Two often underused options are --dry-run and --print-statistics

--dry-run does a sanity check on the input arguments and should be used, especially for complicated arguments, to check that the arguments are acceptable.

--print-statistics used without --pass-destination or --fail-destination simply dumps aggregate statistics to stderr (not stdout) in the following format:

  File <#input files> Read <# of recs read> \
  Pass <# of recs passing the filter> \
  Fail <# of recs failing the filter>

and can be used to do a quick pass through the data to get aggregate counts before going in deeper into the phenomenon being investigated.

--print-filename can be used as a progress meter; during long jobs, it shows which file is currently being read by the application. --print-filename will not provide meaningful results with piped input.

Filters are applied in the order given on the command line. It is best to apply the biggest filters first.

The switches used to create a filter output file are stored in the file itself. Use the rwfileinfo(1) command to see this information.

SEE ALSO
Analysts' Handbook: Using SiLK for Network Traffic Analysis, rwcount(1), rwcut(1), rwfileinfo(1), rwset(1), rwsort(1), rwstats(1), rwtotal(1), rwuniq(1), rwsetbuild(1), libaddrtype(3), libccfilter(3), libpmapfilter(3)