RWUNIQ(1)

NAME
rwuniq - Bin SiLK Flow records by a key and print the bins' volume

SYNOPSIS
  rwuniq --fields=KEY [--dynamic-library=DYNLIB]
        [--presorted-input] [--all-counts]
        [{--flows | --flows=MIN | --flows=MIN-MAX}]
        [{--packets | --packets=MIN | --packets=MIN-MAX}]
        [{--bytes | --bytes=MIN | --bytes=MIN-MAX}]
        [--stime] [--etime]
        [{--sip-distinct | --sip-distinct=MIN | --sip-distinct=MIN-MAX}]
        [{--dip-distinct | --dip-distinct=MIN | --dip-distinct=MIN-MAX}]
        [{--bin-time | --bin-time=SECONDS}] [--epoch-time]
        [{--integer-ips | --zero-pad-ips}] [--integer-sensors]
        [--no-titles] [--no-columns] [--column-separator=CHAR]
        [--no-final-delimiter] [{--delimited | --delimited=CHAR}]
        [--print-filenames] [--copy-input=PATH] [--output-path=PATH]
        [--pager=PAGER_PROG]
        [--ipv6-policy={ignore,asv4,mix,force,only}]
        [--temp-directory=DIR_PATH] [--site-config-file=FILENAME]
        [--pmap-file=PATH] [--pmap-column-width=NUM]
        [--python-file=PATH]
        [{--legacy-timestamps | --legacy-timestamps=NUM}] [FILES...]

DESCRIPTION
rwuniq reads SiLK Flow records and bins them by a key composed of user-specified attributes of the flows. For each bin, a volume is computed, where the volume is any combination of flow count, packet sum, byte sum, earliest start time, most recent end time, distinct source address count, and/or distinct destination address count. Once all the input is processed, the key fields and the volumes are printed for each bin that meets the user-specified minima and maxima.

The SiLK Flow records are read from the files named on the command line or from the standard input when no file names are given and the standard input is not a terminal.

The flow attribute(s) (or field(s)) that make up the key for each bin are selected by the user, with the available fields being similar to those supported by rwcut. See the description of the --fields switch in the OPTIONS section below for the names of the keys. It is an error if the user does not specify the --fields switch. The size of the key is basically unlimited, but a larger key will more quickly use the available the memory leading to slower performance. Likewise, requesting more volume fields also slows performance.

When no volume fields are requested, rwuniq computes the number of flows for each bin and prints all bins that have at least one flow.

The --presorted-input switch allows rwuniq to efficiently process data that has been previously sorted with the rwsort(1) command. With this switch, rwuniq does not need large amounts of memory because it does not bin each flow; instead, it keeps running summations and outputs a bin whenever the key changes. For the output to be meaningful, rwsort and rwuniq must be invoked with the same --fields value.

rwuniq attempts to keep all key and volume data in the computer's memory. If rwuniq runs out of memory, the current key and volume data is written to a temporary file. Once all input has been processed, the data from the temporary files is merged to produce the final output. By default, these temporary files are stored in the /tmp directory. Because these files can be large, it is strongly recommended that /tmp not be used as the temporary directory, and rwuniq will print a warning when /tmp is used. To modify the temporary directory used by rwuniq, provide the --temp-directory switch, set the SILK_TMPDIR environment variable, or set the TMPDIR environment variable.

When SiLK is compiled with IPv6 support, use of --sip-distinct and --dip-distinct switches is limited. Specifically, only one distinct IP count is supported for unsorted input, and no distinct IP counts are supported when when --presorted-input is given. Setting the --ipv6-policy switch to "ignore" or "asv4" will get around this limitation at the expense of ignoring IPv6 addresses.

rwuniq may run out of memory when computing distinct IP counts, causing the counts for some bins to be smaller than the actual number of distinct IPs. When this occurs, a single warning is printed the standard error noting that rwuniq has run out of memory, processing continues, and rwuniq exits with status 16.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

The --fields switch is required. rwuniq will fail when it is not provided.

--fields=KEY
    KEY contains the list of flow attributes (a.k.a. fields or columns) that make up the key into which flows are binned. The columns will be displayed in the order the fields are specified. Each field may be specified once only.

    KEY is a comma separated list of field-names, field-integers, and ranges of field-integers; a range is specified by separating the start and end of the range with a hyphen (-), e.g.,

      --fields=stime,10,1-5

    There is no default value for the --fields switch; the switch must be specified.

    The complete list of built-in fields that the SiLK tool suite supports follows, though note that not all fields are present in all SiLK file formats; when a field is not present, its value is 0.

    sIP,sip,1
        source IP address

    dIP,dip,2
        destination IP address

    sPort,sport,3
        source port for TCP and UDP, or equivalent

    dPort,dport,4
        destination port for TCP and UDP, or equivalent

    protocol,5
        IP protocol

    packets,pkts,6
        packet count

    bytes,7
        byte count

    flags,8
        bit-wise OR of TCP flags over all packets

    sTime,stime,9
        starting time of flow (seconds resolution)

    dur,10
        duration of flow (seconds resolution)

    eTime,etime,11
        end time of flow (seconds resolution)

    sensor,12
        name or ID of sensor at the collection point

    class,20
        class of sensor at the collection point

    type,21
        type of sensor at the collection point

    icmpTypeCode,icmptypecode,25
        include two columns, "iType" and "iCode" that contain the ICMP type and code for ICMP flows; for non-ICMP flows, these columns are empty

    Many SiLK file formats do not store the following fields and their values will always be 0; they are listed here for completeness:

    in,13
        router SNMP input interface

    out,14
        router SNMP output interface

    nhIP,15
        router next hop IP

    SiLK can store flows generated by enhanced collection software that provides more information than NetFlow v5. These flows may support some or all of these additional fields; for flows without this additional information, the field's value is always 0.

    initialFlags,initialflags,26
        TCP flags on first packet in the flow

    sessionFlags,sessionflags,27
        bit-wise OR of TCP flags over all packets except the first in the flow

    attributes,28
        flow attributes set by flow collector:

        "T" flow collector generated a flow record for a long-running connection due to timeout.

        "C" this flow is a continuation of a long-running connection that the collector terminated.

        "F" additional non-ACK packets seen after a packet with the FIN flag set.

    application,29
        guess as to the application generating the flow; value will be standard port for the application, such as 80 for web traffic

    The list of built-in fields may be augmented by run-time loading of plug-ins (shared object files or dynamic libraries) when the plug-in is available. rwuniq automatically looks for the following plug-ins:

    ADDRESS TYPE (addrytype.so)

    stype,16
        for the source IP address, the value 0 if the address is non-routable, 1 if it is internal, or 2 if it is routable and external. See addrtype(3).

    dtype,17
        as stype for the destination IP address

    COUNTRY CODE (ccfilter.so)

    scc,18
        for the source IP, a two-letter country code abbreviation denoting the country who "owns" that IP address. See ccfilter(3).

    dcc,19
        as scc for the destination IP

    PREFIX MAP (pmapfilter.so)

    sval
        value from the user-defined mapping (see the --pmap-file switch) for the source. For an IP-based map, this corresponds to sip. For a proto-port-based map, it is protocol/sport. See pmapfilter(3)

    dval
        as sval for the destination IP or proto/dport.

--dynamic-library=DYNLIB
    Augment the list of fields by using run-time loading of the plug-in (shared object) whose path is DYNLIB. The creation of these plug-ins is beyond the scope of this manual page. When DYNLIB contains a slash ("/"), rwuniq assumes the path to DYNLIB is correct. Otherwise, rwuniq will attempt to find the file in $SILK_PATH/lib/silk, $SILK_PATH/share/lib, $SILK_PATH/lib, and in these directories parallel to the application's directory: lib/silk, share/lib, and lib. If rwuniq does not find the file, it assumes the plug-in is in the current directory. To force rwuniq to look in the current directory first, specify --dynamic-library=./DYNLIB. When the SILK_DYNLIB_DEBUG environment variable is non-empty, rwuniq prints status messages to the standard error as it tries to open each of its plug-ins.

--presorted-input
    Cause rwuniq to assume that it is reading sorted input; i.e., that rwuniq's input was generated by rwsort(1) using the exact same value for the --fields switch. This option allows rwuniq to process an endless stream of records.

The next eight options determine what summations are computed and printed. If none of these eight options are provided, flows are counted for consistency with earlier versions of rwuniq, though this behavior may change in a future release.

--all-counts
    Enable the next five sets of options with their default thresholds; i.e., all possible counts (except the distinct counts) are computed and printed.

--flows
--flows=MIN
--flows=MIN-MAX
    Cause rwuniq to sum the number of flow records in each uniquely keyed bin. When MIN is provided, bins are printed only when they had at least MIN number of flows. When MAX is also provided, bins are printed only when they had no more than MAX flows. A MIN of 0 is treated as 1. When MIN is not provided, a default of 1 is used.

--packets
--packets=MIN
--packets=MIN-MAX
    Cause rwuniq to sum, for each unique key, the number of packets in each flow record. When MIN is provided, keys are printed only when they had at least MIN sum of packets. When MAX is also provided, bins are printed only when they had no more than MAX sum of packets. A MIN of 0 is treated as 1. When MIN is not provided, a default of 1 is used.

--bytes
--bytes=MIN
--bytes=MIN-MAX
    Cause rwuniq to total, for each unique key, the number of bytes in each flow record. When MIN is provided, keys are printed only when they had at least MIN total bytes. When MAX is also provided, bins are printed only when they had no more than MAX total bytes. A MIN of 0 is treated as 1. When MIN is not provided, a default of 1 is used.

--stime
    Cause rwuniq to keep track of the earliest time at which it saw a flow that matched each bin's unique key. This option does not support thresholds.

--etime
    Cause rwuniq to keep track of the latest (most recent) time at which it saw a flow that matched each bin's unique key. This option does not support thresholds.

--sip-distinct
--sip-distinct=MIN
--sip-distinct=MIN-MAX
    Cause rwuniq to count the number of distinct source IP addresses that were seen for each uniquely keyed bin. When MIN is provided, bins are printed only when they had at least MIN distinct sources. When MAX is also provided, bins are printed only when they had no more than MAX distinct sources. A MIN of 0 is treated as 1. When MIN is not provided, a default of 1 is used. When this switch is provided, the sIP field cannot be part of the key.

--dip-distinct
--dip-distinct=MIN
--dip-distinct=MIN-MAX
    As --sip-distinct for destination IP addresses.

--bin-time
--bin-time=SECONDS
    Adjust the key fields 'sTime' and 'eTime' to appear on SECONDS-second boundaries (the floor of the time is used). When no value is provided to the switch, 60-second time bins are used. (When the start-time is the only key field and time binning is desired, consider using rwcount(1) instead.)

The appearance of the output is controlled by this next set of switches:

--epoch-time
    Print timestamps as epoch time (number of seconds since midnight GMT on 1970-01-01).

--integer-ips
    Print IPs as integers. By default, IP addresses are printed in their canonical form.

--zero-pad-ips
    Print IP addresses in their canonical form, but add zeros to the IP address so it fully fills the width of column. For IPv4, use three digits per octet, e.g, 127.000.000.001. For IPv6, use four digits per hexadectet and expand empty hexadectets, e.g.; "0000:0000:0000:0000:0000:FFFF:FF00:0001".

--integer-sensors
    Print the integer ID of the sensor rather than its name.

--no-titles
    Turn off column titles. By default, titles are printed.

--no-columns
    Disable fixed-width columnar output.

--column-separator=C
    Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.

--no-final-delimiter
    Do not print the column separator after the final column. Normally a delimiter is printed.

--delimited
--delimited=C
    Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

--print-filenames
    Prints to the standard error the names of input files as they are opened.

--copy-input=PATH
    Copy all binary input to the specified file or named pipe. PATH can be "stdout" to print flows to the standard output as long as the --output-path switch has been used to redirect rwuniq's ASCII output.

--output-path=PATH
    Determines where the output of rwuniq (ASCII text) is written. If this option is not given, output is written to the standard output.

--pager=PAGER_PROG
    When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the value of the pager is determined to be the empty string, no paging will be performed and all output will be printed to the terminal.

--ipv6-policy=POLICY
    Determine how IPv4 and IPv6 flows are handled when SiLK has been compiled with IPv6 support. When the switch is not provided, the SILK_IPV6_POLICY environment variable is checked for a policy. If it is also unset or contains an invalid policy, the POLICY is mixed. When SiLK has not been compiled with IPv6 support; IPv6 flows are always ignored, regardless of the value passed to this switch or in the SILK_IPV6_POLICY variable. The supported values for POLICY are:

    ignore
        Completely ignore IPv6 flows. Only IPv4 flows will be printed.

    asv4
        Convert IPv6 addresses to IPv4 if possible, otherwise ignore the IPv6 flows.

    mix Process the input as a mixture of IPv4 and IPv6 flows.

    force
        Force IPv4 flows to be converted to IPv6.

    only
        Only process flows that were marked as IPv6 and completely ignore IPv4 flows.

--temp-directory=DIR_PATH
    Specify the name of the directory in which to store data files temporarily when the memory is not large enough to store all the bins and their volumes. This switch overrides the directory specified in the SILK_TMPDIR environment variable, which overrides the directory specified in the TMPDIR variable, which overrides the default, /tmp.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application's directory.

--legacy-timestamps
--legacy-timestamps=NUM
    Specify the format for human readable timestamps, either the default (new) style, "YYYY/MM/DDThh:mm:ss", or the legacy style, "MM/DD/YYYY hh:mm:ss". When this switch is not present, the timestamps will be in the default format. When this switch is present and no argument is given, timestamps are in the legacy format. When an argument is supplied, timestamps will be in the new format if the argument begins with 0, and in the old format if the argument begins with 1. Any other argument to the switch is an error.

--pmap-file=PATH
    When the pmapfilter(3) plug-in is used, this switch gives the path to mapping file.

--pmap-column-width=NUM
    When the pmapfilter plug-in is used, this switch gives the maximum number of characters to use when displaying the textual value of any field.

--python-file=PATH
    When the silkpython(3) plug-in is used, rwuniq reads the Python code from the file PATH to define additional fields that can be used as part of the key. This file should call "register_plugin_field" for each field it defines, with the first argument being the name of the field, and the following named arguments:

    "rec_to_text"
        names a function that maps a "silk.RWRec" object to a text representation of the field.

    "rec_to_bin"
        names a function taking a "silk.RWRec" object and returning a string of size "bin_len" representing a binary value for the field that will be used to make the key

    "bin_to_text"
        names a function mapping the binary value that "rec_to_bin" returns to a textual value suitable for printing to the screen

    "text_len"
        an integer specifying the maximum size of the value returned by "bin_to_text"

    "bin_len"
        an integer specifying the size of the value returned by "rec_to_bin"

    See the example below and silkpython(3) for more details.

When none of --flows, --packets, --bytes, --stime, or --etime is specified, rwuniq acts as if --flows=1 was given.

When one of --bytes, --packets, --stime, --etime, --sip-distinct, or --dip-distinct is given, rwuniq will not compute the total flows unless the --flows switch is also specified.

EXAMPLES
To look for the most prevalent source IP addresses (similar to rwstats(1)):

  rwfilter ... --pass=stdout | rwuniq --field=1 | \
        sort -t '|' -r -k 2 | head -11

              sIP|   Records|
         10.1.1.9|       133|
         10.1.1.1|       120|
         10.1.1.5|       100|
         10.1.1.2|        91|
         10.1.1.4|        82|
         10.1.1.6|        64|
         10.1.1.3|        41|
         10.1.1.7|        29|
        10.1.1.10|        28|
         10.1.1.8|        16|

To look for all TCP traffic with less than 3 packets, and which also appears at least 3 times, combine rwfilter and rwuniq:

  rwfilter --start-date=2003/01/01:00 --end-date=2003/01/01:23 \
        --proto=6 --packets=1-3 --pass=stdout \
        | rwuniq --field=1 --threshold=3 | head -11

              sIP|   Records|
         10.1.1.4|         6|
         10.1.1.6|         4|
         10.1.1.1|       102|
         10.1.1.9|         3|
         10.1.1.2|         4|
         10.1.1.8|        17|
        10.1.1.10|         6|
         10.1.1.7|         7|
         10.1.1.3|         7|
         10.1.1.5|      2491|

You can use PySiLK to create arbitary fields to use as part of the key for rwuniq. Create a Python file that calls the "register_plugin_field" function for each field you wish to create. This function will define the name of the field, a function returning a text value from a record, a function returning a binary value used for uniquing, a function from that binary_value to text, and the lengths of the binary value and text field. The following example specifies two fields, "sip-last-octet" and "dip-last-octet":

 def sip_last_octet_bin(rec):
     return chr(int(rec.sip) & 0xff)

 def dip_last_octet_bin(rec):
     return chr(int(rec.dip) & 0xff)

 def sip_last_octet_txt(rec):
     return int(rec.sip) & 0xff

 def dip_last_octet_txt(rec):
     return int(rec.dip) & 0xff

 def bin_to_txt(bin):
     return ord(bin[0])

 register_plugin_field("sip_last_octet", bin_len=1, text_len=3,
                       rec_to_bin=sip_last_octet_bin,
                       rec_to_text=sip_last_octet_txt,
                       bin_to_text=bin_to_txt)

 register_plugin_field("dip_last_octet", bin_len=1, text_len=3,
                       rec_to_bin=dip_last_octet_bin,
                       rec_to_text=dip_last_octet_txt,
                       bin_to_text=bin_to_txt)

To use this code, specify the name of the Python file in the --python-file switch and list the fields in the --fields switch:

 rwuniq --python-file=sort.py --fields=sip_last_octet --flows infile.rwf

This particular example will use the last octet of the source IP address as a key and count the number of flows from infile,rwf that match that key.

ENVIRONMENT

SILK_IPV6_POLICY
    This environment variable is used as the value for the --ipv6-policy when that switch is not provided.

SILK_PAGER
    When set to a non-empty string, rwuniq automatically invokes this program to display its output a screen at a time. If set to an empty string, rwuniq does not automatically page its output.

PAGER
    When set and SILK_PAGER is not set, rwuniq automatically invokes this program to display its output a screen at a time.

SILK_TMPDIR
    When set and --temp-directory is not specified, rwuniq writes the temporary files it creates to this directory. SILK_TMPDIR overrides the value of TMPDIR.

TMPDIR
    When set and SILK_TMPDIR is not set, rwuniq writes the temporary files it creates to this directory.

SILK_CONFIG_FILE
    This environment variable is used as the value for the --site-config-file when that switch is not provided.

PYTHONPATH
    This environment variable is used by Python to locate modules. When --python-file is specified, rwuniq loads Python which in turn loads the PySiLK module which is comprised of several files (silk/pysilk_nl.so, silk/__init__.py, etc). If this silk/ directory is located outside Python's normal search path (for example, in the SiLK installation tree), it may be necessary to set or modify the PYTHONPATH environment variable to include the parent directory of silk/ so that Python can find the PySiLK module.

SILK_DATA_ROOTDIR
    When the --site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwuniq looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH
    This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwuniq checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share. These directories are also searched when any other configuration file is required (e.g., the country code map). In addition, rwuniq looks for plug-ins in $SILK_PATH/lib/silk, $SILK_PATH/share/lib and $SILK_PATH/lib.

SILK_DYNLIB_DEBUG
    When set to 1, rwuniq prints status messages to the standard error as it tries to open each of its plug-ins.

NOTES
If multiple thresholds are given (e.g., "--bytes=80 --flows=2"), the values must meet all thresholds before the record is printed. For example, if a given key saw a single 100-byte flow, the entry would not printed given the switches above.

rwuniq functionally replaces the combination of

  rwcut | sort | uniq -c

however, rwuniq does not sort its output.

To sort the output of rwuniq that includes IP numbers, consider using the --zero-pad-ips switch or the --integer-ips switch. To convert integer IPs back to dotted-decimal notation, use num2dot(1):

  rwuniq --fields=1 --integer-ip | sort -n | num2dot

To get a list of unique IP addresses in a data set without the counting or threshold abilities of rwuniq, consider using IPsets:

  rwset --sip-set=stdout | rwsetcat --print-ips

For situations where the key and value are each a single field, consider using Bags:

  rwbag --sip-bytes=stdout | rwbagcat

A Top-N list can be created from rwuniq's output by passing the output through the UNIX sort(1) and head(1) commands. For example, to see the top-10 dIP-dPort pairs by packet counts, one would issue:

  rwuniq --fields=2,4 --byte --packet --flow --no-title | \
        sort -n -r -t '|' -k 4 | head -10

For additional Top-N and Bottom-N lists (but list less control over the fields that make up the key), see rwstats(1).

rwuniq supports counting distinct source and/or destination IPs. To see the number of distinct sources for each 10 minute bin, run:

  rwuniq --fields=9 --bin-time=600 --sip-distinct | sort

The final sort is required so that data appears in time-order.

When computing distinct counts over a field, the field may not be part of the key; that is, you cannot have "--fields=sip --sip-distinct". In a future release of rwuniq, the --sip-distinct and --dip-distinct switches may be removed in favor of a --distinct=FIELDS switch.

rwuniq's strength is its ability to build an arbitrary key. For a key of a single IP address, see rwaddrcount(1) and rwbag(1); for a key made up of a single CIDR block (/8, /16, /24 only), a single port, or a single protocol, use rwtotal(1) or rwbag(1).

SEE ALSO
rwfilter(1), rwbag(1), rwset(1), rwaddrcount(1), rwstats(1), rwsort(1), rwtotal(1), rwcount(1), addrtype(3), ccfilter(3), pmapfilter(3), silkpython(3)

BUGS
rwuniq should support sorting its output, by subsets of the key or by value fields.

When time-binning is active and the three time fields (starting-time (sTime), ending-time (eTime), and duration) are present in the key, the duration field's value will be modified to be the difference between the ending and starting times.