RWSET(1)

NAME
rwset - Generate binary IPset files of unique IP addresses

SYNOPSIS
  rwset [--sip-file=FILE] [--dip-file=FILE] [--nhip-file=FILE]
        [--compression-method=COMP_METHOD] [--print-filenames]
        [--copy-input=PATH] [--site-config-file=FILENAME] [FILES...]

DEPRECATED:

  rwset [{ --daddress | --saddress }]
        { --print-ips
          | --integer-ips
          | --set-file=<output-filename>}
        [FILES...]

DESCRIPTION
rwset reads SiLK Flow data from the standard input, a named pipe, or named input files and generates binary IPset file(s) containing the list of the unique source IP addresses, destination IP addresses, and/or next-hop IP addresses.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

One of the following output switches is required:

--sip-file=FILE
    Store the unique source IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard output when FILE is "stdout" and the standard output is not a terminal.

--dip-file=FILE
    Store the unique destination IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard output when FILE is "stdout" and the standard output is not a terminal.

--nhip-file=FILE
    Store the unique next-hop IP addresses in the binary IPset file FILE. rwset will write the IPset file to the standard output when FILE is "stdout" and the standard output is not a terminal.

Only one of the above switches my use "stdout" as the name of the file.

rwset supports these additional switches:

--compression-method=COMP_METHOD
    Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

    none
        Do not compress the output using an external library

    zlib
        Use the zlib(3) library for compressing the output

    lzo1x
        Use the lzo1x algorithm from the LZO real time compression library for compression

    best
        Use whichever available method gives the "best" compression in general, though not necessarily the "best" for this particular output.

--print-filenames
    Prints to the standard error the names of input files as they are opened.

--copy-input=PATH
    Copy all binary input to the specified file or named pipe. PATH can be "stdout" to print flows to the standard output as long as no IPset files are being written there.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the $SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of $SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the $SILK_DATA_ROOTDIR environment variable; the data root directory that is compliled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/, $SILK_PATH/share/, or $SILK_PATH; or in the share/silk/ and share/ directories parallel to the application's directory.

The above output switches allow the user to create multiple IPset files in a single data pass. If none of those switches are given, rwset will fall back to its legacy behavior and generate a single set of IPs, as determined by the following two deprecated switches. If neither switch is given, the source addresses are used:

--saddress
    Generate the set from the source addresses. DEPRECATED: Use the --sip-file switch instead.

--daddress
    Generate the set from destination addresses. DEPRECATED: Use the --dip-file switch instead.

The following legacy switches determine what to do with the IP list generated by the --saddress or --daddress switch:

--print-ips
    Print IP addresses to stdout. DEPRECATED: Use rwsetcat(1) to print the contents of the IPset.

--integer-ips
    Enable the --print-ips option, but print the IPs as integers instead of as dotted decimal. DEPRECATED: Use rwsetcat(1) to print the contents of the IPset.

--set-file=FILE
    Write the IP addresses to a binary IPset file named FILE. DEPRECATED: Use the --sip-file or --dip-file switch instead.

EXAMPLES
rwset is intended to work tightly with rwfilter. For example, consider generating two sets: one of all IP addresses which have 3 packets or less in their sessions, and one with four or more packets.

The first set, for TCP traffic on 03/01/2003 can be generated with:

  rwfilter --start-date=2003/03/01:00 --end-date=2003/03/01:23 \
        --proto=6 --packets=1-3 --pass=stdout | \
        rwset --sip-file=low_packet_tcp.set

The second set with:

  rwfilter --start-date=2003/03/01:00 --end-date=2003/03/01:23
        --proto=6 --packets=4-9999999 --pass=stdout | \
        rwset --sip-file=high_packet_tcp.set

NOTES
IPset files are in a binary format. To create a binary IPset file from a list of dotted decimal IP addresses, use rwsetbuild(1).

Use rwsetcat(1) to see the IP addresses in a binary set file.

rwsetintersect(1) and rwsetunion(1) will allow you to perform set operations on binary IPset files.

The command:

  rwset --sip-file=stdout | rwsetcat

will be faster than rwuniq, but cannot do the thresholding that rwuniq does.

rwset uses rwfilter as an input, so anything that rwfilter passes will become a member of a set (in other words, there's no guarantee that a member of low_packet_tcp.set from above didn't have 4 packets or more in a session, just that there exists at least 1 session for which it had 3 packets or less).

SEE ALSO
rwfilter(1), rwuniq(1), rwsetbuild(1), rwsetcat(1), rwsetintersect(1), rwsetunion(1)