RWBAGTOOL(1)

NAME
rwbagtool - Perform high-level operations on binary Bag files

SYNOPSIS
  rwbagtool [BAGFILE[,BAGFILE...]]
        { --add | --intersect=SETFILE |
          --complement-intersect=SETFILE |
          --divide | --subtract | --coverset | --invert }
        [--mincounter=VALUE] [--maxcounter=VALUE]
        [--minkey=VALUE] [--maxkey=VALUE] [--output-path=OUTPUTFILE]
        [--note-strip] [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD]

DESCRIPTION
rwbagtool performs various operations on Bags. It can add Bags together, subtract a subset of data from a Bag, perform key intersection of a Bag with an IP set, extract the key list of a Bag as an IP set, or filter Bag records based on their counter value.

BAGFILE is a filename, named pipe, or the keyword "stdin". If no Bag file names are given on the command line, rwbagtool attempts to read a Bag from the standard input. If BAGFILE does not contain a Bag, rwbagtool prints an error to stderr and exits abnormally.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--add
    Add all Bag files given on the command line. If any file cannot be parsed as a Bag, rwbagtool prints an error to stderr and exits abnormally. Key values are unioned and counter values are summed. If a key does not exist, it has a counter of zero.

--intersect=SETFILE
    Add Bag files as described for --add, then mask the keys in the resulting Bag using the set in SETFILE. SETFILE is a filename, named pipe, or the keyword "stdin" containing an IP set. If SETFILE does not contain an IP set, rwbagtool prints an error to stderr and exits abnormally. Only keys which match an entry in SETFILE are output.

--complement-intersect=SETFILE
    As --intersect, but only keys which do not match an entry in SETFILE are output.

--coverset
    Add Bag files as described for --add, then output an IPset which contains the keys contained in the resulting Bag.

--invert
    Add Bag files as described for --add, then generate a new Bag whose keys are the counters in the intermediate Bag and whose counter is the number of times the counter was seen. For example, this turns the Bag {sip:flow} into the Bag {flow:count(sip)}. Any counter in the intermediate Bag that is larger than the maximum possible key will be attributed to the maximum key; to prevent this, specify "--maxcounter=4294967295".

--subtract
    Subtract from the first Bag file all subsequent Bag files. If any inputs cannot be parsed as a Bag, rwbagtool prints an error to stderr and exits abnormally. If any counter subtraction results in a negative number, rwbagtool prints an error to stderr and exits abnormally.

--divide
    Divide the first Bag file by all subsequent Bag files. For example,

      ( ( BAGFILE1 / BAGFILE2 ) / BAGFILE3 ) ...

    All Bags must have the same keys (i.e., all Bags must have the same keys with non-zero counters), or else rwbagtool prints an error and exits abnormally. Likewise, Bags do not support fractional values. To work around this limitation, all counters in the first Bag must be at least as large as the counter for the matching key in subsequent Bags.

    Fractional remainders after division are truncated.

--mincounter=VALUE
    Cause the output to contain only those records whose counter value is VALUE or higher. Default is 1.

--maxcounter=VALUE
    Cause the output to contain only those records whose counter value is VALUE or lower. Default is the maximum counter value.

--minkey=VALUE
    Cause the output to contain only those records whose key value is VALUE or higher. Default is 0.0.0.0. Accepts input as an integer or as an IP address in dotted decimal notation.

--maxkey=VALUE
    Cause the output to contain only those records whose key value is VALUE or higher. Default is 255.255.255.255. Accepts input as an integer or as an IP address in dotted decimal notation.

--output-path=OUTPUTFILE
    Redirect output to OUTPUTFILE. OUTPUTFILE is a filename, named pipe, or the keyword "stdout".

--note-strip
    Do not copy the notes (annotations) from the input files to the output file. Normally notes from the input files are copied to the output.

--note-add=TEXT
    Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME
    Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD
    Set the compression method of the output to COMP_METHOD. Some SiLK tools can use an external library to compress their binary output. The list of available compression methods and the default method are set when SiLK is compiled (the --help and --version switches print the available and default compression methods) and depend on which supported libraries are found. SiLK can support:

    none
        Do not compress the output using an external library

    zlib
        Use the zlib(3) library for compressing the output

    lzo1x
        Use the lzo1x algorithm from the LZO real time compression library for compression

    best
        Use whichever available method gives the "best" compression in general, though not necessarily the "best" for this particular output.

EXAMPLES
To add bag1.bag and bag2.bag together:

  rwbagtool --add bag1.bag bag2.bag > newbag.bag

To only store entries with a counter of 3:

  rwbagtool bag1.bag --mincounter=3 --maxcounter=3 > count3.bag

To build the coverset for that Bag:

  rwbagtool count3.bag --coverset > count3.set

SEE ALSO
rwbag(1), rwbagbuild(1), rwbagcat(1), rwfileinfo(1), rwset(1)