RWSTATS(1)

NAME
rwstats - Print interval counts or top-N or bottom-N lists

SYNOPSIS
  rwstats [--overall-stats] [--detail-proto-stats=PROTO[,PROTO]]
        [{--sip | --sip=CIDR}] [{--dip | --dip=CIDR}]
        [--sport] [--dport] [--protocol] [--icmp]
        [--flows] [--packets] [--bytes]
        [--count=N] [--threshold=N] [--percentage=N]
        [--top] [--bottom]
        [--integer-ips] [--no-titles] [--no-columns]
        [--column-separator=CHAR] [--no-final-delimiter]
        [{--delimited | --delimited=CHAR}] [--print-filenames]
        [--copy-input=PATH] [--output-path=PATH] [--pager=PAGER_PROG]
        [--site-config-file=FILENAME] [--legacy-help] [FILES...]

DESCRIPTION
rwstats reads SiLK Flow records from files named on the command line or from the standard input, bins those records by the user-specified specified key, computes the volume (record count or sum of packets or bytes) per bin, sorts the bins by volume, and prints the bins with the greatest or least volume. Alternatively, rwstats can provide statistics for each of bytes, packets, and bytes-per-packet giving minima, maxima, quartile, and interval flow-counts across all flows or across a list of protocols specified by the user.

The number of bins printed can be specified as a fixed value (e.g., print 10 bins), as a threshold (print bins whose byte count is less than 400), or as a percentage of the total volume across all bins (print bins who that contain at least 10% of all the packets).

RWSTATS WILL POTENTIALLY RUN OUT OF MEMORY. See the BUGS section for details.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

The following switches will compute and print, for each of bytes, packets, and bytes per packet, the minimum value, the maximum value, quartiles, and a count of the number of flows that fall into each of one of ten intervals statistics. These switches cannot be combined with the switches that produce Top-N or Bottom-N lists.

--overall-stats
    Print intervals and quartiles across all flows that were read by rwstats.

--detail-proto-stats=PROTO[,PROTO...]
    Print intervals and quartiles for each individual protocol listed as an argument. The argument should be a comma separated list of protocols or ranges of protocols: "1-6,17". Specifying this option implies --overall-stats.

To compute a Top-N or Bottom-N list, one or two key fields must be specified. Currently, rwstats only supports pairing --sip with --dip or pairing --sport with --dport.

--sip
--sip=CIDR
    Use the source address as the key. When CIDR is given, only the most significant CIDR bits of the source address are considered; i.e., the least significant 32-CIDR bits of the source address are set to 0. May be used with --dip to key by IP-pairs.

--dip
--dip=CIDR
    Use the destination address as the key. When CIDR is given, only the most significant CIDR bits of the destination address are considered; i.e., the least significant 32-CIDR bits of the destination address are set to 0. May be used with --sip to key by IP-pairs.

--sport
    Use the source port as the key. May be used with --dport to key by port-pairs.

--dport
    Use the destination port as the key. May be used with --sport to key by port-pairs.

--protocol
    Use the protocol as the key.

--icmp
    Use the ICMP type and code as the key. This switch assumes that all the records fed to rwstats are ICMP. It is not possible to key solely on the ICMP type or the ICMP code.

When computing a Top-N or Bottom-N, all flows that have the same key field(s) will be binned together. The "value" of each bin may be based on the number of flows or on the sum of packets or bytes in the flows. If none of the following value field switches is specified, --flows will be used.

--flows
    Use the flow count as the value.

--packets
    Use the packet count as the value.

--bytes
    Use the byte count as the value.

To determine the value of N for a Top-N (or Bottom-N) list, one of the following switches must be specified:

--count=N
    Print the specified number of key-value pairs with the largest (or smallest) values.

--threshold=N
    Print key-value pairs where the value is greater-than (or less-than) this threshold.

--percentage=N
    Print key-value pairs where the value is greater-than (or less-than) this percentage of the total value.

To determine whether to compute the Top-N or the Bottom-N, specify one of the following switches. If neither switch is given, --top is assumed:

--top
    Print the top N keys and their values.

--bottom
    Print the bottom N keys and their values.

Miscellaneous switches:

--integer-ips
    Print IP addresses as integers. If not specified, IP addresses are printed in dotted decimal form.

--no-titles
    Disable section and column titles. By default, titles are printed.

--no-columns
    Disable fixed-width columnar output.

--column-separator=C
    Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.

--no-final-delimiter
    Do not print the column separator after the final column. Normally a delimiter is printed.

--delimited
--delimited=C
    Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

--print-filenames
    Print to the standard error the names of input files as they are opened.

--copy-input=PATH
    Copy all binary input to the specified file or named pipe. PATH can be "stdout" to print flows to the standard output as long as the --output-path switch has been used to redirect rwaddrcount's ASCII output.

--output-path=PATH
    Determine where the output of rwaddrcount (ASCII text) is written. If this option is not given, output is written to the standard output.

--pager=PAGER_PROG
    When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the value of the pager is determined to be the empty string, no paging will be performed and all output will be printed to the terminal.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the SILK_DATA_ROOTDIR environment variable; the data root directory that is compiled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application's directory.

--legacy-help
    Print help, including legacy switches. See the LEGACY SWITCHES section below for these switches.

  LEGACY SWITCHES
Use of the following switches is discouraged; instead, use the replacement switches as indicated.

--sip-topn=N
    Use: "--sip [--top] [--flows] --count=N"

--sip-top-threshold=N
    Use: "--sip [--top] [--flows] --threshold=N"

--sip-top-pct=N
    Use: "--sip [--top] [--flows] --percentage=N"

--sip-btmn=N
    Use: "--sip --bottom [--flows] --count=N"

--sip-btm-threshold=N
    Use: "--sip --bottom [--flows] --threshold=N"

--sip-btm-pct=N
    Use: "--sip --bottom [--flows] --percentage=N"

--dip-topn=N
    Use: "--dip [--top] [--flows] --count=N"

--dip-top-threshold=N
    Use: "--dip [--top] [--flows] --threshold=N"

--dip-top-pct=N
    Use: "--dip [--top] [--flows] --percentage=N"

--dip-btmn=N
    Use: "--dip --bottom [--flows] --count=N"

--dip-btm-threshold=N
    Use: "--dip --bottom [--flows] --threshold=N"

--dip-btm-pct=N
    Use: "--dip --bottom [--flows] --percentage=N"

--pair-topn=N
    Use: "--sip --dip [--top] [--flows] --count=N"

--pair-top-threshold=N
    Use: "--sip --dip [--top] [--flows] --threshold=N"

--pair-top-pct=N
    Use: "--sip --dip [--top] [--flows] --percentage=N"

--pair-btmn=N
    Use: "--sip --dip --bottom [--flows] --count=N"

--pair-btm-threshold=N
    Use: "--sip --dip --bottom [--flows] --threshold=N"

--pair-btm-pct=N
    Use: "--sip --dip --bottom [--flows] --percentage=N"

--sport-topn=N
    Use: "--sport [--top] [--flows] --count=N"

--sport-top-threshold=N
    Use: "--sport [--top] [--flows] --threshold=N"

--sport-top-pct=N
    Use: "--sport [--top] [--flows] --percentage=N"

--sport-btmn=N
    Use: "--sport --bottom [--flows] --count=N"

--sport-btm-threshold=N
    Use: "--sport --bottom [--flows] --threshold=N"

--sport-btm-pct=N
    Use: "--sport --bottom [--flows] --percentage=N"

--dport-topn=N
    Use: "--dport [--top] [--flows] --count=N"

--dport-top-threshold=N
    Use: "--dport [--top] [--flows] --threshold=N"

--dport-top-pct=N
    Use: "--dport [--top] [--flows] --percentage=N"

--dport-btmn=N
    Use: "--dport --bottom [--flows] --count=N"

--dport-btm-threshold=N
    Use: "--dport --bottom [--flows] --threshold=N"

--dport-btm-pct=N
    Use: "--dport --bottom [--flows] --percentage=N"

--portpair-topn=N
    Use: "--sport --dport [--top] [--flows] --count=N"

--portpair-top-threshold=N
    Use: "--sport --dport [--top] [--flows] --threshold=N"

--portpair-top-pct=N
    Use: "--sport --dport [--top] [--flows] --percentage=N"

--portpair-btmn=N
    Use: "--sport --dport --bottom [--flows] --count=N"

--portpair-btm-threshold=N
    Use: "--sport --dport --bottom [--flows] --threshold=N"

--portpair-btm-pct=N
    Use: "--sport --dport --bottom [--flows] --percentage=N"

--proto-topn=N
    Use: "--protocol [--top] [--flows] --count=N"

--proto-top-threshold=N
    Use: "--protocol [--top] [--flows] --threshold=N"

--proto-top-pct=N
    Use: "--protocol [--top] [--flows] --percentage=N"

--proto-btmn=N
    Use: "--protocol --bottom [--flows] --count=N"

--proto-btm-threshold=N
    Use: "--protocol --bottom [--flows] --threshold=N"

--proto-btm-pct=N
    Use: "--protocol --bottom [--flows] --percentage=N"

--cidr-src=N
    Use: "--sip=N"

--cidr-dest=N
    Use: "--dip=N"

EXAMPLES
  $ rwfilter ...| rwstats --sip --count=4
  INPUT SIZE: 549092 records for 12990 unique keys
  SOURCE IP Key: Top 4 flow counts
          sIP|     Records|%_of_total|   cumul_%|
     10.1.1.1|       36604|  6.666278|  6.666278|
     10.1.1.2|       13897|  2.530906|  9.197184|
     10.1.1.3|       12739|  2.320012| 11.517196|
     10.1.1.4|       11807|  2.150277| 13.667473|

  $ rwfilter ...| rwstats --dip --packet --count=7
  INPUT SIZE: 549092 records for 44654 unique keys
  DESTINATION IP Key: Top 7 packet counts
          dIP|     Packets|%_of_total|   cumul_%|
     10.1.1.1|      217574|  3.286325|  3.286325|
     10.1.1.2|      138177|  2.087081|  5.373407|
     10.1.1.3|      121892|  1.841106|  7.214512|
     10.1.1.4|       97073|  1.466230|  8.680742|
     10.1.1.5|       82284|  1.242851|  9.923593|
     10.1.1.6|       80051|  1.209123| 11.132715|
     10.1.1.7|       73602|  1.111714| 12.244430|

  $ rwfilter ...| rwstats --sip --dip --byte --threshold=100000000
  INPUT SIZE: 549092 records for 107136 unique keys
  SIP/DIP PAIR Key: Top 5 byte counts (threshold 100000000)
          sIP|            dIP|       Bytes|%_of_total|   cumul_%|
     10.1.1.1|       10.1.1.2|   307478707|  9.016177|  9.016177|
     10.1.1.3|       10.1.1.4|   172164463|  5.048367| 14.064544|
     10.1.1.5|       10.1.1.6|   142059589|  4.165604| 18.230147|
     10.1.1.7|       10.1.1.8|   119388394|  3.500818| 21.730965|
     10.1.1.9|      10.1.1.10|   108268824|  3.174759| 24.905725|

  $ rwfilter ...| rwstats --sport --sport --percentage=5
  INPUT SIZE: 549092 records for 56799 unique keys
  SOURCE PORT Key: Top 3 flow counts (5% == 27454)
        sPort|     Records|%_of_total|   cumul_%|
           80|       86677| 15.785515| 15.785515|
           53|       64681| 11.779629| 27.565144|
            0|       47760|  8.697996| 36.263140|

  $ rwfilter ...| rwstats --dport --bottom --count=8
  INPUT SIZE: 549092 records for 44772 unique keys
  DESTINATION PORT Key: Bottom 8 flow counts
        dPort|     Records|%_of_total|   cumul_%|
        19417|           1|  0.000182|  0.000182|
        12110|           1|  0.000182|  0.000364|
        34777|           1|  0.000182|  0.000546|
         8999|           1|  0.000182|  0.000728|
        36404|           1|  0.000182|  0.000911|
        16682|           1|  0.000182|  0.001093|
        27420|           1|  0.000182|  0.001275|
        14162|           1|  0.000182|  0.001457|

  $ rwfilter ... | \
        rwstats --sport --dport --packet --top --threshold=500000
  INPUT SIZE: 366309 records for 130307 unique keys
  SPORT/DPORT PAIR Key: No packet counts above threshold of 500000

  $ rwfilter ... | \
        rwstats --sport --dport --packet --top --threshold=50000
  INPUT SIZE: 366309 records for 130307 unique keys
  SPORT/DPORT PAIR Key: Top 3 packet counts (threshold 50000)
        sPort|          dPort|      Packets|%_of_total|   cumul_%|
         6699|           3607|       138177|  2.468531|  2.468531|
           80|           1179|        59774|  1.067862|  3.536393|
           80|           9659|        50319|  0.898949|  4.435342|

  $ rwfilter ...| rwstats --protocol --bottom --count=10
  INPUT SIZE: 545262 records for 3 unique keys
  PROTOCOL Key: Bottom 10 flow counts
     protocol|     Records|%_of_total|   cumul_%|
            1|       46319|  8.494815|  8.494815|
           17|      132634| 24.324820| 32.819635|
            6|      366309| 67.180365|100.000000|

  $ rwfilter ...| rwstats --detail-proto-stats=6,17
  FLOW STATISTICS--ALL PROTOCOLS:  549092 records
  *BYTES min 28; max 88906238
    quartiles LQ 122.06478 Med 420.30930 UQ 876.21920 UQ-LQ 754.15442
     interval_max|count<=max|%_of_input|   cumul_%|
               40|     35107|  6.393646|  6.393646|
               60|     35008|  6.375616| 12.769263|
              100|     49500|  9.014883| 21.784145|
              150|     40014|  7.287303| 29.071449|
              256|     65444| 11.918586| 40.990034|
             1000|    224016| 40.797535| 81.787569|
            10000|     75708| 13.787853| 95.575423|
           100000|     21981|  4.003154| 99.578577|
          1000000|      1901|  0.346208| 99.924785|
       4294967295|       413|  0.075215|100.000000|
  *PACKETS min 1; max 70023
    quartiles LQ 1.76962 Med 3.68119 UQ 7.61567 UQ-LQ 5.84605
     interval_max|count<=max|%_of_input|   cumul_%|
                3|    232716| 42.381969| 42.381969|
                4|     61407| 11.183372| 53.565341|
               10|    195310| 35.569631| 89.134972|
               20|     33310|  6.066379| 95.201351|
               50|     17686|  3.220954| 98.422304|
              100|      4854|  0.884005| 99.306309|
              500|      2760|  0.502648| 99.808957|
             1000|       373|  0.067930| 99.876888|
            10000|       637|  0.116010| 99.992897|
       4294967295|        39|  0.007103|100.000000|
  *BYTES/PACKET min 28; max 1500
    quartiles LQ 57.98319 Med 90.71150 UQ 164.77250 UQ-LQ 106.78932
     interval_max|count<=max|%_of_input|   cumul_%|
               40|     42568|  7.752435|  7.752435|
               44|     15173|  2.763289| 10.515724|
               60|     91003| 16.573361| 27.089085|
              100|    163850| 29.840173| 56.929258|
              200|    153190| 27.898786| 84.828043|
              400|     39761|  7.241227| 92.069271|
              600|     12810|  2.332942| 94.402213|
              800|      7954|  1.448573| 95.850786|
             1500|     22783|  4.149214|100.000000|
       4294967295|         0|  0.000000|100.000000|

  FLOW STATISTICS--PROTOCOL 6:  366309/549092 records
  *BYTES min 40; max 88906238
    quartiles LQ 310.47331 Med 656.53661 UQ 1089.75344 UQ-LQ 779.28013
     interval_max|count<=max|%_of_proto|   cumul_%|
               40|     29774|  8.128110|  8.128110|
               60|     11453|  3.126595| 11.254706|
              100|      6915|  1.887751| 13.142456|
              150|     16369|  4.468632| 17.611088|
              256|     12651|  3.453642| 21.064730|
             1000|    196881| 53.747246| 74.811976|
            10000|     68989| 18.833553| 93.645529|
           100000|     21099|  5.759891| 99.405420|
          1000000|      1784|  0.487021| 99.892441|
       4294967295|       394|  0.107559|100.000000|
  *PACKETS min 1; max 70023
    quartiles LQ 3.39682 Med 5.85903 UQ 8.80427 UQ-LQ 5.40745
     interval_max|count<=max|%_of_proto|   cumul_%|
                3|     69358| 18.934288| 18.934288|
                4|     55993| 15.285729| 34.220016|
               10|    186559| 50.929407| 85.149423|
               20|     30947|  8.448332| 93.597755|
               50|     16186|  4.418674| 98.016429|
              100|      4204|  1.147665| 99.164094|
              500|      2178|  0.594580| 99.758674|
             1000|       315|  0.085993| 99.844667|
            10000|       537|  0.146598| 99.991264|
       4294967295|        32|  0.008736|100.000000|
  *BYTES/PACKET min 40; max 1500
    quartiles LQ 60.19817 Med 96.78616 UQ 175.08044 UQ-LQ 114.88228
     interval_max|count<=max|%_of_proto|   cumul_%|
               40|     36559|  9.980372|  9.980372|
               44|     14929|  4.075521| 14.055893|
               60|     39593| 10.808634| 24.864527|
              100|    100117| 27.331297| 52.195824|
              200|    111258| 30.372718| 82.568542|
              400|     26020|  7.103293| 89.671834|
              600|      8600|  2.347745| 92.019579|
              800|      7726|  2.109148| 94.128727|
             1500|     21507|  5.871273|100.000000|
       4294967295|         0|  0.000000|100.000000|

  FLOW STATISTICS--PROTOCOL 17:  132634/549092 records
  *BYTES min 32; max 2115559
    quartiles LQ 66.53665 Med 150.61551 UQ 242.44095 UQ-LQ 175.90430
     interval_max|count<=max|%_of_proto|   cumul_%|
               20|         0|  0.000000|  0.000000|
               40|      5195|  3.916794|  3.916794|
               80|     42150| 31.779182| 35.695975|
              130|     11528|  8.691587| 44.387563|
              256|     45497| 34.302667| 78.690230|
             1000|     23401| 17.643289| 96.333519|
            10000|      4447|  3.352836| 99.686355|
           100000|       389|  0.293288| 99.979643|
          1000000|        23|  0.017341| 99.996984|
       4294967295|         4|  0.003016|100.000000|
  *PACKETS min 1; max 8839
    quartiles LQ 0.84383 Med 1.68768 UQ 2.53149 UQ-LQ 1.68766
     interval_max|count<=max|%_of_proto|   cumul_%|
                3|    117884| 88.879171| 88.879171|
                4|      4452|  3.356605| 92.235777|
               10|      6678|  5.034908| 97.270685|
               20|      1766|  1.331484| 98.602168|
               50|      1055|  0.795422| 99.397590|
              100|       368|  0.277455| 99.675046|
              500|       353|  0.266146| 99.941192|
             1000|        33|  0.024880| 99.966072|
            10000|        45|  0.033928|100.000000|
       4294967295|         0|  0.000000|100.000000|
  *BYTES/PACKET min 32; max 1415
    quartiles LQ 63.23827 Med 91.27180 UQ 158.10219 UQ-LQ 94.86392
     interval_max|count<=max|%_of_proto|   cumul_%|
               20|         0|  0.000000|  0.000000|
               24|         0|  0.000000|  0.000000|
               40|      5671|  4.275676|  4.275676|
              100|     70970| 53.508150| 57.783826|
              200|     39298| 29.628904| 87.412730|
              400|     12175|  9.179396| 96.592126|
              600|      4130|  3.113832| 99.705958|
              800|       160|  0.120633| 99.826590|
             1500|       230|  0.173410|100.000000|
       4294967295|         0|  0.000000|100.000000|

ENVIRONMENT

SILK_PAGER
    When set to a non-empty string, rwstats automatically invokes this program to display its output a screen at a time. If set to an empty string, rwstats does not automatically page its output.

PAGER
    When set and SILK_PAGER is not set, rwstats automatically invokes this program to display its output a screen at a time.

SILK_CONFIG_FILE
    This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR
    When the --site-config-file switch is not provided and the SILK_CONFIG_FILE environment variable is not set, rwstats looks for the site configuration file in $SILK_DATA_ROOTDIR/silk.conf.

SILK_PATH
    This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwstats checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.

NOTES
When used in an IPv6 environment, rwstats will process every record as long as the IP address is not part of the key. When aggregating by an IP address or an IP-pair, rwstats will attempt to convert any IPv6 addresses to IPv4. Records that can be converted will be processed, all other records will be silently ignored.

The output of rwstats is similar to that of rwaddrcount(1), rwtotal(1), and rwuniq(1).

To compute Top-N lists for other key combinations or to see values for Records, Packets, and Bytes in a single view, consider using another SiLK tool and passing the output through sort and head. For example, to see the Top-10 lists for sip,sport combinations, counting by Bytes:

  $ rwfilter ...| rwuniq --fields=sip,sport --all --no-titles \
        | sort -r -t '|' -k 3 | head -10

rwstats uses an hash table internally when computing Top-N and Bottom-N lists. rwstats may run of memory when processing IP addresses, especially IP-pairs. If rwstats's hash table does run out of memory, rwstats will stop processing input, print a warning to the standard error, output the entries it has computed to that point, and exit with code 16.

SEE ALSO
rwfilter(1) rwaddrcount(1) rwtotal(1) rwuniq(1)