RWMATCH(1)

NAME
rwmatch - Matches SiLK records from two streams into a common stream

SYNOPSIS
  rwmatch [--site-config-file=FILENAME]
        [--relate=FIELD_PAIR] [--time-delta=DELTA]
        QUERY_FILE RESPONSE_FILE OUTPUT_FILE

DESCRIPTION
rwmatch provides a facility for relating SiLK Flow records contained in two files, called the QUERY_FILE and the RESPONSE_FILE. This relationship results in an OUTPUT_FILE which contains records from the QUERY_FILE and their matching responses from the RESPONSE_FILE.

Matching criteria are defined by using the --relate switch, which specifies that two fields (one in the QUERY_FILE, one in the RESPONSE_FILE) are related. A query and a response record match if all the field pairs specified by --relate are equal, and they occur close to each other in time. The time period is defined as being in the time between the start time and end time of the query flow: if the start time of the response flow >= start or end time of the query flow, it is considered to match. This value can be extended using the --time-delta switch.

Matched records are written to the OUTPUT FILE. Each set of matching records is assigned a match ID which is written into the next hop IP field. The last 24 bits of the next hop IP are used to store the match ID, while the first 8 bits are used to indicate whether the record in question came from the query file (first octet set to 0), or the response file (first octet set to 255). Therefore, the first match identified will have 0.0.0.0 in the query record's next hop IP field, the query record for the second match will have 0.0.0.1 and so on. The corresponding response records will have 255.0.0.0, 255.0.0.1 and so on. Apart from the change to the next hop IP, query and response records are not modified.

The rwmatch application only prints out records which have a query and a response; if a query record does not have a corresponding response, then the query record is discarded. Similarly, if responses do not have a corresponding query, they are discarded.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--relate=FIELD_PAIR
    Specifies a pair of fields separated by a comma, one in the query record, one in the response record. These data in these fields must be identical for two records to match. At least one FIELD_PAIR must be provided; up to 8 FIELD_PAIRs may be provided. The available fields are:

    1   source IP

    2   destination IP

    3   source port

    4   destination port

    5   protocol

    6   packets

    7   bytes

    8   flags

    9   start time

    10  duration

    11  end time

    12  sensor

    FIELD_PAIR is always given as a pair, such as:

    *   --relate="2,1" (query:destination IP :: response:source IP)

    *   --relate="5,5" (query:protocol :: response:protocol)

    FIELD_PAIR is unidirectional; specifying

      --relate=1,2

    will match records where the query record's source IP matches the response record's destination IP, regardless of the values for the other IP's in the set. To match symmetric flow records between hosts, match both the source and destination IP addresses by specifying:

      --relate=2,1 --relate=1,2

    This command will match source and destination IP addresses. When using a port-based protocol (e.g., TCP or UDP), refine the match further by specifying the ports:

      --relate=2,1 --relate=1,2 --relate=3,4 --relate=4,3

    Matching becomes more specific as more fields are added. Since rwmatch discards unmatched records, a highly specific match (such as the last one specified above) will generate more matches (resulting in higher match IDs), but may result in fewer total flows due to certain records being unmatched.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the $SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of $SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified in the $SILK_DATA_ROOTDIR environment variable; the data root directory that is compliled into SiLK (use the --version switch to view this value); the directories $SILK_PATH/share/silk/, $SILK_PATH/share/, or $SILK_PATH; or in the share/silk/ and share/ directories parallel to the application's directory.

--time-delta=DELTA
    Specifies the number of seconds by which a response can follow a query. Responses match queries if

      query:sTime <= response:sTime
      response:sTime <= query:eTime + DELTA

    Use the --time-delta switch to compensate for timing errors between multiple sensors.

LIMITATIONS
rwmatch works by walking through each record in QUERY_FILE; for every query record, it walks through RESPONSE_FILE to look for records which match the query record according to user specification. If it finds matching records, it outputs the query and all corresponding responses. All records are examined once, and if not output, are discarded.

rwmatch requires that the SiLK Flow records in the QUERY_FILE and RESPONSE_FILE be sorted. Since the matching requires the records to occur within --time-delta seconds of each other, the input file should always be sorted with respect to time.

When sorting records for rwmatch, always sort by the fields that file will match on, and then the start time for the record.

For example, to match on the --relate=1,2 case shown above, sort as follows:

  rwsort --field=1,9 source_ips.rwf > source_ips.s.rwf
  rwsort --field=2,9 dest_ips.rwf > dest_ips.s.rwf

And then call rwmatch with

  rwmatch --relate=1,2 source_ips.s.rwf dest_ips.s.rwf matched.rwf

But note that

  rwmatch --relate=2,1 source_ips.s.rwf dest_ips.s.rwf stdout

Will not produce meaningful results, since source_ips.s.rwf was sorted on field 1, and dest_ips.s.rwf was sorted on field 2.

rwmatch assumes that there is parity between queries and responses; that is, for every response record, there exists one query record. Because long-lived sessions are often broken into multiple flows, rwmatch may discard records that are part of a long-lived session. When working with longer-lived sessions, it is advisable to first group records using rwgroup.

EXAMPLES
rwmatch is a generalized matching tool; the most basic function provided by rwmatch is the ability to match both sides of a TCP connection. For example, given incoming and outgoing web traffic in two files web_in.rwf and web_out.rwf

    rwsort --field=1,3,9 web_in.rwf > web_in.s.rwf
    rwsort --field=2,4,9 web_out.rwf > web_out.s.rwf

    rwmatch --relate=1,2 --relate=2,1 --relate=3,4 --relate=4,3
            web_in.s.rwf web_out.s.rwf web.sessions.rwf

Will generate a file consisting of matched sessions for every complete web session in web_in.rwf and web_out.rwf

Because rwmatch can match fields arbitrarily, you can also match records across different protocols. For example, to trace all router responses to a scan attempt, we start by identifying a specific horizontal scan.

  rwfilter --proto=6 --packets=1 --dport=x indata --pass=scanners.rwf
  rwsort --field=1,9 scanners.rwf > scanners.s.rwf

We then identify all ICMP "destination unreachable" messages:

  rwfilter --proto=1 --dport=768-1023 outdata --pass=icmp.rwf
  rwsort --field=2,9 icmp.rwf > icmp.s.rwf

We then match exclusively on IP address

  rwmatch --relate=2,1 icmp.s.rwf scanners.s.rwf result.rwf

The resulting file will consist of single packet flows (from scanners.s.rwf) with an ICMP response (from icmp.s.rwf). Similar queries can be used to identify other multiple-protocol phenomena, such as the results of a traceroute.

SEE ALSO
rwfilter(1), rwsort(1), rwgroup(1)