SENSOR.CONF(1)

NAME
sensor.conf - Configuration file for sensors and probes

DESCRIPTION
To allow for easier configuration, the rwflowpack(8) and flowcap(8) daemons read a text file containing information regarding the sensors and probes the daemon is processing. This manual page describes the syntax of this Sensor Configuration file.

The Sensor Configuration file tries to satisfy the following goals:

1   to tell the collection daemon (rwflowpack or flowcap) how to collect the flow data (e.g., from a network socket, a UNIX domain socket, or a file)

2   to tell rwflowpack how to categorize the data (e.g., as incoming or outgoing) once it has been collected

A probe represents an object that generates a stream of flow-data. The object could be a YAF sensor listening to live traffic, a Cisco router producing NetFlow v5, or software creating flow information from packet capture (tcpdump(1)-like) data.

Probes can be grouped into a logical sensor. A sensor is one or more probes that, for the purpose of analysis, all have the same name.

While probes are defined only in the Sensor Configuration file, sensors should be defined in the Site Configuration file, silk.conf(5). While it is possible to define sensors in the Sensor Configuration file, the analysis tools (e.g., rwfilter(1)) will not be aware of these sensors.

The syntax of the Sensor Configuration file is simple. Blank lines are ignored. Comments, which begin with the "#" character and continue to the end of the line, are also ignored. Every other line should contain a key-value pair, where the key and value are separated by white space. For a key that accepts multiple values, the values must be separated by white space and/or a comma. Repeating a key will overwrite the key's previous value and generate a warning message. Lines are grouped together into blocks. There are two types of blocks: sensor, which defines a new sensor, and sensor-probe, which defines a new probe. The keywords "sensor" and "sensor-probe" introduce a new block which continues until the beginning of the next block. Each key-value pair within the block sets an attribute on the sensor or probe.

  Sensor Block
The following is an example of a sensor block:

  sensor S01
      class all

The sensor block creates a temporary sensor to be used by this invocation of rwflowpack or flowcap. It contains the sensor name and its class. As stated previously, the analysis tools are not aware of sensors that are created in the Sensor Configuration file, and their use is discouraged. A sensor block is most useful during testing.

The following are legal in a sensor block:

"sensor"
    The "sensor" keyword closes the previous sensor or sensor-probe block and defines a new sensor. It is followed by the name of the sensor being defined; this name will be used to generate the names of the data files.

"class"
    The "class" keyword sets the class that the sensor belongs to. In the above example, sensor "S01" belongs to the class "all". Saying a sensor belongs to a class means data collected by that sensor will be considered whenever the name of the class is passed to rwfilter. The name of the class must be one of the classes defined in the silk.conf Site Configuration file. (In addition, the rwflowpack and flowcap daemons must have been aware of the class when they were compiled.)

  Sensor-Probe Block
A sample sensor-probe block showing all possible attributes (but that is illegal because of conflicting attributes) is shown here. (For valid sensor-probe blocks, see the "EXAMPLES" section below.)

  sensor-probe S01
      probe-name netflow
      probe-type netflow
      priority 8
      log-flags none all bad missing

      listen-on-port 9999
      protocol udp
      listen-as-host 192.168.1.1
      listen-on-unix-domain-socket /tmp/sock
      read-from-file /var/tmp/flow-file
      accept-from-host 172.16.22.22

      null-index 0
      isp-ip 10.10.10.10,10.10.10.11
      external-index 1,2,3,4
      internal-index 8,9

      null-ipblock 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder

The first set of values describes the probe:

"sensor-probe"
    The "sensor-probe" keyword closes the previous sensor or sensor-probe block and begins a new sensor-probe block. The value is the name of the sensor for which this probe is a flow collector. The sensor name must be known, either through definition in the silk.conf Site Configuration file or from a sensor block previously defined in this Sensor Configuration file.

"probe-name"
    If this attribute is not present, the name of the probe will be the same as the name of the sensor (i.e., the value given on the sensor-probe line). Note that the probes belonging to a single sensor must each have a unique name. We recommend setting the probe-name to the probe-type. This attribute is used by flowcap to generate file names. When rwflowpack reads these data files, the attribute is used to map from the file name back to the probe. When rwflowpack reads NetFlow records directly, this attribute is only used in error messages.

"probe-type"
    The following probe-types are defined:

    netflow
        A NetFlow probe processes NetFlow V5 protocol data units (PDU) that the daemon reads from a UDP port or from a file. NetFlow may be generated by a router or by software that reads packet capture (pcap(3)) data and generates NetFlow V5 records.

    ipfix
        An IPFIX probe processes Internal Protocol Flow Information eXchange records that the daemon reads from an IPFIX source such as YAF. To support IPFIX probes, SiLK must be built with support for the external library libfixbuf. Both YAF and libfixbuf are available from <http://tools.netsa.cert.org/>.

    When the "probe-type" attribute is not specified, the probe is considered a "netflow" probe.

"priority"
    The priority is an integer value between 1 and 100, inclusive. Values of 1-50 are considered low priority, 51-100 are high. flowcap accumulates flow data from probes into files, and these files are closed after a certain amount of time. Closed high-priority files are always sent to a flowcap client before closed low-priority files. rwflowpack does not use this attribute.

"log-flags"
    This attribute gives specific logging instructions for the probe. Some probe-types will keep track of the sequence numbers in the packets they receive; these probes will write messages to the log file about missing or bad packets. By default, all messages are written by the probes that keep track of this information, but sometimes it is useful to reduce the verbosity of this information or eliminate it entirely. The "log-flags" attribute accepts a list of the following values:

    none
        Log nothing.

    all Log all the conditions listed below; this is the default.

    bad Write messages about bad packets. An example of a bad packet is a NetFlow v5 PDU that reports no flow records.

    missing
        Examine the sequence numbers and write messages about missing packets.

Next, we specify how the probe collects data. For each probe, one and only one the following must be specified:

"listen-on-port"
    This attribute instructs rwflowpack or flowcap to listen on the specified network port in order to collect flow data. When listening to NetFlow from a Cisco router, this is the port given by the Cisco ISO command

      ip flow-export [ip-address] [port]

"listen-on-unix-domain-socket"
    The value contains the path name to a UNIX domain socket. Currently no probes use this collection method.

"read-from-file"
    When this attribute is given, rwflowpack will read PDU records from the specified path on the file system. flowcap does not use this value. The file's length should be an integer multiple of 1464 bytes, where 1464 is the maximum length of the NetFlow v5 PDU. Each 1464 block should contain the 24-byte NetFlow v5 header and space for thirty 48-byte flow records, even if fewer NetFlow records are valid. You can override this attribute by giving the --netflow-file switch to rwflowpack, but the attribute must still be present in the Sensor Configuration file. In this case, we recommend you use "/dev/null" as the file name.

When the daemon is listening on an IP socket ("listen-on-port"), the following attributes may be specified:

"protocol"
    This attribute specifies whether the port on which to listen is an "sctp", "tcp", or "udp" port. The attribute must be specified for IPFIX probe-types, as they have no default protocol. The default (and only permitted value) for all other probe-types is "udp".

"accept-from-host"
    This attribute contains the IP or name of the host that is allowed to connect to the socket. When this attribute is not present, any host may connect. When listening for NetFlow, this attribute would be the IP address of the router as seen from the machine running rwflowpack or flowcap.

"listen-as-host"
    This attribute is used on a multi-homed machine to specify which address to listen on. Its value is a network IP address in dotted-decimal notation. If not present, the program will listen on all the machine's addresses. For listening to NetFlow, the value would be the ip-address given by Cisco ISO command

      ip flow-export [ip-address] [port]

The following attributes are used by rwflowpack to determine in which file a record is stored (i.e., whether the flow is categorized as incoming, outgoing, or dropped [null]). The details are described below in the "PACKING LOGIC" section. Even though flowcap does not use these attributes, any required attributes must be present so that the probe may be verified.

You may categorize any flow, regardless of its probe-type, based on the IP addresses it contains and the IP space of the monitored network. Flows collected by a NetFlow probe may instead be categorized using the SNMP interface information in the flow.

Every probe requires you to specify either the IP space outside the monitored network ("external-ipblock") or, for NetFlow probes, the SNMP interfaces that face outside the network ("external-index"). In addition to specifying the external side, you may specify the values on the internal side ("internal-ipblock" or "internal-index") and the values that represent nonrouted flows ("null-ipblock" or "null-index"). It is an error to specify both the IP space and SNMP interfaces on a probe. The keyword "remainder" may be used for one CIDR block list or SNMP interface list. When "remainder" is specified, that CIDR block list or SNMP interface list is set to all values not seen in the other two lists.

"external-ipblock"
    The value of this attribute is a list of CIDR blocks specifying the IP-space outside of an organization, or the keyword "remainder". Every sensor-probe must have either this attribute or the "external-index" attribute.

"internal-ipblock"
    The value of this attribute is a list of CIDR blocks specifying the internal IP space of an organization, or the keyword "remainder". If not explicitly specified, it is set all CIDR blocks not listed in the "external-ipblock" and "null-ipblock" attributes.

"null-ipblock"
    The value of this attribute is a list of CIDR blocks denoting non-routed packets, or the keyword "remainder".

"external-index"
    The value for this attribute is a list of positive integers that represent the SNMP interface index(es) that connect to external routers or machines, where external means outside the administrative domain being instrumented, or the keyword "remainder". For a border router, the external-indexes would be those connected to the ISP. Either this attribute or the "external-ipblock" attribute is required.

"internal-index"
    The value for this attribute is a list of positive integers that represent the SNMP interface index(es) that connect to the internal network, or the keyword "remainder". If not explicitly specified, it is set all SNMP interfaces not listed in the "external-index" and "null-index" attributes.

"null-index"
    The value for this attribute is the SNMP index of the output interface used by the router to denote a not-routed packet. A packet may be not-routed because the packet is meant for the router itself (e.g., a BGP message) or because it matches an ACL violation. If this attribute is not provided, 0 is used as the default, which is the value used by Cisco routers to denote a null flow.

"isp-ip"
    This attribute is currently unused.

PACKING LOGIC
rwflowpack uses the above values to determine where on disk to store a flow. This packing logic depends on the site that was used when SiLK was configured. This section describes the how a flow's category or type is set for the default SiLK site, twoway, using IP-based splitting. (For SNMP-based splitting, the logic is identical, except the SNMP interfaces are checked.)

ext2ext
    Contains flows where both the source IP and destination IP are listed in the external-ipblock.

innull
    Contains flows where the source IP is listed in the external-ipblock and the destination IP is listed in the null-ipblock.

in,inweb
    Contains flows where the source IP is listed in the external-ipblock and the destination IP is listed in the internal-ipblock. A flow's type is inweb if the protocol is TCP and either the source port or destination port is one of 80, 443, 8080; otherwise it is in.

int2int
    Contains flows where both the source IP and destination IP are listed in the internal-ipblock.

outnull
    Contains flows where the source IP is listed in the internal-ipblock and the destination IP is listed in the null-ipblock.

out,outweb
    Contains flows where the source IP is listed in the internal-ipblock and the destination IP is listed in the external-ipblock.

other
    Contains flows where either

    *   the source IP is not listed in the external-ipblock or internal-ipblock (note that the source could be listed in the null-ipblock)

    *   the destination IP is not listed in the external-ipblock, internal-ipblock, or null-ipblock

EXAMPLES

  Example A
A probe that listens on a UDP port NetFlow from a router and uses the SNMP interfaces to categorize the flows:

  sensor-probe  NF-UDP-SNMP
      probe-name netflow                # optional but recommended
      probe-type netflow                # default value
      priority 8                        # optional
      log-flags all                     # default value
      listen-on-port 9999
      accept-from-host 172.16.22.22     # optional; default: all hosts
      listen-as-host 192.168.1.1        # optional; default: all addrs
      null-index 0                      # default value
      external-index 1,2,3,4
      internal-index 8,9                # line is ignored

  Example B
The same probe but categorizing the flows based on the IP addresses:

  sensor-probe  NF-UDP-CIDR
      probe-name netflow                # optional but recommended
      probe-type netflow                # default value
      priority 8                        # optional
      listen-on-port 9999
      accept-from-host 172.16.22.22     # optional; default: all hosts
      listen-as-host 192.168.1.1        # optional; default: all addrs
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder

  Example C
Instead of listening on a UDP port, suppose those probes processed files containing NetFlow v5 PDUs:

  sensor-probe  NF-FILE-SNMP
      probe-name netflow                # optional but recommended
      probe-type netflow                # default value
      priority 8                        # optional
      log-flags bad                     # ignore missing pkts
      read-from-file /dev/null          # use --netflow-file=<file>
      null-index 0                      # default value
      external-index 1,2,3,4
      internal-index 8,9                # line is ignored

  sensor-probe  NF-FILE-CIDR
      probe-name netflow                # optional but recommended
      probe-type netflow                # default value
      priority 8                        # optional
      log-flags bad                     # ignore missing pkts
      read-from-file /dev/null          # use --netflow-file=<file>
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder

  Example D
A IPFIX probe can listen on a TCP or UDP socket; there is no SNMP information, so CIDR blocks must be used to categorize the flows:

  sensor-probe IPFIX-TCP-CIDR
      probe-name ipfix                  # optional but recommended
      probe-type ipfix
      priority 8                        # optional
      listen-on-port 9999
      protocol tcp
      accept-from-host 172.16.22.22     # optional; default: all hosts
      listen-as-host 192.168.1.1        # optional; default: all addrs
      internal-ipblock 128.2.0.0/16
      external-ipblock remainder

SEE ALSO
silk.conf(5), rwflowpack(8), flowcap(8)