mnoGoSearch 3.3.9 reference manual: Full-featured search engine software
Prev |  | Next

# Robots

## Name

Robots -- defines whether to use `robots.txt`

`indexer.conf`

## Synopsis

Robots {yes | no}

## Description

Robots defines whether to use `robots.txt` and `<META NAME="robots">` exclusions.

Setting Robots to `no` can be useful when running mnoGoSearch for site validation purposes, as well as when indexing your own Web site.

The default value is `yes`, for polite crawling purposes.

## Scope

Robots can be used multiple times and affects on all following Server and Realm commands until the end of the configuration file, or until the next Robots command.

## Examples

    Robots no

## See also

CrawlDelay, Index, IndexIf, NoIndexIf.

Prev | Home | Next
ReverseAlias | Up | SaveSectionSize