# SimpleRobotRulesParser (Crawler-commons 1.0 API)
Date: 2019-03-19

crawlercommons.robots

## Class SimpleRobotRulesParser

- java.lang.Object
- - crawlercommons.robots.BaseRobotsParser
  - - crawlercommons.robots.SimpleRobotRulesParser

All Implemented Interfaces:
  Serializable

    public class SimpleRobotRulesParser
    extends BaseRobotsParser

This implementation of `BaseRobotsParser` retrieves a set of `rules` for an agent with the given name from the `robots.txt` file of a given domain.

The class fulfills two tasks. The first one is the parsing of the `robots.txt` file done in `parseContent(String, byte[], String, String)`. During the parsing process the parser searches for the provided agent name(s). If the parser finds a matching name, the set of rules for this name is parsed and returned as the result. Note that if more than one agent name is given to the parser, it parses the rules for the first matching agent name inside the file and skips all following user agent groups. It doesn't matter which of the other given agent names would match additional rules inside the file. Thus, if more than one agent name is given to the parser, the result can be influenced by the order of rule sets inside the `robots.txt` file.

Note that the parser always parses the entire file, even if a matching agent name group has been found, as it needs to collect all of the sitemap directives.

If no rule set matches any of the provided agent names, the rule set for the `'*'` agent is returned. If there is no such rule set inside the `robots.txt` file, a rule set allowing all resource to be crawled is returned.

The crawl delay is parsed and added to the rules. Note that if the crawl delay inside the file exceeds a maximum value, the crawling of all resources is prohibited. The default maximum value is defined with `DEFAULT_MAX_CRAWL_DELAY`=. The default value can be changed using the constructor.

The second task of this class is to generate a set of rules if the fetching of the `robots.txt` file fails. The `failedFetch(int)` method returns a predefined set of rules based on the given error code. If the status code is indicating a client error (status code = 4xx) we can assume that the `robots.txt` file is not there and crawling of all resources is allowed. If the status code equals a different error code (3xx or 5xx) the parser assumes a temporary error and a set of rules prohibiting any crawling is returned.

See Also:
  Serialized Form

### Constructor Summary

Constructors
Constructor and Description
`SimpleRobotRulesParser()`
`SimpleRobotRulesParser(long maxCrawlDelay, int maxWarnings)`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`SimpleRobotRules` `failedFetch(int httpStatusCode)` The fetch of robots.txt failed, so return rules appropriate give the HTTP status code.
`long` `getMaxCrawlDelay()`
`int` `getMaxWarnings()`
`int` `getNumWarnings()`
`static void` `main(String[] args)`
`SimpleRobotRules` `parseContent(String url, byte[] content, String contentType, String robotNames)` Parse the robots.txt file in content, and return rules appropriate for processing paths by userAgent.
`void` `setMaxCrawlDelay(long maxCrawlDelay)`
`void` `setMaxWarnings(int maxWarnings)`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### SimpleRobotRulesParser

    public SimpleRobotRulesParser()

#### SimpleRobotRulesParser

    public SimpleRobotRulesParser(long maxCrawlDelay,
                                  int maxWarnings)

### Method Detail

#### failedFetch

    public SimpleRobotRules failedFetch(int httpStatusCode)

Description copied from class: `BaseRobotsParser`
The fetch of robots.txt failed, so return rules appropriate give the HTTP status code.

Specified by:
  `failedFetch` in class `BaseRobotsParser`
Parameters:
  `httpStatusCode` - a failure status code (NOT 2xx)
Returns:
  robot rules

#### parseContent

    public SimpleRobotRules parseContent(String url,
                                         byte[] content,
                                         String contentType,
                                         String robotNames)

Description copied from class: `BaseRobotsParser`
Parse the robots.txt file in content, and return rules appropriate for processing paths by userAgent. Note that multiple agent names may be provided as comma-separated values; the order of these shouldn't matter, as the file is parsed in order, and each agent name found in the file will be compared to every agent name found in robotNames. Also note that names are lower-cased before comparison, and that any robot name you pass shouldn't contain commas or spaces; if the name has spaces, it will be split into multiple names, each of which will be compared against agent names in the robots.txt file. An agent name is considered a match if it's a prefix match on the provided robot name. For example, if you pass in "Mozilla Crawlerbot-super 1.0", this would match "crawlerbot" as the agent name, because of splitting on spaces, lower-casing, and the prefix match rule.

Specified by:
  `parseContent` in class `BaseRobotsParser`
Parameters:
  `url` - URL that content was fetched from (for reporting purposes)
  `content` - raw bytes from the site's robots.txt file
  `contentType` - HTTP response header (mime-type)
  `robotNames` - name(s) of crawler, to be used when processing file contents (just the name portion, w/o version or other details)
Returns:
  robot rules.

#### getNumWarnings

    public int getNumWarnings()

#### getMaxWarnings

    public int getMaxWarnings()

#### setMaxWarnings

    public void setMaxWarnings(int maxWarnings)

#### getMaxCrawlDelay

    public long getMaxCrawlDelay()

#### setMaxCrawlDelay

    public void setMaxCrawlDelay(long maxCrawlDelay)

#### main

    public static void main(String[] args)
                     throws MalformedURLException,
                            IOException

Throws:
  `MalformedURLException`
  `IOException`