# SimpleRobotRulesParser (Crawler-commons 1.4 API)

Package crawlercommons.robots

## Class SimpleRobotRulesParser

- java.lang.Object
- - crawlercommons.robots.BaseRobotsParser
  - - crawlercommons.robots.SimpleRobotRulesParser

All Implemented Interfaces:
  `Serializable`

    public class SimpleRobotRulesParser
    extends BaseRobotsParser

Robots.txt parser following RFC 9309, supporting the Sitemap and Crawl-delay extensions.

This implementation of `BaseRobotsParser` retrieves a set of `rules` for an agent with the given name from the `robots.txt` file of a given domain. The implementation follows RFC 9309. The following robots.txt extensions are supported:

- the Crawl-delay directive
- the Sitemap directive

The class fulfills two tasks. The first one is the parsing of the `robots.txt` file, done in `parseContent(String, byte[], String, Collection)`. During the parsing process the parser searches for the provided agent name(s). If the parser finds a matching name, the set of rules for this name is parsed and added to the list of rules returned later. If another group of rules is matched in the robots.txt file, also the rules of this group are added to the returned list of rules. See RFC 9309, section 2.2.1 about the merging of groups of rules.

By default and following RFC 9309, section 2.2.1, the user-agent name ("product token") is matched literally but case-insensitive over the full name. See `setExactUserAgentMatching(boolean)` for details of agent name matching and a legacy substring prefix matching mode.

`SimpleRobotRulesParser` allows to pass multiple agent names as a collection. All rule groups matching any of the agent names are followed and merged into one set of rules. The order of the agent names in the collection does not affect the selection of rules.

The parser always parses the entire file to select all matching rule groups, and also to collect all of the sitemap directives.

If no rule set matches any of the provided user-agent names, or if an empty collection of agent names is passed, the rule set for the `'*'` agent is returned. If there is no such rule set inside the `robots.txt` file, a rule set allowing all resource to be crawled is returned.

The crawl-delay is parsed and added to the rules. Note that if the crawl delay inside the file exceeds a maximum value, the crawling of all resources is prohibited. The default maximum value is defined with `DEFAULT_MAX_CRAWL_DELAY`=300000L milliseconds. The default value can be changed using the constructor (`SimpleRobotRulesParser(long, int)` or via `setMaxCrawlDelay(long)`.

The second task of this class is to generate a set of rules if the fetching of the `robots.txt` file fails. The `failedFetch(int)` method returns a predefined set of rules based on the given error code. If the status code is indicating a client error (status code = 4xx) we can assume that the `robots.txt` file is not there and crawling of all resources is allowed. If the status code equals a different error code (3xx or 5xx) the parser assumes a temporary error and a set of rules prohibiting any crawling is returned.

Note that fetching of the robots.txt file is outside the scope of this class. It must be implemented in the calling code, including the following of "at least five consecutive redirects" as required by RFC 9309, section 2.3.1.2.

See Also:
  Serialized Form

### Field Summary

Fields
`static long` `DEFAULT_MAX_CRAWL_DELAY` Default max Crawl-Delay in milliseconds, see `setMaxCrawlDelay(long)`
`static int` `DEFAULT_MAX_WARNINGS` Default max number of warnings logged during parse of any one robots.txt file, see `setMaxWarnings(int)`
`protected static Pattern` `USER_AGENT_PRODUCT_TOKEN_MATCHER` Pattern to match a valid user-agent product tokens as defined in RFC 9309, section 2.2.1

### Constructor Summary

Constructors
`SimpleRobotRulesParser()`
`SimpleRobotRulesParser​(long maxCrawlDelay, int maxWarnings)`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
`SimpleRobotRules` `failedFetch​(int httpStatusCode)` The fetch of robots.txt failed, so return rules appropriate for the given HTTP status code.
`long` `getMaxCrawlDelay()` Get configured max crawl delay.
`int` `getMaxWarnings()` Get max number of logged warnings per robots.txt
`int` `getNumWarnings()` Get the number of warnings due to invalid rules/lines in the latest processed robots.txt file (see `parseContent(String, byte[], String, String)`.
`boolean` `isExactUserAgentMatching()`
`protected static boolean` `isValidUserAgentToObey​(String userAgent)` Validate a user-agent product token as defined in RFC 9309, section 2.2.1
`static void` `main​(String[] args)`
`SimpleRobotRules` `parseContent​(String url, byte[] content, String contentType, String robotNames)` Deprecated.
`SimpleRobotRules` `parseContent​(String url, byte[] content, String contentType, Collection<String> robotNames)` Parse the robots.txt file in content, and return rules appropriate for processing paths by userAgent.

`void`

`setExactUserAgentMatching​(boolean exactMatching)`

Set how the user-agent names in the robots.txt (`User-agent:` lines) are matched with the provided robot names: (with exact matching) follow the Robots Exclusion Protocol RFC 9309 and match user agent literally but case-insensitive over the full string length: Crawlers set their own name, which is called a product token, to find relevant groups.

`void` `setMaxCrawlDelay​(long maxCrawlDelay)` Set the max value in milliseconds accepted for the Crawl-Delay directive.
`void` `setMaxWarnings​(int maxWarnings)` Set the max number of warnings about parse errors logged per robots.txt
`protected String[]` `splitRobotNames​(String robotNames)` Split a string listing user-agent / robot names into tokens.
`protected boolean` `userAgentProductTokenPartialMatch​(String agentName, Collection<String> targetTokens)`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### USER_AGENT_PRODUCT_TOKEN_MATCHER

    protected static final Pattern USER_AGENT_PRODUCT_TOKEN_MATCHER

Pattern to match a valid user-agent product tokens as defined in RFC 9309, section 2.2.1

#### DEFAULT_MAX_WARNINGS

    public static final int DEFAULT_MAX_WARNINGS

Default max number of warnings logged during parse of any one robots.txt file, see `setMaxWarnings(int)`

See Also:
  Constant Field Values

#### DEFAULT_MAX_CRAWL_DELAY

    public static final long DEFAULT_MAX_CRAWL_DELAY

Default max Crawl-Delay in milliseconds, see `setMaxCrawlDelay(long)`

See Also:
  Constant Field Values

### Constructor Detail

#### SimpleRobotRulesParser

    public SimpleRobotRulesParser()

#### SimpleRobotRulesParser

    public SimpleRobotRulesParser​(long maxCrawlDelay,
                                  int maxWarnings)

Parameters:
  `maxCrawlDelay` - see `setMaxCrawlDelay(long)`
  `maxWarnings` - see `setMaxWarnings(int)`

### Method Detail

#### isValidUserAgentToObey

    protected static boolean isValidUserAgentToObey​(String userAgent)

Validate a user-agent product token as defined in RFC 9309, section 2.2.1

Parameters:
  `userAgent` - user-agent token to verify
Returns:
  true if the product token is valid

#### failedFetch

    public SimpleRobotRules failedFetch​(int httpStatusCode)

The fetch of robots.txt failed, so return rules appropriate for the given HTTP status code.

Set rules for response status codes following RFC 9309, section 2.3.1

:

- Success (HTTP 200-299): throws `IllegalStateException` because the response content needs to be parsed
- "Unavailable" Status (HTTP 400-499): allow all
- "Unreachable" Status (HTTP 500-599): disallow all
- every other HTTP status code is treated as "allow all", but further visits on the server are deferred (see `BaseRobotRules.setDeferVisits(boolean)`)

Specified by:
  `failedFetch` in class `BaseRobotsParser`
Parameters:
  `httpStatusCode` - a failure status code (NOT 2xx)
Returns:
  robot rules

#### parseContent

    @Deprecated
    public SimpleRobotRules parseContent​(String url,
                                         byte[] content,
                                         String contentType,
                                         String robotNames)

Deprecated.
Description copied from class: `BaseRobotsParser`
Parse the robots.txt file in content, and return rules appropriate for processing paths by userAgent. Note that multiple agent names may be provided as comma-separated values. How agent names are matched against user-agent lines in the robots.txt depends on the implementing class. Also note that names are lower-cased before comparison, and that any robot name you pass shouldn't contain commas or spaces; if the name has spaces, it will be split into multiple names, each of which will be compared against agent names in the robots.txt file. An agent name is considered a match if it's a prefix match on the provided robot name. For example, if you pass in "Mozilla Crawlerbot-super 1.0", this would match "crawlerbot" as the agent name, because of splitting on spaces, lower-casing, and the prefix match rule.

Specified by:
  `parseContent` in class `BaseRobotsParser`
Parameters:
  `url` - URL that robots.txt content was fetched from. A complete and valid URL (e.g., https://example.com/robots.txt) is expected. Used to resolve relative sitemap URLs and for logging/reporting purposes.
  `content` - raw bytes from the site's robots.txt file
  `contentType` - HTTP response header (mime-type)
  `robotNames` - name(s) of crawler, to be used when processing file contents (just the name portion, w/o version or other details)
Returns:
  robot rules.

#### splitRobotNames

    protected String[] splitRobotNames​(String robotNames)

Split a string listing user-agent / robot names into tokens. Splitting is done at comma and/or whitespace, the tokens are converted to lower-case.

Parameters:
  `robotNames` - robot / user-agent string
Returns:
  array of user-agent / robot tokens

#### parseContent

    public SimpleRobotRules parseContent​(String url,
                                         byte[] content,
                                         String contentType,
                                         Collection<String> robotNames)

Parse the robots.txt file in content, and return rules appropriate for processing paths by userAgent.

Multiple agent names can be passed as a collection. See `setExactUserAgentMatching(boolean)` for details how agent names are matched. If multiple agent names are passed, all matching rule groups are followed and merged into one set of rules. The order of the agent names in the collection does not affect the selection of rules.

If none of the provided agent names is matched, rules addressed to the wildcard user-agent (`*`) are selected.

Specified by:
  `parseContent` in class `BaseRobotsParser`
Parameters:
  `url` - URL that robots.txt content was fetched from. A complete and valid URL (e.g., https://example.com/robots.txt) is expected. Used to resolve relative sitemap URLs and for logging/reporting purposes.
  `content` - raw bytes from the site's robots.txt file
  `contentType` - content type (MIME type) from HTTP response header
  `robotNames` - crawler (user-agent) name(s), used to select rules from the robots.txt file by matching the names against the user-agent lines in the robots.txt file. Robot names should be single token names, without version or other parts. Names must be lower-case, as the user-agent line is also converted to lower-case for matching. If the collection is empty, the rules for the wildcard user-agent (`*`) are selected. The wildcard user-agent name should not be contained in robotNames.
Returns:
  robot rules.

#### userAgentProductTokenPartialMatch

    protected boolean userAgentProductTokenPartialMatch​(String agentName,
                                                        Collection<String> targetTokens)

#### getNumWarnings

    public int getNumWarnings()

Get the number of warnings due to invalid rules/lines in the latest processed robots.txt file (see `parseContent(String, byte[], String, String)`. Note: an incorrect value may be returned if the processing of the robots.txt happened in a different than the current thread.

Returns:
  number of warnings

#### getMaxWarnings

    public int getMaxWarnings()

Get max number of logged warnings per robots.txt

#### setMaxWarnings

    public void setMaxWarnings​(int maxWarnings)

Set the max number of warnings about parse errors logged per robots.txt

#### getMaxCrawlDelay

    public long getMaxCrawlDelay()

Get configured max crawl delay.

Returns:
  the configured max. crawl delay, see `setMaxCrawlDelay(long)`

#### setMaxCrawlDelay

    public void setMaxCrawlDelay​(long maxCrawlDelay)

Set the max value in milliseconds accepted for the Crawl-Delay directive. If the value in the robots.txt is greater than the max. value, all pages are skipped to avoid that overtly long Crawl-Delays block fetch queues and make the crawling slow. Note: the value is in milliseconds as some sites use floating point numbers to define the delay.

#### setExactUserAgentMatching

    public void setExactUserAgentMatching​(boolean exactMatching)

Set how the user-agent names in the robots.txt (`User-agent:` lines) are matched with the provided robot names:

- (with exact matching) follow the Robots Exclusion Protocol RFC 9309 and match user agent literally but case-insensitive over the full string length:

  > Crawlers set their own name, which is called a product token, to find relevant groups. The product token MUST contain only upper and lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens ("-"). [...] Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group.

- (without exact matching) split the user-agent and robot names at whitespace into words and perform a prefix match (one of the user-agent words must be a prefix of one of the robot words, eg. the robot name `WebCrawler/3.0` matches the robots.txt directive

       User-agent: webcrawler

  This prefix matching on words allows that crawler developers lazily use the HTTP User-Agent string also for the robots.txt parser. It does not cover the case when the HTTP User-Agent string is used in the robots.txt.

Parameters:
  `exactMatching` - if true, configure exact user-agent name matching. If false, disable exact matching and do prefix matching on user-agent words.

#### isExactUserAgentMatching

    public boolean isExactUserAgentMatching()

Returns:
  whether exact user-agent matching is configured, see `setExactUserAgentMatching(boolean)`

#### main

    public static void main​(String[] args)
                     throws MalformedURLException,
                            IOException

Throws:
  `MalformedURLException`
  `IOException`