# crawlercommons.robots (Crawler-commons 0.6-SNAPSHOT API)
Date: 2015-05-26

# Package crawlercommons.robots

The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.

See: Description

Class Summary
Class | Description
--- | ---
BaseRobotRules | Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.
BaseRobotsParser |
RobotUtils |
SimpleRobotRules | Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.
SimpleRobotRulesParser |

Enum Summary
Enum | Description
--- | ---
SimpleRobotRules.RobotRulesMode |

## Package crawlercommons.robots Description

The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.