# crawlercommons.robots (Crawler-commons 0.7 API)
Date: 2016-11-21

# Package crawlercommons.robots

The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.

See: Description

Class Summary
BaseRobotRules: Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.
BaseRobotsParser
RobotUtils
SimpleRobotRules: Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.
SimpleRobotRules.RobotRule: Single rule that maps from a path prefix to an allow flag.
SimpleRobotRulesParser: This implementation of `BaseRobotsParser` retrieves a set of `rules` for an agent with the given name from the `robots.txt` file of a given domain.

Enum Summary
SimpleRobotRules.RobotRulesMode

## Package crawlercommons.robots Description

The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.