# Uses of Package crawlercommons.robots (Crawler-commons 0.10 API)
Date: 2018-06-05

# Uses of Package
crawlercommons.robots

Packages that use crawlercommons.robots
crawlercommons.robots: The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.

Classes in crawlercommons.robots used by crawlercommons.robots
Class and Description
BaseRobotRules Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.
BaseRobotsParser
SimpleRobotRules Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.
SimpleRobotRules.RobotRule Single rule that maps from a path prefix to an allow flag.
SimpleRobotRules.RobotRulesMode