# Uses of Package crawlercommons.robots (Crawler-commons 1.4 API)

# Uses of Package
crawlercommons.robots

Packages that use crawlercommons.robots
crawlercommons.robots: The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.

Classes in crawlercommons.robots used by crawlercommons.robots
BaseRobotRules: Result from parsing a single robots.txt file – a set of allow/disallow rules to check whether a given URL is allowed, and optionally a Crawl-delay and Sitemap URLs.
BaseRobotsParser: Robots.txt parser definition.
SimpleRobotRules: Result from parsing a single robots.txt file – a set of allow/disallow rules to check whether a given URL is allowed, and optionally a Crawl-delay and Sitemap URLs.
SimpleRobotRules.RobotRule: Single rule that maps from a path prefix to an allow flag.
SimpleRobotRules.RobotRulesMode