See: Description
Class | Description |
---|---|
BaseRobotRules |
Result from parsing a single robots.txt file - which means we get a set of
rules, and a crawl-delay.
|
BaseRobotsParser | |
RobotUtils | |
SimpleRobotRules |
Result from parsing a single robots.txt file - which means we get a set of
rules, and a crawl-delay.
|
SimpleRobotRulesParser |
Enum | Description |
---|---|
SimpleRobotRules.RobotRulesMode |
Copyright © 2015. All rights reserved.