# RobotUtils (Crawler-commons 0.7 API)
Date: 2016-11-21

crawlercommons.robots

## Class RobotUtils

- java.lang.Object
- - crawlercommons.robots.RobotUtils

    public class RobotUtils
    extends Object

### Constructor Summary

Constructors
Constructor and Description
`RobotUtils()`

### Method Summary

All Methods Static Methods Concrete Methods
`static BaseHttpFetcher` `createFetcher(BaseHttpFetcher fetcher)`
`static BaseHttpFetcher` `createFetcher(UserAgent userAgent, int maxThreads)`
`static long` `getMaxFetchTime()`
`static BaseRobotRules` `getRobotRules(BaseHttpFetcher fetcher, BaseRobotsParser parser, URL robotsUrl)` Externally visible, static method for use in tools and for testing.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### RobotUtils

    public RobotUtils()

### Method Detail

#### createFetcher

    public static BaseHttpFetcher createFetcher(BaseHttpFetcher fetcher)

#### createFetcher

    public static BaseHttpFetcher createFetcher(UserAgent userAgent,
                                                int maxThreads)

#### getMaxFetchTime

    public static long getMaxFetchTime()

#### getRobotRules

    public static BaseRobotRules getRobotRules(BaseHttpFetcher fetcher,
                                               BaseRobotsParser parser,
                                               URL robotsUrl)

Externally visible, static method for use in tools and for testing. Fetch the indicated robots.txt file, parse it, and generate rules.

Parameters:
  `fetcher` - Fetcher for downloading robots.txt file
  `parser` - a `BaseRobotsParser` to use for obtaining appropriate rules
  `robotsUrl` - URL to robots.txt file
Returns:
  Robot rules