# BaseRobotsParser (Crawler-commons 0.6-SNAPSHOT API)
Date: 2015-05-26

crawlercommons.robots

## Class BaseRobotsParser

- java.lang.Object
- - crawlercommons.robots.BaseRobotsParser

All Implemented Interfaces:
  Serializable

Direct Known Subclasses:
  SimpleRobotRulesParser

    public abstract class BaseRobotsParser
    extends Object
    implements Serializable

See Also:
  Serialized Form

### Constructor Summary

Constructors
Constructor and Description
`BaseRobotsParser()`

### Method Summary

Methods
`abstract BaseRobotRules` `failedFetch(int httpStatusCode)` The fetch of robots.txt failed, so return rules appropriate give the HTTP status code.
`abstract BaseRobotRules` `parseContent(String url, byte[] content, String contentType, String robotName)` Parse the robots.txt file in <content>, and return rules appropriate for processing paths by <userAgent>

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### BaseRobotsParser

    public BaseRobotsParser()

### Method Detail

#### parseContent

    public abstract BaseRobotRules parseContent(String url,
                              byte[] content,
                              String contentType,
                              String robotName)

Parse the robots.txt file in <content>, and return rules appropriate for processing paths by <userAgent>

Parameters:
  `url` - URL that content was fetched from (for reporting purposes)
  `content` - raw bytes from the site's robots.txt file
  `contentType` - HTTP response header (mime-type)
  `robotName` - name of crawler, to be used when processing file contents (just the name portion, w/o version or other details)
Returns:
  robot rules.

#### failedFetch

    public abstract BaseRobotRules failedFetch(int httpStatusCode)

The fetch of robots.txt failed, so return rules appropriate give the HTTP status code.

Parameters:
  `httpStatusCode` - a failure status code (NOT 2xx)
Returns:
  robot rules