Package | Description |
---|---|
crawlercommons.fetcher.http |
This package concerns the fetching of files over the HTTP protocol:
Extending from
BaseHttpFetcher (which itself extends BaseFetcher ) the
SimpleHttpFetcher provides the Crawler Commons HTTP fetching implementation. |
crawlercommons.robots |
The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.
|
Class and Description |
---|
BaseHttpFetcher |
BaseHttpFetcher.RedirectMode |
UserAgent
User Agent enables us to describe characteristics of any Crawler Commons
agent.
|
Class and Description |
---|
BaseHttpFetcher |
UserAgent
User Agent enables us to describe characteristics of any Crawler Commons
agent.
|
Copyright © 2015. All rights reserved.