# BaseFetcher (Crawler-commons 0.6-SNAPSHOT API)
Date: 2015-05-26

crawlercommons.fetcher

## Class BaseFetcher

- java.lang.Object
- - crawlercommons.fetcher.BaseFetcher

All Implemented Interfaces:
  Serializable

Direct Known Subclasses:
  BaseHttpFetcher, SimpleFileFetcher

    public abstract class BaseFetcher
    extends Object
    implements Serializable

See Also:
  Serialized Form

### Field Summary

Fields
`protected int` `_defaultMaxContentSize`
`protected Map<String,Integer>` `_maxContentSizes`
`protected Set<String>` `_validMimeTypes`
`static int` `DEFAULT_MAX_CONTENT_SIZE`

### Constructor Summary

Constructors
Constructor and Description
`BaseFetcher()`

### Method Summary

Methods
`abstract void` `abort()` Terminate any async request being processed.
`void` `addValidMimeType(String validMimeType)`
`void` `addValidMimeTypes(Set<String> validMimeTypes)`
`FetchedResult` `get(String url)`
`abstract FetchedResult` `get(String url, Payload payload)` Get the content stored in the resource referenced by <url>
`int` `getDefaultMaxContentSize()`
`int` `getMaxContentSize(String mimeType)`
`protected static String` `getMimeTypeFromContentType(String contentType)`
`Set<String>` `getValidMimeTypes()`
`void` `setDefaultMaxContentSize(int defaultMaxContentSize)`
`void` `setMaxContentSize(String mimeType, int maxContentSize)`
`void` `setValidMimeTypes(Set<String> validMimeTypes)`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### DEFAULT_MAX_CONTENT_SIZE

    public static final int DEFAULT_MAX_CONTENT_SIZE

See Also:
  Constant Field Values

#### _maxContentSizes

    protected Map<String,Integer> _maxContentSizes

#### _defaultMaxContentSize

    protected int _defaultMaxContentSize

#### _validMimeTypes

    protected Set<String> _validMimeTypes

### Constructor Detail

#### BaseFetcher

    public BaseFetcher()

### Method Detail

#### setDefaultMaxContentSize

    public void setDefaultMaxContentSize(int defaultMaxContentSize)

#### getDefaultMaxContentSize

    public int getDefaultMaxContentSize()

#### setMaxContentSize

    public void setMaxContentSize(String mimeType,
                         int maxContentSize)

#### getMaxContentSize

    public int getMaxContentSize(String mimeType)

#### getValidMimeTypes

    public Set<String> getValidMimeTypes()

#### setValidMimeTypes

    public void setValidMimeTypes(Set<String> validMimeTypes)

#### addValidMimeTypes

    public void addValidMimeTypes(Set<String> validMimeTypes)

#### addValidMimeType

    public void addValidMimeType(String validMimeType)

#### get

    public FetchedResult get(String url)
                      throws BaseFetchException

Throws:
  `BaseFetchException`

#### getMimeTypeFromContentType

    protected static String getMimeTypeFromContentType(String contentType)

#### get

    public abstract FetchedResult get(String url,
                    Payload payload)
                               throws BaseFetchException

Get the content stored in the resource referenced by <url>

Parameters:
  `url` -
  `payload` -
Returns:
Throws:
  `BaseFetchException`

#### abort

    public abstract void abort()

Terminate any async request being processed.