# SiteMapParser (Crawler-commons 0.10 API)
Date: 2018-06-05

crawlercommons.sitemaps

## Class SiteMapParser

- java.lang.Object
- - crawlercommons.sitemaps.SiteMapParser

    public class SiteMapParser
    extends Object

### Field Summary

Fields
`protected Set<String>` `acceptedNamespaces` Set of namespaces (if `strictNamespace`) accepted by the parser.
`static org.slf4j.Logger` `LOG`
`static int` `MAX_BYTES_ALLOWED` Sitemaps (including sitemap index files) "must be no larger than 50MB (52,428,800 bytes)" as specified in the Sitemaps XML format (before Nov.
`protected boolean` `strict` True (by default) meaning that invalid URLs should be rejected, as the official docs allow the siteMapURLs to be only under the base url: http://www.sitemaps.org/protocol.html#location
`protected boolean` `strictNamespace` Indicates whether the parser should work with the namespace from the specifications or any namespace.

### Constructor Summary

Constructors
Constructor and Description
`SiteMapParser()`
`SiteMapParser(boolean strict)`
`SiteMapParser(boolean strict, boolean allowPartial)`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`void` `addAcceptedNamespace(String namespaceUri)` Add namespace URI to set of accepted namespaces.
`void` `addAcceptedNamespace(String[] namespaceUris)` Add namespace URIs to set of accepted namespaces.
`boolean` `isStrict()`
`boolean` `isStrictNamespace()`
`AbstractSiteMap` `parseSiteMap(byte[] content, URL url)` Parse a sitemap, given the content bytes and the URL.
`AbstractSiteMap` `parseSiteMap(String contentType, byte[] content, AbstractSiteMap sitemap)` Returns a processed copy of an unprocessed sitemap object, i.e.
`AbstractSiteMap` `parseSiteMap(String contentType, byte[] content, URL url)` Parse a sitemap, given the MIME type, the content bytes, and the URL.
`AbstractSiteMap` `parseSiteMap(URL onlineSitemapUrl)` Returns a SiteMap or SiteMapIndex given an online sitemap URL Please note that this method is a static method which goes online and fetches the sitemap then parses it This method is a convenience method for a user who has a sitemap URL and wants a "Keep it simple" way to parse it.
`protected AbstractSiteMap` `processGzippedXML(URL url, byte[] response)` Decompress the gzipped content and process the resulting XML Sitemap.
`protected SiteMap` `processText(URL sitemapUrl, byte[] content)` Process a text-based Sitemap.
`protected SiteMap` `processText(URL sitemapUrl, InputStream stream)` Process a text-based Sitemap.
`protected AbstractSiteMap` `processXml(URL sitemapUrl, byte[] xmlContent)` Parse the given XML content.
`protected AbstractSiteMap` `processXml(URL sitemapUrl, InputSource is)` Parse the given XML content.
`void` `setStrictNamespace(boolean s)` Sets the parser to allow any namespace or just the one from the specification, or any accepted namespace (see `addAcceptedNamespace(String)`).
`static boolean` `urlIsValid(String sitemapBaseUrl, String testUrl)` See if testUrl is under sitemapBaseUrl.
`void` `walkSiteMap(AbstractSiteMap sitemap, java.util.function.Consumer<SiteMapURL> action)` Traverse a sitemap, recursively fetching and traversing the content of any enclosed sitemap index, and performing the specified action for each sitemap URL until all URLs have been processed or the action throws an exception.
`void` `walkSiteMap(URL onlineSitemapUrl, java.util.function.Consumer<SiteMapURL> action)` Fetch a sitemap from the specified URL, recursively fetching and traversing the content of any enclosed sitemap index, and performing the specified action for each sitemap URL until all URLs have been processed or the action throws an exception.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### LOG

    public static final org.slf4j.Logger LOG

#### MAX_BYTES_ALLOWED

    public static final int MAX_BYTES_ALLOWED

Sitemaps (including sitemap index files) "must be no larger than 50MB (52,428,800 bytes)" as specified in the Sitemaps XML format (before Nov. 2016 the limit has been 10MB).

See Also:
  Constant Field Values

#### strict

    protected boolean strict

True (by default) meaning that invalid URLs should be rejected, as the official docs allow the siteMapURLs to be only under the base url: http://www.sitemaps.org/protocol.html#location

#### strictNamespace

    protected boolean strictNamespace

Indicates whether the parser should work with the namespace from the specifications or any namespace. Defaults to false.

#### acceptedNamespaces

    protected Set<String> acceptedNamespaces

Set of namespaces (if `strictNamespace`) accepted by the parser. URLs from other namespaces are ignored.

### Constructor Detail

#### SiteMapParser

    public SiteMapParser()

#### SiteMapParser

    public SiteMapParser(boolean strict)

#### SiteMapParser

    public SiteMapParser(boolean strict,
                         boolean allowPartial)

### Method Detail

#### isStrict

    public boolean isStrict()

Returns:
  whether invalid URLs will be rejected (where invalid means that the url is not under the base url)

#### isStrictNamespace

    public boolean isStrictNamespace()

Returns:
  whether the parser allows any namespace or just the one from the specification (or any namespace accepted, `addAcceptedNamespace(String)`)

#### setStrictNamespace

    public void setStrictNamespace(boolean s)

Sets the parser to allow any namespace or just the one from the specification, or any accepted namespace (see `addAcceptedNamespace(String)`). Note enabling strict namespace checking always adds the namespace defined by the current sitemap specificiation (`Namespace.SITEMAP`) to the list of accepted namespaces.

#### addAcceptedNamespace

    public void addAcceptedNamespace(String namespaceUri)

Add namespace URI to set of accepted namespaces.

Parameters:
  `namespaceUri` -

#### addAcceptedNamespace

    public void addAcceptedNamespace(String[] namespaceUris)

Add namespace URIs to set of accepted namespaces.

Parameters:
  `namespaceUris` -

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(URL onlineSitemapUrl)
                                 throws UnknownFormatException,
                                        IOException

Returns a SiteMap or SiteMapIndex given an online sitemap URL Please note that this method is a static method which goes online and fetches the sitemap then parses it This method is a convenience method for a user who has a sitemap URL and wants a "Keep it simple" way to parse it.

Parameters:
  `onlineSitemapUrl` - URL of the online sitemap
Returns:
  Extracted SiteMap/SiteMapIndex or null if the onlineSitemapUrl is null
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(String contentType,
                                        byte[] content,
                                        AbstractSiteMap sitemap)
                                 throws UnknownFormatException,
                                        IOException

Returns a processed copy of an unprocessed sitemap object, i.e. transfer the value of getLastModified(). Please note that the sitemap input stays unchanged. Note that contentType is assumed to be correct; in general it is more robust to use the method that doesn't take a contentType, but instead detects this using Tika.

Parameters:
  `contentType` - MIME type of content
  `content` - raw bytes of sitemap file
  `sitemap` - an `AbstractSiteMap` implementation
Returns:
  Extracted SiteMap/SiteMapIndex
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(byte[] content,
                                        URL url)
                                 throws UnknownFormatException,
                                        IOException

Parse a sitemap, given the content bytes and the URL.

Parameters:
  `content` - raw bytes of sitemap file
  `url` - URL to sitemap file
Returns:
  Extracted SiteMap/SiteMapIndex
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(String contentType,
                                        byte[] content,
                                        URL url)
                                 throws UnknownFormatException,
                                        IOException

Parse a sitemap, given the MIME type, the content bytes, and the URL. Note that contentType is assumed to be correct; in general it is more robust to use the method that doesn't take a contentType, but instead detects this using Tika.

Parameters:
  `contentType` - MIME type of content
  `content` - raw bytes of sitemap file
  `url` - URL to sitemap file
Returns:
  Extracted SiteMap/SiteMapIndex
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### walkSiteMap

    public void walkSiteMap(URL onlineSitemapUrl,
                            java.util.function.Consumer<SiteMapURL> action)
                     throws UnknownFormatException,
                            IOException

Fetch a sitemap from the specified URL, recursively fetching and traversing the content of any enclosed sitemap index, and performing the specified action for each sitemap URL until all URLs have been processed or the action throws an exception.

This method is a convenience method for a user who has a sitemap URL and wants a simple way to traverse it.

Exceptions thrown by the action are relayed to the caller.

Parameters:
  `onlineSitemapUrl` - URL of the online sitemap
  `action` - The action to be performed for each element
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error fetching the content of any `URL`

#### walkSiteMap

    public void walkSiteMap(AbstractSiteMap sitemap,
                            java.util.function.Consumer<SiteMapURL> action)
                     throws UnknownFormatException,
                            IOException

Traverse a sitemap, recursively fetching and traversing the content of any enclosed sitemap index, and performing the specified action for each sitemap URL until all URLs have been processed or the action throws an exception.

This method is a convenience method for a user who has a sitemap and wants a simple way to traverse it.

Exceptions thrown by the action are relayed to the caller.

Parameters:
  `sitemap` - The sitemap to traverse
  `action` - The action to be performed for each element
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error fetching the content of any `URL`

#### processXml

    protected AbstractSiteMap processXml(URL sitemapUrl,
                                         byte[] xmlContent)
                                  throws UnknownFormatException

Parse the given XML content.

Parameters:
  `sitemapUrl` - URL to sitemap file
  `xmlContent` - the byte[] backing the sitemapUrl
Returns:
  The site map
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap

#### processText

    protected SiteMap processText(URL sitemapUrl,
                                  byte[] content)
                           throws IOException

Process a text-based Sitemap. Text sitemaps only list URLs but no priorities, last mods, etc.

Parameters:
  `sitemapUrl` - URL to sitemap file
  `content` - the byte[] backing the sitemapUrl
Returns:
  The site map
Throws:
  `IOException` - if there is an error reading in the site map content

#### processText

    protected SiteMap processText(URL sitemapUrl,
                                  InputStream stream)
                           throws IOException

Process a text-based Sitemap. Text sitemaps only list URLs but no priorities, last mods, etc.

Parameters:
  `sitemapUrl` - URL to sitemap file
  `stream` - content stream
Returns:
  The site map
Throws:
  `IOException` - if there is an error reading in the site map content

#### processGzippedXML

    protected AbstractSiteMap processGzippedXML(URL url,
                                                byte[] response)
                                         throws IOException,
                                                UnknownFormatException

Decompress the gzipped content and process the resulting XML Sitemap.

Parameters:
  `url` - - URL of the gzipped content
  `response` - - Gzipped content
Returns:
  the site map
Throws:
  `UnknownFormatException` - if there is an error parsing the gzip
  `IOException` - if there is an error reading in the gzip `URL`

#### processXml

    protected AbstractSiteMap processXml(URL sitemapUrl,
                                         InputSource is)
                                  throws UnknownFormatException

Parse the given XML content.

Parameters:
  `sitemapUrl` - a sitemap `URL`
  `is` - an `InputSource` backing the sitemap
Returns:
  the site map
Throws:
  `UnknownFormatException` - if there is an error parsing the `InputSource`

#### urlIsValid

    public static boolean urlIsValid(String sitemapBaseUrl,
                                     String testUrl)

See if testUrl is under sitemapBaseUrl. Only URLs under sitemapBaseUrl are valid.

Parameters:
  `sitemapBaseUrl` -
  `testUrl` -
Returns:
  true if testUrl is under sitemapBaseUrl, false otherwise