# SiteMapParserSAX (Crawler-commons 0.8 API)
Date: 2017-06-07

crawlercommons.sitemaps

## Class SiteMapParserSAX

- java.lang.Object
- - crawlercommons.sitemaps.SiteMapParser
  - - crawlercommons.sitemaps.SiteMapParserSAX

    public class SiteMapParserSAX
    extends SiteMapParser

### Field Summary

Fields
`static org.slf4j.Logger` `LOG`
`static int` `MAX_BYTES_ALLOWED` Sitemaps (including sitemap index files) "must be no larger than 50MB (52,428,800 bytes)" as specified in the Sitemaps XML format (before Nov.
`protected boolean` `strict` True (by default) meaning that invalid URLs should be rejected, as the official docs allow the siteMapURLs to be only under the base url: http://www.sitemaps.org/protocol.html#location

### Constructor Summary

Constructors
Constructor and Description
`SiteMapParserSAX()`
`SiteMapParserSAX(boolean strict)`
`SiteMapParserSAX(boolean strict, boolean allowPartial)`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`protected void` `addUrlIntoSitemap(String urlStr, SiteMap siteMap, String lastMod, String changeFreq, String priority, int urlIndex)` Adds the given URL to the given sitemap while showing the relevant logs
`boolean` `isStrict()`
`AbstractSiteMap` `parseSiteMap(byte[] content, URL url)` Parse a sitemap, given the content bytes and the URL.
`AbstractSiteMap` `parseSiteMap(String contentType, byte[] content, AbstractSiteMap sitemap)` Returns a processed copy of an unprocessed sitemap object, i.e.
`AbstractSiteMap` `parseSiteMap(String contentType, byte[] content, URL url)` Parse a sitemap, given the MIME type, the content bytes, and the URL.
`AbstractSiteMap` `parseSiteMap(URL onlineSitemapUrl)` Returns a SiteMap or SiteMapIndex given an online sitemap URL Please note that this method is a static method which goes online and fetches the sitemap then parses it This method is a convenience method for a user who has a sitemap URL and wants a "Keep it simple" way to parse it.
`protected AbstractSiteMap` `processGzippedXML(URL url, byte[] response)` Decompress the gzipped content and process the resulting XML Sitemap.
`protected SiteMap` `processText(URL sitemapUrl, byte[] content)` Process a text-based Sitemap.
`protected SiteMap` `processText(URL sitemapUrl, InputStream stream)` Process a text-based Sitemap.
`protected AbstractSiteMap` `processXml(URL sitemapUrl, byte[] xmlContent)` Parse the given XML content.
`protected AbstractSiteMap` `processXml(URL sitemapUrl, InputSource is)` Parse the given XML content.
`static boolean` `urlIsValid(String sitemapBaseUrl, String testUrl)` See if testUrl is under sitemapBaseUrl.

### Methods inherited from class crawlercommons.sitemaps.SiteMapParser

`getElementAttributeValue, getElementValue, parseAtom, parseRSS, parseSitemapIndex, parseSyndicationFormat, parseXmlSitemap`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### LOG

    public static final org.slf4j.Logger LOG

#### MAX_BYTES_ALLOWED

    public static final int MAX_BYTES_ALLOWED

Sitemaps (including sitemap index files) "must be no larger than 50MB (52,428,800 bytes)" as specified in the Sitemaps XML format (before Nov. 2016 the limit has been 10MB).

See Also:
  Constant Field Values

#### strict

    protected boolean strict

True (by default) meaning that invalid URLs should be rejected, as the official docs allow the siteMapURLs to be only under the base url: http://www.sitemaps.org/protocol.html#location

### Constructor Detail

#### SiteMapParserSAX

    public SiteMapParserSAX()

#### SiteMapParserSAX

    public SiteMapParserSAX(boolean strict)

#### SiteMapParserSAX

    public SiteMapParserSAX(boolean strict,
                            boolean allowPartial)

### Method Detail

#### isStrict

    public boolean isStrict()

Overrides:
  `isStrict` in class `SiteMapParser`
Returns:
  whether invalid URLs will be rejected (where invalid means that the url is not under the base url)

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(URL onlineSitemapUrl)
                                 throws UnknownFormatException,
                                        IOException

Returns a SiteMap or SiteMapIndex given an online sitemap URL Please note that this method is a static method which goes online and fetches the sitemap then parses it This method is a convenience method for a user who has a sitemap URL and wants a "Keep it simple" way to parse it.

Overrides:
  `parseSiteMap` in class `SiteMapParser`
Parameters:
  `onlineSitemapUrl` - URL of the online sitemap
Returns:
  Extracted SiteMap/SiteMapIndex or null if the onlineSitemapUrl is null
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(String contentType,
                                        byte[] content,
                                        AbstractSiteMap sitemap)
                                 throws UnknownFormatException,
                                        IOException

Returns a processed copy of an unprocessed sitemap object, i.e. transfer the value of getLastModified(). Please note that the sitemap input stays unchanged. Note that contentType is assumed to be correct; in general it is more robust to use the method that doesn't take a contentType, but instead detects this using Tika.

Overrides:
  `parseSiteMap` in class `SiteMapParser`
Parameters:
  `contentType` - MIME type of content
  `content` - raw bytes of sitemap file
  `sitemap` - an `AbstractSiteMap` implementation
Returns:
  Extracted SiteMap/SiteMapIndex
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(byte[] content,
                                        URL url)
                                 throws UnknownFormatException,
                                        IOException

Parse a sitemap, given the content bytes and the URL.

Overrides:
  `parseSiteMap` in class `SiteMapParser`
Parameters:
  `content` - raw bytes of sitemap file
  `url` - URL to sitemap file
Returns:
  Extracted SiteMap/SiteMapIndex
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(String contentType,
                                        byte[] content,
                                        URL url)
                                 throws UnknownFormatException,
                                        IOException

Parse a sitemap, given the MIME type, the content bytes, and the URL. Note that contentType is assumed to be correct; in general it is more robust to use the method that doesn't take a contentType, but instead detects this using Tika.

Overrides:
  `parseSiteMap` in class `SiteMapParser`
Parameters:
  `contentType` - MIME type of content
  `content` - raw bytes of sitemap file
  `url` - URL to sitemap file
Returns:
  Extracted SiteMap/SiteMapIndex
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### processXml

    protected AbstractSiteMap processXml(URL sitemapUrl,
                                         byte[] xmlContent)
                                  throws UnknownFormatException

Parse the given XML content.

Overrides:
  `processXml` in class `SiteMapParser`
Parameters:
  `sitemapUrl` - URL to sitemap file
  `xmlContent` - the byte[] backing the sitemapUrl
Returns:
  The site map
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap

#### processText

    protected SiteMap processText(URL sitemapUrl,
                                  byte[] content)
                           throws IOException

Process a text-based Sitemap. Text sitemaps only list URLs but no priorities, last mods, etc.

Overrides:
  `processText` in class `SiteMapParser`
Parameters:
  `sitemapUrl` - URL to sitemap file
  `content` - the byte[] backing the sitemapUrl
Returns:
  The site map
Throws:
  `IOException` - if there is an error reading in the site map content

#### processText

    protected SiteMap processText(URL sitemapUrl,
                                  InputStream stream)
                           throws IOException

Process a text-based Sitemap. Text sitemaps only list URLs but no priorities, last mods, etc.

Overrides:
  `processText` in class `SiteMapParser`
Parameters:
  `sitemapUrl` - URL to sitemap file
  `stream` - content stream
Returns:
  The site map
Throws:
  `IOException` - if there is an error reading in the site map content

#### processGzippedXML

    protected AbstractSiteMap processGzippedXML(URL url,
                                                byte[] response)
                                         throws IOException,
                                                UnknownFormatException

Decompress the gzipped content and process the resulting XML Sitemap.

Overrides:
  `processGzippedXML` in class `SiteMapParser`
Parameters:
  `url` - - URL of the gzipped content
  `response` - - Gzipped content
Returns:
  the site map
Throws:
  `UnknownFormatException` - if there is an error parsing the gzip
  `IOException` - if there is an error reading in the gzip `URL`

#### processXml

    protected AbstractSiteMap processXml(URL sitemapUrl,
                                         InputSource is)
                                  throws UnknownFormatException

Parse the given XML content.

Overrides:
  `processXml` in class `SiteMapParser`
Parameters:
  `sitemapUrl` - a sitemap `URL`
  `is` - an `InputSource` backing the sitemap
Returns:
  the site map
Throws:
  `UnknownFormatException` - if there is an error parsing the `InputSource`

#### addUrlIntoSitemap

    protected void addUrlIntoSitemap(String urlStr,
                                     SiteMap siteMap,
                                     String lastMod,
                                     String changeFreq,
                                     String priority,
                                     int urlIndex)

Adds the given URL to the given sitemap while showing the relevant logs

Overrides:
  `addUrlIntoSitemap` in class `SiteMapParser`
Parameters:
  `urlStr` - an URL string to add to the `SiteMap`
  `siteMap` - the sitemap to add URL(s) to
  `lastMod` - last time the `SiteMapURL` was modified
  `changeFreq` - the `SiteMapURL` change frquency
  `priority` - priority of this `SiteMapURL`
  `urlIndex` - index position to which this entry has been added

#### urlIsValid

    public static boolean urlIsValid(String sitemapBaseUrl,
                                     String testUrl)

See if testUrl is under sitemapBaseUrl. Only URLs under sitemapBaseUrl are valid.

Parameters:
  `sitemapBaseUrl` -
  `testUrl` -
Returns:
  true if testUrl is under sitemapBaseUrl, false otherwise