# SiteMapParser (Crawler-commons 0.7 API)
Date: 2016-11-21

crawlercommons.sitemaps

## Class SiteMapParser

- java.lang.Object
- - crawlercommons.sitemaps.SiteMapParser

    public class SiteMapParser
    extends Object

### Field Summary

Fields
`static org.slf4j.Logger` `LOG`
`static int` `MAX_BYTES_ALLOWED` Sitemap docs must be limited to 10MB (10,485,760 bytes)
`protected boolean` `strict` True (by default) meaning that invalid URLs should be rejected, as the official docs allow the siteMapURLs to be only under the base url: http://www.sitemaps.org/protocol.html#location

### Constructor Summary

Constructors
Constructor and Description
`SiteMapParser()`
`SiteMapParser(boolean strict)`

### Method Summary

All Methods Instance Methods Concrete Methods
`protected void` `addUrlIntoSitemap(String urlStr, SiteMap siteMap, String lastMod, String changeFreq, String priority, int urlIndex)` Adds the given URL to the given sitemap while showing the relevant logs
`protected String` `getElementAttributeValue(Element elem, String elementName, String attributeName)` Get the element's attribute value.
`protected String` `getElementValue(Element elem, String elementName)` Get the element's textual content.
`boolean` `isStrict()`
`protected void` `parseAtom(SiteMap sitemap, Element elem, Document doc)` Parse the XML document which is assumed to be in Atom format.
`protected void` `parseRSS(SiteMap sitemap, Document doc)` Parse XML document which is assumed to be in RSS format.
`AbstractSiteMap` `parseSiteMap(byte[] content, URL url)` Parse a sitemap, given the content bytes and the URL.
`AbstractSiteMap` `parseSiteMap(String contentType, byte[] content, AbstractSiteMap sitemap)` Returns a processed copy of an unprocessed sitemap object, i.e.
`AbstractSiteMap` `parseSiteMap(String contentType, byte[] content, URL url)` Parse a sitemap, given the MIME type, the content bytes, and the URL.
`AbstractSiteMap` `parseSiteMap(URL onlineSitemapUrl)` Returns a SiteMap or SiteMapIndex given an online sitemap URL
`protected SiteMapIndex` `parseSitemapIndex(URL url, NodeList nodeList)` Parse XML that contains a Sitemap Index.
`protected SiteMap` `parseSyndicationFormat(URL sitemapUrl, Document doc)` Parse the XML document, looking for a feed element to determine if it's an Atom doc rss to determine if it's an RSS doc.
`protected SiteMap` `parseXmlSitemap(URL sitemapUrl, Document doc)` Parse XML that contains a valid Sitemap.
`protected AbstractSiteMap` `processGzip(URL url, byte[] response)` Decompress the gzipped content and process the resulting XML Sitemap.
`protected SiteMap` `processText(String sitemapUrl, byte[] content)` Process a text-based Sitemap.
`protected AbstractSiteMap` `processXml(URL sitemapUrl, byte[] xmlContent)` Parse the given XML content.
`protected AbstractSiteMap` `processXml(URL sitemapUrl, InputSource is)` Parse the given XML content.
`protected boolean` `urlIsValid(String sitemapBaseUrl, String testUrl)` See if testUrl is under sitemapBaseUrl.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### LOG

    public static final org.slf4j.Logger LOG

#### MAX_BYTES_ALLOWED

    public static final int MAX_BYTES_ALLOWED

Sitemap docs must be limited to 10MB (10,485,760 bytes)

See Also:
  Constant Field Values

#### strict

    protected boolean strict

True (by default) meaning that invalid URLs should be rejected, as the official docs allow the siteMapURLs to be only under the base url: http://www.sitemaps.org/protocol.html#location

### Constructor Detail

#### SiteMapParser

    public SiteMapParser()

#### SiteMapParser

    public SiteMapParser(boolean strict)

### Method Detail

#### isStrict

    public boolean isStrict()

Returns:
  whether invalid URLs will be rejected (where invalid means that the url is not under the base url)

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(URL onlineSitemapUrl)
                                 throws UnknownFormatException,
                                        IOException

Returns a SiteMap or SiteMapIndex given an online sitemap URL

Please note that this method is a static method which goes online and fetches the sitemap then parses it

This method is a convenience method for a user who has a sitemap URL and wants a "Keep it simple" way to parse it.

Parameters:
  `onlineSitemapUrl` - URL of the online sitemap
Returns:
  Extracted SiteMap/SiteMapIndex or null if the onlineSitemapUrl is null
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(String contentType,
                                        byte[] content,
                                        AbstractSiteMap sitemap)
                                 throws UnknownFormatException,
                                        IOException

Returns a processed copy of an unprocessed sitemap object, i.e. transfer the value of getLastModified(). Please note that the sitemap input stays unchanged. Note that contentType is assumed to be correct; in general it is more robust to use the method that doesn't take a contentType, but instead detects this using Tika.

Parameters:
  `contentType` - MIME type of content
  `content` - raw bytes of sitemap file
  `sitemap` - an `AbstractSiteMap` implementation
Returns:
  Extracted SiteMap/SiteMapIndex
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(byte[] content,
                                        URL url)
                                 throws UnknownFormatException,
                                        IOException

Parse a sitemap, given the content bytes and the URL.

Parameters:
  `content` - raw bytes of sitemap file
  `url` - URL to sitemap file
Returns:
  Extracted SiteMap/SiteMapIndex
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### parseSiteMap

    public AbstractSiteMap parseSiteMap(String contentType,
                                        byte[] content,
                                        URL url)
                                 throws UnknownFormatException,
                                        IOException

Parse a sitemap, given the MIME type, the content bytes, and the URL. Note that contentType is assumed to be correct; in general it is more robust to use the method that doesn't take a contentType, but instead detects this using Tika.

Parameters:
  `contentType` - MIME type of content
  `content` - raw bytes of sitemap file
  `url` - URL to sitemap file
Returns:
  Extracted SiteMap/SiteMapIndex
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap
  `IOException` - if there is an error reading in the site map `URL`

#### processXml

    protected AbstractSiteMap processXml(URL sitemapUrl,
                                         byte[] xmlContent)
                                  throws UnknownFormatException

Parse the given XML content.

Parameters:
  `sitemapUrl` - URL to sitemap file
  `xmlContent` - the byte[] backing the sitemapUrl
Returns:
  The site map
Throws:
  `UnknownFormatException` - if there is an error parsing the sitemap

#### processText

    protected SiteMap processText(String sitemapUrl,
                                  byte[] content)
                           throws IOException

Process a text-based Sitemap. Text sitemaps only list URLs but no priorities, last mods, etc.

Parameters:
  `sitemapUrl` - a string sitemap URL
  `sitemapUrl` - URL to sitemap file
  `content` - the byte[] backing the sitemapUrl
Returns:
  The site map
Throws:
  `IOException` - if there is an error reading in the site map String

#### processGzip

    protected AbstractSiteMap processGzip(URL url,
                                          byte[] response)
                                   throws IOException,
                                          UnknownFormatException

Decompress the gzipped content and process the resulting XML Sitemap.

Parameters:
  `url` - - URL of the gzipped content
  `response` - - Gzipped content
Returns:
  the site map
Throws:
  `UnknownFormatException` - if there is an error parsing the gzip
  `IOException` - if there is an error reading in the gzip `URL`

#### processXml

    protected AbstractSiteMap processXml(URL sitemapUrl,
                                         InputSource is)
                                  throws UnknownFormatException

Parse the given XML content.

Parameters:
  `sitemapUrl` - a sitemap `URL`
  `is` - an `InputSource` backing the sitemap
Returns:
  the site map
Throws:
  `UnknownFormatException` - if there is an error parsing the `InputSource`

#### parseXmlSitemap

    protected SiteMap parseXmlSitemap(URL sitemapUrl,
                                      Document doc)

Parse XML that contains a valid Sitemap. Example of a Sitemap:

     <?xml version="1.0" encoding="UTF-8"?>
       <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
         <url>
           <loc>http://www.example.com/</loc>
           <lastmod>lastmod>2005-01-01</lastmod>
           <changefreq>monthly</changefreq>
           <priority>0.8</priority>
         </url>
         <url>
           <loc>http://www.example.com/catalog?item=12&amp;desc=vacation_hawaii</loc>
           <changefreq>weekly</changefreq>
         </url>
       </urlset>

Parameters:
  `sitemapUrl` - a sitemap `URL`
  `doc` - a `Document` sitemap snippet
Returns:
  The sitemap

#### parseSitemapIndex

    protected SiteMapIndex parseSitemapIndex(URL url,
                                             NodeList nodeList)

Parse XML that contains a Sitemap Index. Example Sitemap Index:

     <?xml version="1.0" encoding="UTF-8"?>
       <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
         <sitemap>
           <loc>http://www.example.com/sitemap1.xml.gz</loc>
           <lastmod>2004-10-01T18:23:17+00:00</lastmod>
         </sitemap>
         <sitemap>
           <loc>http://www.example.com/sitemap2.xml.gz</loc>
           <lastmod>2005-01-01</lastmod>
         </sitemap>
       </sitemapindex>

Parameters:
  `url` - - URL of Sitemap Index
  `nodeList` - a `NodeList` backing the sitemap
Returns:
  The site map index

#### parseSyndicationFormat

    protected SiteMap parseSyndicationFormat(URL sitemapUrl,
                                             Document doc)
                                      throws UnknownFormatException

Parse the XML document, looking for a feed element to determine if it's an Atom doc rss to determine if it's an RSS doc.

Parameters:
  `sitemapUrl` - the URL location of the Sitemap
  `doc` - - XML document to parse
Returns:
  The sitemap
Throws:
  `UnknownFormatException` - if XML does not appear to be Atom or RSS

#### parseAtom

    protected void parseAtom(SiteMap sitemap,
                             Element elem,
                             Document doc)

Parse the XML document which is assumed to be in Atom format. Atom 1.0 example:

     <?xml version="1.0" encoding="utf-8"?>
       <feed xmlns="http://www.w3.org/2005/Atom">
         <title>Example Feed</title>
         <subtitle>A subtitle.</subtitle>
         <link href="http://example.org/feed/" rel="self"/>
         <link href="http://example.org/"/>
         <modified>2003-12-13T18:30:02Z</modified>
         <author>
           <name>John Doe</name>
           <email>johndoe@example.com</email>
         </author>
         <id>urn:uuid:60a76c80-d399-11d9-b91C-0003939e0af6</id>
         <entry>
           <title>Atom-Powered Robots Run Amok</title>
           <link href="http://example.org/2003/12/13/atom03"/>
           <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
           <updated>2003-12-13T18:30:02Z</updated>
           <summary>Some text.</summary>
         </entry>
         ...
       </feed>

Parameters:
  `sitemap` - a `SiteMap` backing the Atom feed
  `elem` - `Element`'s to populate from the Sitemap
  `doc` - `Document` to populate with the parse output

#### parseRSS

    protected void parseRSS(SiteMap sitemap,
                            Document doc)

Parse XML document which is assumed to be in RSS format. RSS 2.0 example:

     <?xml version="1.0"?>
       <rss version="2.0">
         <channel>
           <title>Lift Off News</title>
           <link>http://liftoff.msfc.nasa.gov/</link>
           <description>Liftoff to Space Exploration.</description>
           <language>en-us</language>
           <pubDate>Tue, 10 Jun 2003 04:00:00 GMT</pubDate>
           <lastBuildDate>Tue, 10 Jun 2003 09:41:01 GMT</lastBuildDate>
           <docs>http://blogs.law.harvard.edu/tech/rss</docs>
           <generator>Weblog Editor 2.0</generator>
           <managingEditor>editor@example.com</managingEditor>
           <webMaster>webmaster@example.com</webMaster>
           <ttl>5</ttl>
           <item>
             <title>Star City</title>
             <link>http://liftoff.msfc.nasa.gov/news/2003/news-starcity.asp</link>
             <description>How do Americans get ready to work with Russians aboard the
             International Space Station? They take a crash course in culture,
             language and protocol at Russia's Star City.
             </description>
             <pubDate>Tue, 03 Jun 2003 09:39:21 GMT</pubDate>
             <guid>http://liftoff.msfc.nasa.gov/2003/06/03.html#item573</guid>
           </item>
           <item>
             <title>Space Exploration</title>
             <link>http://liftoff.msfc.nasa.gov/</link>
             <description>Sky watchers in Europe, Asia, and parts of Alaska and Canada
             will experience a partial eclipse of the Sun on Saturday, May 31.
             </description>
             <pubDate>Fri, 30 May 2003 11:06:42 GMT</pubDate>
             <guid>http://liftoff.msfc.nasa.gov/2003/05/30.html#item572</guid>
           </item>
         </channel>
       </rss>

Parameters:
  `sitemap` - a `SiteMap` object to populate with the RCC content
  `doc` - `Document` to populate with the parse output

#### getElementValue

    protected String getElementValue(Element elem,
                                     String elementName)

Get the element's textual content.

Parameters:
  `elem` -
  `elementName` -
Returns:
  The element value

#### getElementAttributeValue

    protected String getElementAttributeValue(Element elem,
                                              String elementName,
                                              String attributeName)

Get the element's attribute value.

Parameters:
  `elem` -
  `elementName` -
  `attributeName` -
Returns:
  The element attribute value

#### addUrlIntoSitemap

    protected void addUrlIntoSitemap(String urlStr,
                                     SiteMap siteMap,
                                     String lastMod,
                                     String changeFreq,
                                     String priority,
                                     int urlIndex)

Adds the given URL to the given sitemap while showing the relevant logs

Parameters:
  `urlStr` - an URL string to add to the `SiteMap`
  `siteMap` - the sitemap to add URL(s) to
  `lastMod` - last time the `SiteMapURL` was modified
  `changeFreq` - the `SiteMapURL` change frquency
  `priority` - priority of this `SiteMapURL`
  `urlIndex` - index position to which this entry has been added

#### urlIsValid

    protected boolean urlIsValid(String sitemapBaseUrl,
                                 String testUrl)

See if testUrl is under sitemapBaseUrl. Only URLs under sitemapBaseUrl are valid.

Parameters:
  `sitemapBaseUrl` -
  `testUrl` -
Returns:
  true if testUrl is under sitemapBaseUrl, false otherwise