# All Classes (Crawler-commons 1.4 API)

# All Classes

All Classes Class Summary Enum Summary Exception Summary
AbstractSiteMap: SiteMap or SiteMapIndex
AbstractSiteMap.SitemapType: Various Sitemap types
BaseRobotRules: Result from parsing a single robots.txt file – a set of allow/disallow rules to check whether a given URL is allowed, and optionally a Crawl-delay and Sitemap URLs.
BaseRobotsParser: Robots.txt parser definition.
BasicURLNormalizer: Code borrowed from Apache Nutch.
BasicURLNormalizer.Builder: A builder class for the `BasicURLNormalizer`.
BasicURLNormalizer.IdnNormalization
CrawlerCommons
DelegatorHandler: Provides a base SAX handler for parsing of XML documents representing sub-classes of AbstractSiteMap.
EffectiveTldFinder: To determine the actual domain name of a host name or URL requires knowledge of the various domain registrars and their assignment policies.
EffectiveTldFinder.EffectiveTLD: EffectiveTLD objects hold one line of the public suffix list: the suffix (`com`, `co.uk`, etc.) for IDN suffixes: both the ASCII and IDN variant (`xn--p1ai` and `рф`) and the properties required to parse host/domain names given in the public suffix list (wildcard suffix, exception, in private domain section)
Extension: Sitemap extensions supported by the parser.
ExtensionHandler: Handler to be called for elements in the namespace of a sitemap extension.
ExtensionMetadata: Container for attributes of a `SiteMapURL` defined by a sitemap extension.
ImageAttributes: Data model for Google extension to the sitemap protocol regarding images indexing, as per http://www.google.com/schemas/sitemap-image/1.1
ImageHandler: Handle SAX events in the Google Image sitemap extension namespace.
LinkAttributes: Data model for Google extension to the sitemap protocol regarding alternate links indexing.
LinksHandler: Handle SAX events in the Google Image sitemap extension namespace.
MimeTypeDetector
MobileAttributes: Google mobile sitemap attributes, see http://www.google.de/schemas/sitemap-mobile/1.0/ and https://www.google.com/schemas/sitemap-mobile/1.0/sitemap-mobile.xsd: Mobile sitemaps just contain an empty "mobile" tag to identify a URL as having mobile content.
MobileHandler: Handle SAX events in the Google Mobile sitemap extension namespace.
Namespace: supported sitemap formats: https://www.sitemaps.org/protocol.html#otherformats
NewsAttributes: Data model for Google's extension to the sitemap protocol regarding news indexing, as per http ://www.google.com/schemas/sitemap-news/0.9.
NewsAttributes.NewsGenre
NewsHandler: Handle SAX events in the Google News sitemap extension namespace.
PaidLevelDomain: Routines to extract the PLD (paid-level domain, as per the IRLbot paper) from a hostname or URL.
SimpleRobotRules: Result from parsing a single robots.txt file – a set of allow/disallow rules to check whether a given URL is allowed, and optionally a Crawl-delay and Sitemap URLs.
SimpleRobotRules.RobotRule: Single rule that maps from a path prefix to an allow flag.
SimpleRobotRules.RobotRulesMode
SimpleRobotRulesParser: Robots.txt parser following RFC 9309, supporting the Sitemap and Crawl-delay extensions.
SiteMap
SiteMapIndex
SiteMapParser
SiteMapTester: Sitemap Tool for recursively fetching all URL's from a sitemap (and all of it's children)
SiteMapURL: The SitemapUrl class represents a URL found in a Sitemap.
SiteMapURL.ChangeFrequency: Allowed change frequencies
SkipLeadingWhiteSpaceInputStream: Wraps a stream and skips over leading whitespace (at beginning of file) in the wrapped stream.
Strings: Util functions for manipulating strings.
SuffixTrie<V>
SuffixTrie.LookupResult<V>: Wrapper for results when a string is checked for suffixes contained in the suffix trie.
SuffixTrie.Node<V>
UnknownFormatException: Exception thrown if the format of a sitemap failed to parse.
URLFilter
VideoAttributes: Data model for Google extension to the sitemap protocol regarding images indexing, as per http://www.google.com/schemas/sitemap-video/1.1
VideoAttributes.VideoPrice
VideoAttributes.VideoPriceResolution
VideoAttributes.VideoPriceType
VideoHandler: Handle SAX events in the Google Video sitemap extension namespace.