# EffectiveTldFinder (Crawler-commons 0.9 API)
Date: 2017-10-27

crawlercommons.domains

## Class EffectiveTldFinder

- java.lang.Object
- - crawlercommons.domains.EffectiveTldFinder

    public class EffectiveTldFinder
    extends Object

To determine the actual domain name of a host name or URL requires knowledge of the various domain registrars and their assignment policies. The best publicly available knowledge base is the public suffix list maintained and available at publicsuffix.org. This class implements the publicsuffix.org ruleset and uses a copy of the public suffix list. data file format. For more information, see

- publicsuffix.org
- Wikipedia article about the public suffix list
- Mozilla's Effective TLD Service: for historic reasons the class name stems from the term "effective top-level domain" (eTLD)

This class just needs "effective_tld_names.dat" in the classpath. If you want to configure it with other data, call `EffectiveTldFinder.getInstance()` `.initialize(InputStream)`. Updates to the public suffix list can be found here:

- https://publicsuffix.org/list/public_suffix_list.dat
- https://publicsuffix.org/list/effective_tld_names.dat (same as public_suffix_list.dat)
- https://raw.githubusercontent.com/publicsuffix/list/master/ public_suffix_list.dat

## ICANN vs. Private Domains

The public suffix list (see section "divisions") is subdivided into "ICANN" and "PRIVATE" domains. To restrict the EffectiveTldFinder to "ICANN" domains only, (re)initialize it by `EffectiveTldFinder.getInstance()` `.initialize(true)` or `.initialize(InputStream, true)`. This will exclude the PRIVATE domain section from the public suffix list.

### Nested Class Summary

Nested Classes
`static class` `EffectiveTldFinder.EffectiveTLD`

### Field Summary

Fields
`static String` `COMMENT`
`static char` `DOT`
`static String` `DOT_REGEX`
`static String` `ETLD_DATA`
`static String` `EXCEPTION`
`static String` `WILD_CARD`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`static String` `getAssignedDomain(String hostname)` This method uses the effective TLD to determine which component of a FQDN is the NIC-assigned domain name.
`static String` `getAssignedDomain(String hostname, boolean strict)` This method uses the effective TLD to determine which component of a FQDN is the NIC-assigned domain name.
`static EffectiveTldFinder.EffectiveTLD` `getEffectiveTLD(String hostname)` Get EffectiveTLD for host name using the singleton instance of EffectiveTldFinder.
`static Map<String,EffectiveTldFinder.EffectiveTLD>` `getEffectiveTLDs()`
`static EffectiveTldFinder` `getInstance()` Get singleton instance of EffectiveTldFinder with default configuration.
`boolean` `initialize(boolean excludePrivateDomains)` (Re)initialize EffectiveTldFinder with built-in public suffix list.
`boolean` `initialize(InputStream effectiveTldDataStream)` (Re)initialize EffectiveTldFinder with custom public suffix list.
`boolean` `initialize(InputStream effectiveTldDataStream, boolean excludePrivateDomains)` (Re)initialize EffectiveTldFinder with custom public suffix list.
`boolean` `isConfigured()`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### ETLD_DATA

    public static final String ETLD_DATA

See Also:
  Constant Field Values

#### COMMENT

    public static final String COMMENT

See Also:
  Constant Field Values

#### DOT_REGEX

    public static final String DOT_REGEX

See Also:
  Constant Field Values

#### EXCEPTION

    public static final String EXCEPTION

See Also:
  Constant Field Values

#### WILD_CARD

    public static final String WILD_CARD

See Also:
  Constant Field Values

#### DOT

    public static final char DOT

See Also:
  Constant Field Values

### Method Detail

#### getInstance

    public static EffectiveTldFinder getInstance()

Get singleton instance of EffectiveTldFinder with default configuration.

Returns:
  singleton instance of EffectiveTldFinder

#### initialize

    public boolean initialize(boolean excludePrivateDomains)

(Re)initialize EffectiveTldFinder with built-in public suffix list.

Parameters:
  `excludePrivateDomains` - whether to exclude the public suffixes listed in the PRIVATE domain section (opposed to "ICANN" domains)
Returns:
  true if (re)initialization was successful

#### initialize

    public boolean initialize(InputStream effectiveTldDataStream)

(Re)initialize EffectiveTldFinder with custom public suffix list.

Parameters:
  `effectiveTldDataStream` - content of public suffix list as input stream
Returns:
  true if (re)initialization was successful

#### initialize

    public boolean initialize(InputStream effectiveTldDataStream,
                              boolean excludePrivateDomains)

(Re)initialize EffectiveTldFinder with custom public suffix list.

Parameters:
  `effectiveTldDataStream` - content of public suffix list as input stream
  `excludePrivateDomains` - whether to exclude the public suffixes listed in the PRIVATE domain section (opposed to "ICANN" domains)
Returns:
  true if (re)initialization was successful

#### getEffectiveTLDs

    public static Map<String,EffectiveTldFinder.EffectiveTLD> getEffectiveTLDs()

#### getEffectiveTLD

    public static EffectiveTldFinder.EffectiveTLD getEffectiveTLD(String hostname)

Get EffectiveTLD for host name using the singleton instance of EffectiveTldFinder.

Parameters:
  `hostname` - the hostname for which to find the `EffectiveTldFinder.EffectiveTLD`
Returns:
  the `EffectiveTldFinder.EffectiveTLD`

#### getAssignedDomain

    public static String getAssignedDomain(String hostname)

This method uses the effective TLD to determine which component of a FQDN is the NIC-assigned domain name.

Parameters:
  `hostname` - a string for which to obtain a NIC-assigned domain name
Returns:
  the NIC-assigned domain name or as fall-back the hostname if no FQDN with valid TLD is found

#### getAssignedDomain

    public static String getAssignedDomain(String hostname,
                                           boolean strict)

This method uses the effective TLD to determine which component of a FQDN is the NIC-assigned domain name.

Parameters:
  `hostname` - a string for which to obtain a NIC-assigned domain name
  `strict` - do not return the hostname as fall-back if a FQDN with valid TLD cannot be determined
Returns:
  the NIC-assigned domain name, null if strict and no FQDN with valid TLD is found

#### isConfigured

    public boolean isConfigured()