# BasicURLNormalizer (Crawler-commons 1.2 API)
Date: 2021-10-08

crawlercommons.filters.basic

## Class BasicURLNormalizer

- java.lang.Object
- - crawlercommons.filters.URLFilter
  - - crawlercommons.filters.basic.BasicURLNormalizer

    public class BasicURLNormalizer
    extends URLFilter

Code borrowed from Apache Nutch. Converts URLs to a normal form:

- remove dot segments in path: `/./` or `/../`
- remove default ports, e.g. 80 for protocol `http://`
- normalize percent-encoding in URL paths

### Nested Class Summary

Nested Classes
`static class` `BasicURLNormalizer.Builder` A builder class for the `BasicURLNormalizer`.
`static class` `BasicURLNormalizer.IdnNormalization`

### Field Summary

Fields
`static org.slf4j.Logger` `LOG`

### Constructor Summary

Constructors
Constructor and Description
`BasicURLNormalizer()`
`BasicURLNormalizer(BasicURLNormalizer.Builder builder)`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`String` `filter(String urlString)` Returns a modified version of the input URL or null if the URL should be removed
`static String` `formatQueryParameters(List<crawlercommons.filters.basic.BasicURLNormalizer.NameValuePair> parameters)` Formats a list of query parameter name-value pairs into a query parameter string.
`static void` `main(String[] args)`
`static BasicURLNormalizer.Builder` `newBuilder()` Create a new builder object for creating a customized `BasicURLNormalizer` object.
`static List<crawlercommons.filters.basic.BasicURLNormalizer.NameValuePair>` `parseQueryParameters(String s, int queryStartIdx, Set<String> queryElementsToRemove)` Receives the URL query string and parses it into a list of name-value pairs.
`static String` `unescapePath(String path)` Remove % encoding from path segment in URL for characters which should be unescaped according to RFC3986.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### LOG

    public static final org.slf4j.Logger LOG

### Constructor Detail

#### BasicURLNormalizer

    public BasicURLNormalizer()

#### BasicURLNormalizer

    public BasicURLNormalizer(BasicURLNormalizer.Builder builder)

### Method Detail

#### filter

    public String filter(String urlString)

Description copied from class: `URLFilter`
Returns a modified version of the input URL or null if the URL should be removed

Specified by:
  `filter` in class `URLFilter`
Parameters:
  `urlString` - a URL string to check against filter(s)
Returns:
  a filtered URL

#### parseQueryParameters

    public static List<crawlercommons.filters.basic.BasicURLNormalizer.NameValuePair> parseQueryParameters(String s,
                                                                                                           int queryStartIdx,
                                                                                                           Set<String> queryElementsToRemove)

Receives the URL query string and parses it into a list of name-value pairs. Optionally, allows to remove query parameters.

Parameters:
  `s` - a String containing the URL file (as per java.net.URL.getFile(), i.e., the path + query + fragment)
  `queryStartIdx` - the index position of the query part in the string .
  `queryElementsToRemove` - a set of query parameter names to be ignored while parsing the query parameters.

#### formatQueryParameters

    public static String formatQueryParameters(List<crawlercommons.filters.basic.BasicURLNormalizer.NameValuePair> parameters)

Formats a list of query parameter name-value pairs into a query parameter string.

Parameters:
  `parameters` - the query parameter name-value pairs
Returns:
  a URL query string

#### unescapePath

    public static String unescapePath(String path)

Remove % encoding from path segment in URL for characters which should be unescaped according to RFC3986.

#### newBuilder

    public static BasicURLNormalizer.Builder newBuilder()

Create a new builder object for creating a customized `BasicURLNormalizer` object.

Returns:

#### main

    public static void main(String[] args)
                     throws IOException

Throws:
  `IOException`