mirror of
https://github.com/crawler-commons/crawler-commons
synced 2024-05-28 20:36:05 +02:00
33 lines
1.5 KiB
Plaintext
33 lines
1.5 KiB
Plaintext
Crawler-Commons Change Log
|
|
|
|
Release 0.3
|
|
- Upgraded to Tika 1.4 (jnioche)
|
|
- [SiteMap] added utility class for testing sitemaps (jnioche)
|
|
- Issue 16: remove ant scripts and configuration (lewismc)
|
|
- Issue 27: [SiteMap] Unnecessary String concatenations when logging + in SiteMapURL.toString() (jnioche)
|
|
- Issue 26: [SiteMap] Set correct default priority for URL in a sitemap file (jnioche)
|
|
- Issue 25: [Robots] Robots parser should not lowercase sitemap URLs (jnioche)
|
|
- Issue 29: [SiteMap] try urls when <loc> element is missing (jnioche)
|
|
|
|
Release 0.2
|
|
- Move to pure Maven for CC build lifecycle (lewismc)
|
|
- Move Javadoc out of core code (lewismc)
|
|
- Substantiate Javadoc (lewismc)
|
|
- Review default.properties (lewismc)
|
|
- add HTTP status code & reason to FetchedResult (Fuad Efendi via kkrugler)
|
|
- support for multiple user agent names (Tejas Patil via kkrugler)
|
|
- added javadoc generation, publish in /doc/javadoc (kkrugler)
|
|
- switch to using eclipse-formatter.properties (kkrugler)
|
|
- support robots.txt files that have UTF-16LE and UTF-16BE BOMs (kkrugler)
|
|
- support for user agent names that contain spaces (kkrugler)
|
|
- fixed handling of BOM in sitemaps (Vivek Magotra via kkrugler)
|
|
- refactoring of SiteMap objects (Hannes Schwarz via jnioche)
|
|
- added simple support for the file: protocol (kkrugler)
|
|
- cleaned up packaging and added "install" target (kkrugler)
|
|
|
|
Release 0.1
|
|
- parsing robots.txt
|
|
- parsing sitemaps
|
|
- URL analyzer which returns Top Level Domains
|
|
- a simple HttpFetcher
|