mirror/ crawler-commons
1
0
Fork 0
mirror of https://github.com/crawler-commons/crawler-commons synced 2024-05-24 19:36:06 +02:00
Code Issues
535 Commits 8 Branches 15 Tags 5.8 MiB
Commit Graph

535 Commits

Author SHA1 Message Date
Aecio Santos aaa3113e55 Update CHANGES.txt 2018-05-14 11:51:43 -04:00
Aecio Santos fd1e7fcffe SimpleRobotRulesParser: Expose MAX_CRAWL_DELAY #194 
- Makes MAX_CRAWL_DELAY configurable through class constructor
 2018-05-13 20:10:55 -04:00
Aecio Santos 7bef14d386 Make RobotRules accessible #134 
- Makes SimpleRobotRulesParser._rules property protected
  and adds getters for SimpleRobotRulesParser._rules and
  RobotRules's properties
- Changes SimpleRobotRulesParser return type from BaseRobotRules
  to SimpleRobotRules to allow access to concrete class without
  nasty type casts while still obeying super class contract
 2018-05-13 20:07:54 -04:00
Julien Nioche e25309d26c
Add JAX-B dependencies to POM (#207) 
* Add JAX-B dependencies to POM, fixes #196

* mentioned in CHANGES.txt
 2018-05-03 11:04:03 +01:00
Sebastian Nagel 8ca46ffe2c Merge pull request #190 from thunken:features/for-each 
Add SiteMapParser#walkSiteMap(URL,Consumer)
 2018-04-25 14:08:27 +02:00
Sebastian Nagel 7d3eccfa63 Add changelog entry and fix unit test 2018-04-25 14:06:33 +02:00
Luc Boruta 355684ff20 Add methods to parse and iterate sitemap 
SiteMapParser#walkSiteMap(URL,Consumer)
 2018-04-25 14:04:58 +02:00
Sebastian Nagel fa76a594b6
Merge pull request #205 from sebastian-nagel/cc-201-202-203-204 
Improve sitemap parsing
 2018-04-25 09:51:49 +02:00
Sebastian Nagel 0ef7cf87fa Improve sitemap parsing 
- ignore query part of URL to determine sitemap location prefix
  for URL validation, fixes #202
- resolve relative links in RSS feeds, fixes #203
- allow non-continuous content (containing XML entities or CDATA)
  when parsing links in RSS feeds, fixes #204
- extract links from <guid> elements in RSS feeds, fixes #201
 2018-04-25 09:36:27 +02:00
Sebastian Nagel a9277acde2
Merge pull request #200 from sebastian-nagel/cc-198-fix-regressions 
Improve MIME detection for sitemaps
 2018-04-25 09:19:27 +02:00
Sebastian Nagel a6b3178fc7 Simplify MIME detection: 
- handle BOM and leading white space together
- remove parameter to detect patterns at a specific offset
 2018-04-24 14:32:28 +02:00
Sebastian Nagel 6714ea57d6 Fix error message format string 2018-04-18 14:53:21 +02:00
Sebastian Nagel 72aa177738 RDF-based RSS feeds: map MIME type, detect from content 2018-04-18 14:44:10 +02:00
Sebastian Nagel 4780678af0 Improve logging of content type detection for gzip-compressed sitemaps 2018-04-18 14:43:52 +02:00
Sebastian Nagel 8a34e25c41
Merge pull request #206 from sebastian-nagel/cc-145-bad-url-warnings 
Sitemaps limit on "bad url" log messages, fixes #145
 2018-04-16 13:37:39 +02:00
Sebastian Nagel 907be2343f Format fix: add braces, complete CHANGES.txt 2018-04-16 13:36:06 +02:00
Sebastian Nagel af084f7ee4 Sitemaps limit on "bad url" log messages, fixes #145 
- degrade log level to debug for lines which are not valid
- only log first 1024 characters of line
 2018-04-13 11:16:47 +02:00
Sebastian Nagel 34c19d811a Fix MIME detection for sitemaps: 
- avoid NPE if no MIME type has been detected
- allow optional leading white space before MIME patterns
  (after optional BOM)
 2018-04-10 19:17:16 +02:00
Ken Krugler 12155888bc
Add reference to issue #199 2018-04-02 12:59:17 -07:00
Ken Krugler 14153c4eae
Merge pull request #198 from crawler-commons/kkrugler_no-tika 
Remove Tika dependency
 2018-04-02 12:55:40 -07:00
Ken Krugler 165888ba7e Use locale with String.format() 2018-03-28 14:40:55 -07:00
Ken Krugler 2cff74427d Remove Tika dependency 2018-03-27 15:40:50 -07:00
Julien Nioche 5132651a6e Upgraded Tika to 1.17, fixes #189 2018-03-20 17:20:07 +00:00
Ken Krugler 4b13456bc9
Merge pull request #188 from sebastian-nagel/cc-179-effective-tldfinder-parse-idns 
EffectiveTldFinder to parse Internationalized Domain Names (IDN)
 2017-12-08 09:16:37 -08:00
Sebastian Nagel 49bf37c6d9 Update CHANGES.txt 2017-12-08 09:42:52 +01:00
Sebastian Nagel 6de3f26da8 - add missing license header 
- add debug log messages
- add unit test for invalid hostname (www..example..com)
 2017-12-07 21:23:26 +01:00
Sebastian Nagel 95c6e3886a EffectiveTldFinder to parse Internationalized Domain Names (IDN), 
fixes #179
- allow eTLDs to appear as IDN variants
  * one or more dot-separated parts as IDN,
  * optionally others as punycoded IDNA
- speed-up lookup of public suffixes / eTLDs using a suffix trie
- update public suffix list to recent version
- add unit tests to test for wildcard eTLDs
 2017-12-06 21:47:01 +01:00
Sebastian Nagel b63b9b62ce
Merge pull request #187 from sebastian-nagel/effective-tldfinder-main 
Add main() to EffectiveTldFinder
 2017-12-06 16:52:41 +01:00
Sebastian Nagel 5c1ad13cea Add main() to EffectiveTldFinder 2017-12-01 10:54:03 +01:00
Sebastian Nagel bde59ff33e
Merge pull request #186 from sebastian-nagel/cc-185-effective-tld-finder-private-domains 
Allow to include/exclude private domains / effective TLDs without re-initialization, fixes #185
 2017-12-01 10:22:56 +01:00
Sebastian Nagel f598af8d28 - address TODO of #184: do not use of private suffixes 
for PaidLevelDomain detection
- adapt unit tests: 'de.com' is a private suffix,
  consequently 'xxx.de.com' isn't a PaidLevelDomain
- rebase on master, apply code format
 2017-11-06 14:03:35 +01:00
Sebastian Nagel cda1bdee82 Allow to include/exclude private domains / effective TLDs without re-initialization, fixes #185 
- add optional flag excludePrivate to getAssignedDomain(...) / getEffectiveTLD(...)
- remove methods which exclude private eTLD during initialization
 2017-11-06 11:48:30 +01:00
Ken Krugler ac78f25356
Merge pull request #184 from crawler-commons/183_paidleveldomain-with-tld 
Use TLD for PLD code
 2017-11-05 15:04:01 -08:00
Ken Krugler 2b58c5050c merge with master 2017-11-05 14:55:30 -08:00
Ken Krugler aeb0cb91a2 Update CHANGES.txt 2017-11-05 14:53:58 -08:00
Ken Krugler 03163d0b3f Add pending test for private domains 2017-11-05 14:51:14 -08:00
Julien Nioche af0a013776 Released 0.9 2017-10-31 09:42:57 +00:00
Ken Krugler cb701a9da9 Use slf4j formatting support 
And avoid forbidden api check for calling String.format without a
locale.
 2017-10-28 09:59:28 -07:00
Ken Krugler c0a158d086 Fix minor typo 2017-10-27 13:53:28 -07:00
Ken Krugler 569ec9b8cd Use TLD for PLD code 2017-10-27 13:51:29 -07:00
Julien Nioche 535f96236c [maven-release-plugin] prepare for next development iteration 2017-10-27 11:13:15 +01:00
Julien Nioche 9a54f53c14 [maven-release-plugin] prepare release crawler-commons-0.9 2017-10-27 11:13:07 +01:00
Julien Nioche f3e37f37da Updated change log prior to 0.9 release 2017-10-27 11:08:47 +01:00
Julien Nioche ee69049db0 Sitemap: removed DOM parser, fixes #177 (#181) 2017-10-27 07:48:22 +01:00
Sebastian Nagel d2de87cf92 Normalize RSS pubDate, fixes #166 for SiteMapParserSAX (#180) 2017-10-18 14:07:05 +01:00
Sebastian Nagel b34e951a90 Merge pull request #178 from sebastian-nagel/cc-172-effective-tld-finder 
Improve EffectiveTldFinder, fixes #172
 2017-10-17 22:12:29 +02:00
Sebastian Nagel 0b70328671 EffectiveTLDFinder improvements, fixes #172 
- update public suffix list to recent version of
      https://publicsuffix.org/list/public_suffix_list.dat
- add method flag to force a check whether the domain has a valid
      effective TLD listed in the public suffix list
- fix mixed case hostnames (wWW.eXample.com)
- improve parsing of public suffix list
 2017-10-17 17:15:16 +02:00
Sebastian Nagel 8dea17e003 Add tests from publicsuffixlist.org 2017-10-17 13:29:14 +02:00
Sebastian Nagel 5e9d70fec6 EffectiveTldFinder: optionally exclude private domains (opposed to ICANN domains) 2017-10-17 13:29:14 +02:00
Sebastian Nagel bba44925a5 Update public suffix list to recent version of 
https://publicsuffix.org/list/public_suffix_list.dat
 2017-10-17 13:23:15 +02:00