mirror/ crawler-commons
1
0
Fork 0
mirror of https://github.com/crawler-commons/crawler-commons synced 2024-05-03 22:26:15 +02:00
Code Issues
622 Commits 8 Branches 15 Tags 5.8 MiB
Commit Graph

622 Commits

Author SHA1 Message Date
kkrugler_lists@transpac.com 1b2e98afc6 Fixed up missing test dependency on slf4j-log4j 
Added more tests for "interesting" robots.txt files.
 2012-07-04 23:56:22 +00:00
digitalpebble 55da554846 added CHANGES.txt + refactoring of SiteMap objects (thanks to Hannes Schwarz) 2011-07-25 10:23:21 +00:00
kkrugler_lists@transpac.com 5324eb415d Added simple support for the file: protocol. 
Cleaned up packaging.

Added "install" target.
 2011-07-21 17:28:53 +00:00
digitalpebble 02dff51b08 package : copy build files to dist dir 2011-07-12 12:23:26 +00:00
digitalpebble@googlemail.com 790cbf9aae changing version to 0.2-SNAPSHOT 2011-07-06 18:49:12 +00:00
digitalpebble f92364a0ce Changed year to 2011 + distribute jar containing resources + copy license to root of distributed package 2011-07-06 14:56:15 +00:00
digitalpebble 6082b42c04 Added Apache License 2.0 2011-07-06 14:52:36 +00:00
digitalpebble d4c4461a73 reformat pom.xml + added stage task to build.xml 2011-07-06 14:18:35 +00:00
digitalpebble@googlemail.com d429b67182 pre-initial release : added dev info to pom.xml + ANT tasks for deployment to Maven public repository 2011-07-06 11:02:04 +00:00
kkrugler_lists@transpac.com 5d533e7e0d Add jar that's only in (currently unavailable) 101tec Nexus repo, so at least users can manually install it 2011-07-01 17:42:11 +00:00
kkrugler_lists@transpac.com 5d6825f34e Remove unneeded dependency on 101tec and Apache snapshot repositories 2011-07-01 17:02:23 +00:00
digitalpebble@googlemail.com 6b977fd672 Added missing license headers 2011-06-04 09:28:57 +00:00
kkrugler_lists@transpac.com acfaa17090 Test code for robots.txt processing code, HTTP fetcher 2011-06-03 22:14:01 +00:00
kkrugler_lists@transpac.com 1bcd2062a8 Test code for robots.txt processing code, HTTP fetcher 2011-06-03 22:12:47 +00:00
kkrugler_lists@transpac.com cd55b6b92e Preliminary versions of robots.txt processing code, HTTP fetcher 2011-06-03 21:30:09 +00:00
kkrugler_lists@transpac.com b37cdd0baa Preliminary versions of robots.txt processing code, HTTP fetcher 2011-06-03 21:29:34 +00:00
digitalpebble ced3685969 unified logging with slf4j 2010-06-04 11:16:20 +00:00
digitalpebble 9c1bbdc544 SiteMap Parser : see discussion on [http://groups.google.com/group/crawler-commons/browse_thread/thread/9813bba4bcde8ace/] 2010-04-26 10:54:16 +00:00
digitalpebble 89d9dc7b4e improved list of compound tlds - see NUTCH-786 2010-02-09 09:36:28 +00:00
kkrugler_lists@transpac.com bf8ba66115 Rolled in Ian's patches to pom.xml and build.xml 
Rolled in Ian's EffectiveTldFinder code & test cases.

Fixed "dist" target for build.
 2009-12-12 00:22:44 +00:00
kkrugler_lists@transpac.com ddc38ce331 Change name of format from "Bixo" to "Crawler-commons" 2009-12-04 04:19:21 +00:00
kkrugler_lists@transpac.com 288dca1504 Initial commit of build system, plus some paid-level domain extraction code from Bixo. 2009-12-04 04:13:38 +00:00