Sebastian Nagel d685bafb2d [Robots.txt] SimpleRobotRulesParser main() to follow five redirects (#428) ... when fetching robots.txt over HTTP as required by RFC 9309		2023-07-11 14:49:00 +01:00
..
main	[Robots.txt] SimpleRobotRulesParser main() to follow five redirects (#428)	2023-07-11 14:49:00 +01:00
test	[Robots.txt] Empty disallow statement not to clear other rules, fixes #422 (#424)	2023-07-11 14:47:33 +01:00