dotya.ml/gitea-templates
dotya.ml/ gitea-templates
2
1
Fork 0
Code Issues 2 Pull Requests 1 Projects 2 Releases Activity

robots: match moar botz;disallow direct tree scans

This commit is contained in:
surtur 2021-10-08 16:15:54 +02:00
parent 144cf942ec
commit e53748c078
Signed by: wanderer
GPG Key ID: 19CE1EC1D9E0486D
1 changed files with 6 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
custom/public/robots.txt
View File

@ -1,16 +1,20 @@
User-agent: MJ12bot
Disallow: /
User-agent: AdsBot-Google
User-agent: AdsBot*
Disallow: /
User-agent: Mediapartners*
Disallow: /
User-agent: SemrushBot*
User-agent: *SemrushBot*
Disallow: /
@ -78,9 +82,5 @@ Disallow: /search
Disallow: */search
Disallow: /*q=
Disallow: /*.atom
Allow: /*/*/tree/master
Allow: /*/*/tree/main
Allow: /*/*/blob/master
Allow: /*/*/blob/main
Crawl-delay: 1