1
1
mirror of https://gitlab.archlinux.org/archlinux/infrastructure.git synced 2025-01-18 08:06:16 +01:00
infrastructure/roles/archweb/files/robots.txt
Thorben Günther 37f0dd08d2 archweb: Add robots.txt
Closes #358
2021-07-10 22:14:32 +02:00

7 lines
149 B
Plaintext

User-agent: *
Disallow: /packages/search/
Disallow: /packages/?
Disallow: /packages/?*
Sitemap: https://www.archlinux.org/sitemap.xml
Crawl-delay: 2