mirror of
https://gitlab.archlinux.org/archlinux/infrastructure.git
synced 2025-01-18 08:06:16 +01:00
8691ab7394
As the queries for searching are the most 'heavy' queries we can disallow bots to scrape through all search results. Archweb uses the same robots.txt rules for disallowing indexing search results.
5 lines
74 B
Plaintext
5 lines
74 B
Plaintext
User-agent: *
|
|
Disallow: /packages/?
|
|
Disallow: /packages/?*
|
|
Crawl-delay: 2
|