mirror/infrastructure
1
1
Fork 0
mirror of https://gitlab.archlinux.org/archlinux/infrastructure.git synced 2025-01-18 08:06:16 +01:00
Code Issues
aur_robots.txt
infrastructure/roles/aurweb/files
History
Jelle van der Waa 8691ab7394
Disallow scraping the search results on the AUR 
As the queries for searching are the most 'heavy' queries we can
disallow bots to scrape through all search results. Archweb uses the
same robots.txt rules for disallowing indexing search results.
2020-09-02 21:53:48 +02:00
..
robots.txt Disallow scraping the search results on the AUR 2020-09-02 21:53:48 +02:00