mirror of
https://gitlab.archlinux.org/archlinux/infrastructure.git
synced 2025-01-18 08:06:16 +01:00
f562b4e927
This should i.e. forbid crawlers to index all of the git diffs which put's unneccessary load on the server and is not really of benefit to be indexed anyways. Link: https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/610 Reviewed-by: Sven-Hendrik Haase <svenstaro@gmail.com> Reviewed-by: Levente Polyak <anthraxx@archlinux.org> Signed-off-by: Christian Heusel <christian@heusel.eu>
13 lines
458 B
Plaintext
13 lines
458 B
Plaintext
User-agent: *
|
|
# block the search page from indexing, as the search is done via URL parameters
|
|
Disallow: /packages?*
|
|
# block all interactive things from being indexed, such as posting requests
|
|
Disallow: /pkgbase/*
|
|
# block all account pages from being indexed, as they require login anyways
|
|
Disallow: /account/*
|
|
# block the cgit interface except for the useful things
|
|
Disallow: /cgit/aur.git/*
|
|
Allow: /cgit/aur.git/tree
|
|
Allow: /cgit/aur.git/log
|
|
Crawl-delay: 2
|