Nginx does not directly support cgi scripts so we rely on fcgiwrap. All
git repositories under /srv/repos are exported if they have a special
git-daemon-export-ok file in their .git directory.
This reverts commit d20dd7e134.
PAM 1.5.3 (2023/05/09) deprecated pam_lastlog which is no longer built
by default, so switch back to the stock PAM configuration for Dovecot.
With the ongoing git migration[1] our GitLab will gain a lot more usage,
so GitLab should get the default ssh port and then DevOps can use a
non-standard port.
We will redirect the old port (222) to the new port for some time, so
fetching won't break for existing local repositories.
[1] https://archlinux.org/news/git-migration-announcement/
This reverts commit 6d5a8767b4.
Let's try to find out the real reason first. Plus, this might not have
worked well, as the default systemd restart time limit would not catch
repeated restarts.
The gitlab-exporter service occasionally fails with errors like:
Error: missing: `response data`
Caused by: MissingData("response data")
Let's see if restarting it on errors automatically helps avoid this.
This also simplifies the request limit configuration as json API and
feed requests are cached hence a more strict request limit is not really
necessary anymore.
- Configure a uwsgi cache of 1GB for all json API and feed requests
- Cache JSON API and feed requests for 5 minutes
- Use a single global request limit zone
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
Add "noqa no-changed-when" tags to handlers using the command module.
Perhaps it is wrong of ansible-lint to flag these, since handlers are
not the best place to have conditional execution.
This script goes through all open bugs in the Arch Linux and Community
tracker and extracts the packagename from "[$pkgname]" and tries match
it to the list of packages in the repo. If there is no match the package
is assumed to be dropped from the repo and printed.
This script will give false positives, but not enough which requires
some extra filtering.
Liberally add "noqa no-changed-when" tags to the problematic tasks,
except for two "systemd-tmpfiles --create" calls. For these we can
simply include the creates= parameter in the command module's call.
Implement ip based rate limiting on gitlab-pages. This ensures we avoid
slowing down the gitlab server when extensive requests are made against
our gitlab-pages like WKD sync from Archlinux keyring service
https://docs.gitlab.com/ee/administration/pages/#rate-limits
Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
This alert only triggers for america.mirror.pkgbuild.com. Ideally, we
should be able to increase the trigger point for high-bandwidth boxes.
I don't see a straightforward way to implement it, so disable for now.
Keycloak is apparently sending over 4k worth of response headers under
some circumstances (maybe when the client sends a stale cookie?), which
causes Nginx to return a 502 error and log "upstream sent too big header
while reading response header from upstream". This is likely also
related to this upstream issue[1]. So bump the buffer to 8k.
[1] https://github.com/keycloak/keycloak/issues/16181
All lists expect arch-mirrors-announce and aur-requests[1] require the
user to be a member before they can post. Moderating mails from
nonmembers are a lot of work and most of the mails are spam, so let's
just reject them. Mails to arch-mirrors-announce and aur-requests from
nonmembers will still be checked manually, as you aren't required to be
subscribed[1].
[1] https://wiki.archlinux.org/index.php?title=General_guidelines&oldid=750602#Reply_to_the_mailing_list
This role will still handle setting up nginx and rsyncd, due to specific
configuration requirements these services have.
We're also effectively relieving build.archlinux.org of rsyncd duties as
it is not something it should be doing anyway.
There is no reason these should have such restrictive permissions, and
missing the execute bit appears to be a mistake. Let /etc/ssh keep the
mode from the package (0755). Use the same for the includes directory.
C.UTF-8 is installed by default, so we can avoid messing with locale-gen
by using that.
All the postgres servers (excluding matrix due to[1]) have been migrated
with the following commands:
# sudo -u postgres pg_dumpall > d
# sed "s/LOCALE = 'C'/LOCALE = 'C.UTF-8'/" -i d
# systemctl stop postgresql.service
# mv /var/lib/postgres/data{,.old}
$ ansible-playbook --diff -t postgres playbooks/<host>.yml
# sudo -u postgres psql < d
[1] 19a57f4a37/docs/postgres.md (fixing-incorrect-collate-or-ctype)
Fix #470
- enable_gzip: grafana listens on localhost, nginx handles compression
- admin_user: initial admin creation is disabled in our config
- strict_transport_security: the same header is set by nginx
- strict_transport_security_max_age_seconds: unused without the above
On asia.mirror.pkgbuild.com, 'smartctl -a --json $disk' has been exiting
with code 64. From smartctl(1) code 64 corresponds to "Bit 6: The device
error log contains records of errors". Since we're not interested in old
errors, ignore it.
This has become outdated (missing new dedicated servers) and its usage
can be replicated by checking if ansible_virtualization_role == "host".
For Ansible ad hoc commands, '!hcloud' can be used to the same effect.
Symlinking home.json to archive.json causes a duplicate, as both
dashboards have the same uid, and Grafana won't keep the dashboards
updated when there are duplicates[1]. Instead just change
default_home_dashboard_path to point to the archive.json dashboard.
[1] "dashboards provisioning provider has no database write permissions
because of duplicates"
WireGuard was setup to provide a internal network with confidentiality,
authenticity and integrity[1]. This migrate the remaining Prometheus
exporters to use the internal WireGuard network.
[1] 664deb67 ("WireGuard all hosts")
Fix #384
Expose aurweb RPC using goaurrpc to reduce the load on the server.
Additionally we can now geo-serve this ro reduce load and bandwidth.
Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
This commit brings in four new routes to nginx:
- /archives/metadata.git
- /archives/users.git
- /archives/pkgbases.git
- /archives/pkgnames.git
See https://gitlab.archlinux.org/archlinux/aurweb/-/blob/master/doc/git-archive.md
For now, we will be updating the repositories once every 10 minutes.
Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
Co-signed by: Kevin Morris <kevr@0cost.org>
group_vars/all was enabling just the sshd jail so move this into the
fail2ban role defaults. patchwork, security and wiki were redefining
fail2ban_jails without deviating from the group_vars/all default and
can therefore be dropped.
I have needed to use compsize on multiple occasions, and thus had to
temporarily install it. As it is already installed on 9 machines and
is generally useful, make it part of the standard tool set for btrfs.
The traffic hitting ping.archlinux.org has lately been exhausting its
default nf_conntrack_max limit of 64k. Bump it to 256k (which is also
the default limit found on systems with more than 4G of memory).
Suggested-by: Kristian Klausen <kristian@klausen.dk>
With the final lists migrated to mailman3[1], the mailman2 server can
finally be killed.
When the mailman3 server was initially setup[2], it was done on a
separate server because the mailman and mailman3 packages conflicted,
and the traffic was routed over wireguard (HTTP, LMTP and SMTP).
Instead of installing mailman3 on the original lists.al.org server and
transferring the data, it was easier just to install the missing pieces
(basically Postfix and adjusting the Nginx configuration) on the ml3
server and move the IPs (to keep the IP mail reputation).
So basically the following was done:
- The IPs for the original lists.al.org was moved to the mailman3.al.org
server
- The mailman2 datadir was transferred to mailman3.al.org server, so we
can keep the pipermail links alive, and import missing mails if needed
- The original lists.al.org server was decommissioned
- The mailman3.al.org server was renamed to lists.al.org
- The missing pieces was added to the mailman3 role (basically Postfix +
Nginx adjustments)
- The mailman role was deleted and the mailman3 role renamed to mailman
[1] 75ac7d09 ("mailman: Fourth and final batch of mailman3 migrated lists")
[2] 9294828f ("Setup mailman3 server")
Fix #59
These roles are very similar and can be merged into a single new role.
Note: The archive mirror is changed from a 4-hour sync to minutely for
conformity with the other two mirrors. In practice this doesn't matter
as it was already taking over 4 hours to finish and was starting again
right after its previous run.
The service was enabled in arch-boxes to account for "hardware clock is
not in UTC, but instead UTC+X"[1], in our case the (VM) hardware clock
is in UTC and we therfor don't need the slow systemd-time-wait-sync
service (+30 seconds).
[1] e23d3c57a0
From time to time aurweb is failing with "Too many open files"
errors[1], this could indicate a bug in aurweb or perhaps the limit is
just too low. Let's try doubling the limit and see if it helps.
[1] https://gitlab.archlinux.org/archlinux/aurweb-errors/-/issues/275
The code isn't vulnerable to nginx alias traversal[1][2], nevertheless
it should only match /static/ and not e.g. /staticfoobar.
[1] d94f18a7 ("Fix nginx alias traversal")
[2] 641060d635/docs/en/plugins/aliastraversal.md
Fixes: 9294828f ("Setup mailman3 server")
Whoosh is used by default, but it is slow at indexing (multiple hours
for just aur-requests) and searching e.g. aur-requests isn't possible
(it is slow and uses 3G+ of memory resulting in it getting OOM-killed).
Xapian indexed everything in just 76 minutes and searching aur-requests
now works and is plenty fast.
Co-authored-by: Evangelos Foutras <evangelos@foutrelis.com>
All lists have been migrated to mailman3[1] and mailman3 is what users
should use, so show its interface by default and not the mailman2
interface.
[1] 75ac7d09 ("mailman: Fourth and final batch of mailman3 migrated lists")
arch-general
aur-general
aur-requests
It has been decided not to migrate the following unlisted and unused
lists:
arch-magazine
arch-notifications
arch-test
mailman
Enable kernel lockdown in confidentiality mode to restrict how the root user can interact with the kernel.
See https://wiki.archlinux.org/title/Security#Kernel_lockdown_mode and https://man.archlinux.org/man/kernel_lockdown.7
This could prevent a scenario where a malicious kernel module or access to some interface that kernel lockdown prevents, would allow or assist in escaping the KVM.
It is not very likely as there needs to be an exploitable vulnerability in the hypervisor.
To make it more secure, the host too would need to enable kernel lockdown.
In the end this may only give some sense of security, but, as we all know, that's all that matters anyway.
Useful if we wanted to create a Geo-based archive consisting of machines
in the archive_mirrors group (though this will likely not happen because
it'd break archlinux-repro due to the ~4 hour sync delay).
http_requests_total contains requests from debuginfo.al.org host
as well as from aur.al.org so filter them on job 'aurweb'
Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
Using plain "borg" resolves to /usr/local/bin/borg which is the wrapper
for our main backup host. This causes the offsite backup to be executed
with BORG_REPO set to the main backup destination.
While the above doesn't cause any issues with the backup script/service,
because borg invocations specify the backup destination as an argument,
it's not ideal and/or correct. Adjust borg_cmd to include the full path
of /usr/bin/borg, thus removing any ambiguity.
rsync.net has upgraded to borg 1.2 and we can now run borg compact on
both rsync.net and the Hetzner storage box.
Fixes: 17927c9aa4 ("borg_client: run compact after pruning on borg 1.2")
Update /etc/rebuilderd-sync.conf to use the Geo host mirror instead of
europe.mirror.pkgbuild.com (for added reliability in case the latter's
availability is impacted).
The /api/v0/build/report endpoint has received POSTs up to 161M so far
this year (2022). In 2021 there had been POSTs of sizes up to 404M and
up to 814M for 2020. Multiple hundreds of MB seem a bit excessive, but
we should be able to do up to 200M.
Upstream archlinux-repro was already using europe.mirror.pkgbuild.com as
its bootstrap and regular mirror. Furthermore, since [1] it has switched
to the Geo mirror. Remove both vars from /etc/archlinux-repro/repro.conf
and use the default mirror values (which are more than suitable).
[1] https://github.com/archlinux/archlinux-repro/commit/c024b892d07a
The burst size of 300 reportedly allows ~150 git operations. This might
not always be sufficient when installing a lot of packages from the AUR.
Specify a higher burst size to cover most legit use cases, even if this
makes us more susceptible to abuse.
"Disabling revoked keys in keyring" when running "pacman-key --populate"
is very slow (easily +20 seconds), in our case the boot is now ~27
seconds faster (tested on secure-runner1). The pacman master private key
is removed to prevent malicious actors from injecting packages, a new
key is generated by pacman-init.service on boot.
Changes:
- Switch to arch-boxes' base image
- Verify the base image's signature
- Use the new "latest" symlink, instead of parsing the HTML for
finding the latest arch-boxes image[1]
- Create the base image by using arch-chroot and friends, instead of
creating a full-blown VM
- Create the VMs from domain XML template instead of virt-clone
- Switch mirror to geo.mirror.pkgbuild.com
- Try to follow "filesystem hierarchy" standards for where to place
configuration (id_ed25519) and "vendor data" (arch-boxes.asc and
domain_template.xml)
- Use a ed25519 key instead of RSA key
- Only start the "update base image" server if network and DNS are up
- Misc fixes and cleanups
[1] https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/552
For some workloads running in a container is too restrictive, ex:
arch-boxes (loop device, filesystem mount, pacstrap) and archiso
(pacstrap). Currently they both run a TCG accelerated QEMU VM, which is
very slow and painful to work with. We should provide a better option to
our users!
This adds a hardware accelerated VM for this kinds of workloads, which
is way faster and you can do whatever you like (mostly)!
Fix #283
This is meant as a extra "backup" and as another way for our users to
fetch PKGBUILDs from the AUR. It also allows the community to create
their own (perhaps better) "AUR" API/database as all essential data is
now available (this + [2]).
At the monent this is experimental and we aren't committing to keeping
it around.
[1] https://github.com/archlinux/aur
[2] http://aur.archlinux.org/packages-meta-ext-v1.json.gz
Jan Alexander Steffens (heftig)
Evangelos Foutras
Leonidas Spyropoulos
nl6720
Kristian Klausen
Jelle van der Waa
Levente Polyak
Evangelos Foutras
lb-wilson
moson-mo
Matt-1-2-3
Kevin Morris