1
1
Fork 0
mirror of https://gitlab.archlinux.org/archlinux/infrastructure.git synced 2024-05-24 19:06:03 +02:00
Commit Graph

145 Commits

Author SHA1 Message Date
Kristian Klausen 24e73359c6
Decommission patchwork.archlinux.org and replace it with a static copy[1]
As announced on the mailing list[2] pacman has been migrated to gitlab
and there is no real use for patchwork left, so it can be
decommissioned. A static copy[1] is kept around for the time being to
avoid link rot.

[1] https://gitlab.archlinux.org/archlinux/patchwork-archive
[2] https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/message/7B6R5HVEC67U7B2VQ3SKUVXU4RDCRRMM/

Fix #487
2023-01-08 22:10:49 +01:00
Kristian Klausen 893a95f329
Kill the mailman2 server and put the mailman3 server in its place
With the final lists migrated to mailman3[1], the mailman2 server can
finally be killed.

When the mailman3 server was initially setup[2], it was done on a
separate server because the mailman and mailman3 packages conflicted,
and the traffic was routed over wireguard (HTTP, LMTP and SMTP).

Instead of installing mailman3 on the original lists.al.org server and
transferring the data, it was easier just to install the missing pieces
(basically Postfix and adjusting the Nginx configuration) on the ml3
server and move the IPs (to keep the IP mail reputation).

So basically the following was done:
- The IPs for the original lists.al.org was moved to the mailman3.al.org
  server
- The mailman2 datadir was transferred to mailman3.al.org server, so we
  can keep the pipermail links alive, and import missing mails if needed
- The original lists.al.org server was decommissioned
- The mailman3.al.org server was renamed to lists.al.org
- The missing pieces was added to the mailman3 role (basically Postfix +
  Nginx adjustments)
- The mailman role was deleted and the mailman3 role renamed to mailman

[1] 75ac7d09 ("mailman: Fourth and final batch of mailman3 migrated lists")
[2] 9294828f ("Setup mailman3 server")

Fix #59
2022-09-28 21:04:41 +02:00
Evangelos Foutras 595e3316c4
aurweb/rspamd: append .vault to vaulted secrets
This avoids triggering a GitLab push rule which rejects files that look
like secrets.
2022-09-17 22:10:08 +03:00
Evangelos Foutras 578b781966
Capitalize the handler name in handler invocations
Fixes: 26f289b72b ("Capitalize the first letter of all task names")
2022-08-29 21:46:39 +03:00
Evangelos Foutras 83f0066a14
Replace references to Packet.net with Equinix Metal
Also tweak the documentation on rebuilderd workers and add runner1.
2022-07-10 12:42:15 +03:00
Kristian Klausen 2a74897bfb
Add Vagrant Cloud account
Vagrant Cloud has been used for years by arch-boxes[1] for publishing
Vagrant boxes. Access to the organization[2] was handed out to a few
members of the DevOps team and the creator of the organization
(arch-boxes maintainer at the time).

With this commit the control of the organization is handed over to the
DevOps team through a new Vagrant Cloud account.

[1] https://gitlab.archlinux.org/archlinux/arch-boxes
[2] https://app.vagrantup.com/archlinux/
2022-05-29 21:20:06 +02:00
Kristian Klausen d1c23d5d96
Update host keys for mailman3.archlinux.org
The server has been reimaged to be sure the playbook and roles work as
intended.
2022-05-14 22:52:16 +02:00
Kristian Klausen 9294828f15
Setup mailman3 server
We want to migrate to mailman3 as mailman2 is basically unmaintained and
requires Python 2 which is EOL.

Because the mailman and mailman3 packages conflict and we don't want to
perform a big bang migration, mailman3 must be deployed on a separate
server. mailman-web (mailman3's web interface) hasn't been packaged yet,
so for now we are using my homebrewed PKGBUILD[1].

[1] https://gist.github.com/klausenbusk/5982063f95c503754a51ed2fefb8915e

Ref #59
2022-05-14 22:51:59 +02:00
Evangelos Foutras afb582b108
geomirror: extract acme dns challenge into new role
- add the new role to redirect.archlinux.org
- release mirror.pkgbuild.com of all DNS duties
2022-05-14 14:22:32 +03:00
Evangelos Foutras 248e57b3c5
Remove [node_exporters]/[wireguard] from inventory
All servers are part of these groups which makes them redundant.
2022-05-12 19:40:49 +03:00
Evangelos Foutras b4d60ae2f6
Move highly sensitive secrets to new "super" vault
The idea bebind this is to be able to give vault access to new DevOps
members without giving away more important credentials like Hetzner's.
2022-05-07 17:45:19 +03:00
Kristian Klausen 9f65f99c6b
Add GeoIP domain for our sponsored mirros
We had a GeoIP mirror in the past based on nginx and its GeoIP module,
but it didn't perform very well, due to the high latency (asking a
central server for the package and then redirected to the closest
mirror).

One of the reasons for offering this service, is so we can relieve
mirror.pkgbuild.com which is burning a ton of traffic (50TB/month),
likely due to it being the default mirror in our Docker image. Another
reason is so we can offer a link to our arch-boxes images in libosinfo
(used by gnome-boxes, virt-install and virt-manager), with good enough
performance for most users.

This time we take a different approach and use a DNS based solution,
which means the latency penalty is only paid once (the first DNS
request). The downside is that the mirrors must have a valid certificate
for the same domain name, which makes using third-party mirrors a
challenge. So for now, we are just using the sponsored mirorrs
controlled by the DevOps team.

Fix #101
2022-04-13 03:10:09 +02:00
Kristian Klausen af5d4b845e
Decommission aur-dev
With the PHP->Python port done[1][2], there isn't much need for aur-dev
anynmore. Most things can also be tested locally and aur-dev haven't got
any love since the port (ex: allowing the aurweb maintainers to deploy
without asking DevOps).

[1] https://lists.archlinux.org/pipermail/aur-general/2022-February/036786.html
[2] https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/525
2022-04-11 14:55:53 +02:00
Evangelos Foutras 03600a8cc4
Place borg host vaults under host_vars/localhost/
Kind of sensitive information that doesn't need to be available to all
hosts.
2022-02-26 11:08:30 +02:00
Evangelos Foutras 6c60c6bd40
tasks/sync-ssh-hostkeys: allow custom known_hosts
Change docs/ssh-known_hosts.txt to be partially managed by Ansible, so
custom entries can be added to the top of the file. Use the new format
to write down the host keys of our two borg hosts.
2022-02-25 09:16:35 +02:00
Brett Cornwall 3c5c7f6975 doc: clean up the markdown formatting 2022-02-23 20:45:48 +00:00
Evangelos Foutras 0643978a10
gitlab: Do away with generating tarball backups
Using GitLab's official backup tool takes too much time and, more
importantly, space; /srv/gitlab is a bit over 430G but backing it
up nearly exhausts its 1TB volume.

As we're creating btrfs snapshots and backing those up with borg, it
seems unnecessary to also create tarballs of the same data. GitLab's
documentation mentions snapshots as a viable backup strategy, and to
the restored system it should seem like recovering from a power loss.

[1] https://docs.gitlab.com/ee/raketasks/backup_restore#alternative-backup-strategies
2022-02-05 22:46:52 +02:00
Kristian Klausen 9a7483832c Provision debuginfod server 2022-02-04 21:13:19 +00:00
Jelle van der Waa d88c0b953e
Initialize gluebuddy host 2022-01-21 10:30:05 +01:00
Jelle van der Waa bf5a165303
Add smart monitoring using a textcollector
Collects the smart data using smartctl and outputs them in the
textcollector dir. This expects smartd to be configured to regularly
self tests on a regular interval to detect if a disk is broken.
2022-01-04 21:25:41 +01:00
Jelle van der Waa 83025e6de7
docs: document passwd -d for mail.archlinux.org 2021-12-01 13:17:03 +01:00
Evangelos Foutras 200df4f884
Avoid using username placeholders in backup docs
These are already known (so no need to hide them) and are fairly static
(so variables are more of a hindrance) so it's better to use the actual
usernames in the documentation. Also, simplify the first example given.
2021-11-06 20:02:28 +02:00
Evangelos Foutras 69994e900a
Complete rsync.net account migration
New username; separate and longer account manager + storage passwords.

Also, have to use --remote-path=borg1 when interacting with rsync.net.
2021-11-06 19:50:31 +02:00
Evangelos Foutras 21f7ce7925
Remove outdated Borg textcollector bit from docs
prometheus-borg-textcollector is no longer started by timer, but instead
defines a WantedBy= relationship with the borg-backup{,-offsite} service.
2021-11-05 14:50:25 +02:00
Jelle van der Waa 101f428fdb
Mention postgresql/gitlab/mariadb backups
Document how we backup our databases/gitlab instances.
2021-09-04 22:00:01 +02:00
Jelle van der Waa 77753e266f
Update banning docs for wireguard 2021-08-01 17:03:00 +02:00
Kristian Klausen 208a533910
Remove reference to disabled STARTTLS Submission (port 587)
Disabled in:
0ae67c4a ("postfix: Disable STARTTLS Submission (port 587)")
2021-07-30 15:49:21 +02:00
Kristian Klausen 3ba230b17c Replace runner1 with a new bigger box
CPU: Intel Xeon E5-2620 -> E-2288G
Disk: 2x~1TB -> 2x~500GB
2021-07-21 00:40:59 +02:00
Kristian Klausen 2304dc5caa Split the postfix role into a role for mail.a.o and the clients
The role for the clients is named postfix_null (per [1]) and it's much
simpler and cleaner than the postfix role. I hope can cleanup the
postfix role at a later date.

[1] http://www.postfix.org/STANDARD_CONFIGURATION_README.html#null_client
2021-07-16 20:02:05 +02:00
Jelle van der Waa 314aa3556e
Rate limit archlinux.org and fail2ban abusers
Add a default rate limit for 20 req/s for the uwsgi endpoint and
automatically ban users who reach this limit. The nginx-limit-req rule
does not ban users who reach the rss limit as these are not likely DoS
attempts.
2021-07-11 21:10:38 +02:00
Kristian Klausen 9a513cadd4
docs/email: Remove reference to removed SMTP port 10027
The port was removed in:
4729ba40 ("postfix: Remove special "fast-path" smtpd")
2021-07-09 22:40:09 +02:00
Kristian Klausen 79f7d59910 Goodbye luna
https://lists.archlinux.org/pipermail/arch-dev-public/2021-July/030471.html

Fix #86
2021-07-04 12:46:01 +00:00
Jelle van der Waa 0d2c553db0
Document rsync.net OTP token setup 2021-07-03 15:22:12 +02:00
Evangelos Foutras 62c78dfa1c
Use sub-accounts for backups to Hetzner Storage Box
This offers improved separation between the server backups and should
avoid bumping against the storage box 10 concurrent connection limit.

Fixes: https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/362
2021-07-02 19:04:19 +03:00
Kristian Klausen ac8e5863c5 Update host keys and known hosts 2021-06-30 09:30:31 +00:00
Kristian Klausen 41c5a5e26c Add initial playbook for lists.archlinux.org
nginx, certbot, postfix and mailman are still missing and the DNS is
still pointing to luna.
2021-06-30 09:30:31 +00:00
Kristian Klausen a998e80dcf Change IRC network to Libera Chat[1]
[1] https://archlinux.org/news/move-of-official-irc-channels-to-liberachat/
2021-06-03 20:54:17 +00:00
Jan Alexander Steffens (heftig) 407163f39b
matrix: Move IRC bridge to Libera Chat 2021-06-01 18:44:21 +02:00
Jelle van der Waa 7c2af1598d
Add aur-dev to our servers list 2021-05-28 14:40:29 +02:00
Jelle van der Waa d014d839a4
Document our Gitlab Servicedesk aliases 2021-05-23 21:58:30 +02:00
Leonidas Spyropoulos e5773374fe fail2ban: Ban IPs based on nginx request abuse
Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
2021-05-23 19:50:25 +00:00
Jelle van der Waa 7b216d5710
Add otp documentation for uptimerobot 2021-05-18 22:51:16 +02:00
Jelle van der Waa c6baa1dfc9
Document adding a dashboard to our public instance 2021-05-15 18:08:25 +02:00
Jelle van der Waa 1f37eb16ef
Document how to add metrics to dashboards.archlinux.org
Document how to whitelist some metrics for the public Grafana instance.

Closes: #334
2021-05-15 17:52:07 +02:00
Jelle van der Waa d6626844c0
Add AUR monitoring documentation 2021-05-15 17:18:06 +02:00
Jelle van der Waa 4321309d76
Add docs for firewalld-cmd banning 2021-05-15 13:41:26 +02:00
Kristian Klausen 9ef30adb21 Mention dashboards.archlinux.org in the docs 2021-05-13 23:36:44 +02:00
Jakub Klinkovský 63736ab38d add docs/testing.md 2021-04-26 23:37:30 +00:00
Jelle van der Waa a9c2da23e0
Add Loki documentation to monitoring 2021-04-08 22:02:10 +02:00
Kristian Klausen 7235e726d6
Implement centralized logging
Fix #263
2021-04-08 20:33:43 +02:00