After=network-online.target is likely insufficient by itself.
Fixes: 104d08a1db ("archweb: Put most services after network-online.target")
Fixes: c844d0cb6c ("Split storage box monitoring into new text collector")
We noticed readlinks and reporead on gemini failing to connect to the
archweb database immediately after rebooting. To fix this, place them
after network-online.target. Do the same for all but one of the other
service units even though they run on timers 10-15 minutes after boot
for completeness and correctness.
We used to set query_cache_type to 0 in the default settings but we were
also setting query_cache_size to a non-zero/non-default value, which was
in turn re-enabling the query cache. Update the configuration to reflect
the actual cache state and make sure query_cache_size is set to zero for
the "query_cache_type = 0" case.
Now that the setting controls the real state of the query cache, disable
it for bbs.archlinux.org; its hit rate is small compared to insert rate.
Hetzner DNS has been delaying many responses for 5 seconds, causing
outgoing federation work to pile up, almost running into OOM before we
noticed.
I don't know if were being throttled because federation makes a *lot* of
requests. Anyway, using Cloudflare DNS seems to solve it.
Enable DNSOverTLS for this because we can.
This cache was seeing constant evictions at its former size (70k).
With a size of 200k it now stabilizes at ~122k entries with a
significant drop in server load.
The dashboards is from [1] and fixed with:
sed 's/${DS_PROMETHEUS}/$datasource/g' -i synapse.json
[1] c167e09fe5/contrib/grafana
Closes: #290
Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
The --delay-updates option results in 6G memory usage per archive mirror
for a total of ~18G memory used on gemini when all three archive mirrors
are syncing. Less important (but still revelant!) is the memory usage on
each mirror, which climbs to about 11G during each synchronization.
Removing the --delay-updates option should be OK considering the archive
hosts data that almost never changes. Without this option, rsync is able
to do a sequential scan which uses 90M of memory (per archive mirror) on
gemini and about 250M on each mirror individually.
Using a temporary directory outside of /srv/ftp was meant to protect
against incomplete files from being synced by downstream mirrors. It
does not achieve this to much effect though; each file gets uploaded
to the temporary directory but then immediately moved under a .~tmp~
directory at its target location (.~tmp~ because of --delay-updates,
otherwise the file would be renamed to its final path).
The `--delay-updates` option by itself sufficiently protects against
temp files being transferred to downstream mirrors; when used by the
receiver, it automatically adds an exclude rule for ~.tmp~, behaving
exactly like we want it to. As such, the `--temp-dir` option doesn't
provide any further benefit and can be removed.
- Replace --delete-after with more efficient --delete-delay.
- Move "-p" together with the other short options.
- Remove reference to empty ${VERBOSE} variable.
This reverts commit 75f9ca3cc6.
This should be fixed in rsync versions newer than 3.2.3. In Arch the fix
has been shipped in the rsync 3.2.3-4 package, which our own mirrors now
have been updated to.
[1] https://github.com/WayneD/rsync/issues/192
Kristian Klausen
Ira ㋡
Jakub Klinkovský
Jan Alexander Steffens (heftig)
Evangelos Foutras
Jelle van der Waa
Leonidas Spyropoulos