Kristian Klausen
a0ef2cb7cc
loki/nginx: Add recording rule for tracking upstream cache status[1]
...
This enables us to calculate the cache hit ratio, which may help
determine whether more caching would be beneficial.
Please note that this only counts requests for which caching is enabled
(e.g. {fastcgi,proxy}_cache is configured), e.g. for static served files
cache_status will be "".
[1] http://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_cache_status
2024-12-22 15:30:40 +01:00
Kristian Klausen
a08f79607c
loki/nginx: Add recording rule for tracking HTTP/TLS version and cipher
...
Mainly because we are curious. The data may also be used to decide if we
want to drop older versions of TLS.
2024-12-22 15:07:59 +01:00
Kristian Klausen
c9e9b3c6f9
loki/prometheus: Add plumbing for using loki recording rules[1]
...
The plan is to use this for creating metrics from the nginx log lines
(e.g. requests per second).
[1] https://grafana.com/docs/loki/latest/alert/#recording-rules
2024-12-22 15:07:54 +01:00
Evangelos Foutras
b74f08dd4d
loki: switch index to TSDB
...
Missed this earlier for the upgrade to Loki 3.0.
2024-07-03 08:05:18 +03:00
Evangelos Foutras
ed822cb33c
loki: set compactor.delete_request_store: filesystem
...
Loki 3.0 requires this to be set when retention is enabled.
2024-07-02 20:38:02 +03:00
Evangelos Foutras
ac94d691f9
loki: bump schema version to v13
...
This is needed to upgrade to Loki 3.0.
2024-07-02 20:26:26 +03:00
Evangelos Foutras
3d1b95ee24
loki: increase split_queries_by_interval to 24h
...
The cloud server running Loki isn't powerful enough to do aggressive
query splitting; it makes queries execute slower and generates a lot
more queries, resulting in "too many outstanding requests" in recent
Loki versions. (Maybe a bug?)
2022-06-07 23:55:22 +03:00
Evangelos Foutras
f73b1657fe
loki: set server.log_level to "warn"
...
The default log level is "info" which logs too much to the journal.
2022-06-07 01:21:34 +03:00
Evangelos Foutras
6a3410a8f0
loki: set common.instance_addr to 127.0.0.1
...
Since upgrading to Loki 2.5, the query frontend was trying to connect to
the wireguard address instead of 127.0.0.1. This appears to be caused by
upstream commit c0bec07e0de1a23cb72e8834069 ("Loki: Implement common net
interface/instance addr") which now defaults to "" instead of 127.0.0.1.
2022-06-07 00:05:47 +03:00
Evangelos Foutras
a9988b6c67
loki: disable usage reporting
...
Doubt the Loki team is interested in our usage data, and it's also a
risk to the confidentiality of our logs (even though it's anonymous).
2022-06-07 00:05:47 +03:00
Evangelos Foutras
90023f025e
loki: bump storage schema to v12
...
< wCPO> foutrelis: v12 fixes the "all chunks in the same directory"-issue
2022-06-06 23:25:14 +03:00
Evangelos Foutras
e586a80f44
loki: lower log retention period to 1 month
...
One month of logs is about 40G and suffices for our log retrieval needs.
2022-06-06 23:25:13 +03:00
Kristian Klausen
58d2705a5b
loki: Shorten retention period to 2 months to reduce space usage
...
We are running short on storage and don't really need 3 months of log,
so reduce it to 2 months instead of expanding the disk.
2022-02-18 20:35:05 +01:00
Kristian Klausen
8cc4407ad6
loki: Simplify loki.yaml
...
Loki v2.4.0 introduced a common config section[1][2] making the config
much smaller and better defaults.
So stripped loki.yaml down, removed options where the default is "good
enough" and tweaked retention settings per the upgrade notes[3].
So the config should basically be the example loki.yaml[4] + compactor,
retention enabled and a few tweaks (ex: compress chunks with zstd).
[1] https://github.com/grafana/loki/releases/tag/v2.4.0
[2] https://grafana.com/docs/loki/latest/configuration/#common_config
[3] https://grafana.com/docs/loki/latest/upgrading/#the-single-binary-no-longer-runs-a-table-manager
[4] https://github.com/grafana/loki/blob/v2.4.0/cmd/loki/loki-local-config.yaml
2021-11-17 22:47:34 +01:00
Jan Alexander Steffens (heftig)
a2fd81c717
roles/loki: Install custom OOM configuration
...
- When loki hits 5.5G RSS, aggressively reclaim its memory. This puts
it under memory pressure.
- When loki hits 6G RSS, invoke the kernel OOM killer.
- When loki is under severe memory pressure, have systemd-oomd kill it.
2021-09-27 18:30:49 +02:00
Kristian Klausen
9d1b7d7b1f
loki: Retention period should be a multiple of periodic table duration
...
Fixes: 49091634 ("loki: Bump the retention period to 3 months")
2021-04-27 23:08:10 +02:00
Kristian Klausen
4909163498
loki: Bump the retention period to 3 months
...
We want to be able to look at trends and past incidents.
2021-04-27 02:50:49 +02:00
Sven-Hendrik Haase
3bddbf991e
loki: Increase max_query_series from default 500 to 10000
...
The default is very restrictive.
2021-04-26 20:32:13 +02:00
Kristian Klausen
7235e726d6
Implement centralized logging
...
Fix #263
2021-04-08 20:33:43 +02:00
