1
1
mirror of https://github.com/docker-mailserver/docker-mailserver synced 2024-10-18 10:18:07 +02:00

breaking: Refactor getmail support (#4156)

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
This commit is contained in:
Casper 2024-08-17 12:14:59 +02:00 committed by GitHub
parent fb57905aa3
commit b2978fd760
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
20 changed files with 198 additions and 59 deletions

@ -6,6 +6,19 @@ All notable changes to this project will be documented in this file. The format
> **Note**: Changes and additions listed here are contained in the `:edge` image tag. These changes may not be as stable as released changes.
### Breaking
- **getmail6** has been refactored: ([#4156](https://github.com/docker-mailserver/docker-mailserver/pull/4156))
- The [DMS config volume](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/optional-config/#volumes) now has support for `getmailrc_general.cf` for overriding [common default settings](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/mail-getmail/#common-options). If you previously mounted this config file directly to `/etc/getmailrc_general` you should switch to our config volume support.
- IMAP/POP3 example configs added to our [`config-examples`](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/config-examples/getmail).
- ENV [`GETMAIL_POLL`](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/environment/#getmail_poll) now supports values above 30 minutes.
- Added `getmail` as a new service for `supervisor` to manage, replacing cron for periodic polling.
- Generated getmail configuration files no longer set the `message_log` option. Instead of individual log files per config, the [default base settings DMS configures](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/target/getmail/getmailrc_general) now enables `message_log_syslog`. This aligns with how other services in DMS log to syslog where it is captured in `mail.log`.
- Getmail configurations have changed location from the base of the DMS Config Volume, to the `getmail/` subdirectory. Any existing configurations **must be migrated manually.**
- DMS v14 mistakenly relocated the getmail state directory to the DMS Config Volume as a `getmail/` subdirectory.
- This has been corrected to `/var/lib/getmail` (_if you have mounted a DMS State Volume to `/var/mail-state`, `/var/lib/getmail` will be symlinked to `/var/mail-state/lib-getmail`_).
- To preserve this state when upgrading to DMS v15, **you must manually migrate `getmail/` from the _DMS Config Volume_ to `lib-getmail/` in the _DMS State Volume_.**
### Security
- **Fail2ban:**

@ -212,7 +212,8 @@ EOF
RUN echo 'Reason_Message = Message {rejectdefer} due to: {spf}.' >>/etc/postfix-policyd-spf-python/policyd-spf.conf
COPY target/fetchmail/fetchmailrc /etc/fetchmailrc_general
COPY target/getmail/getmailrc /etc/getmailrc_general
COPY target/getmail/getmailrc_general /etc/getmailrc_general
COPY target/getmail/getmail-service.sh /usr/local/bin/
COPY target/postfix/main.cf target/postfix/master.cf /etc/postfix/
# DH parameters for DHE cipher suites, ffdhe4096 is the official standard 4096-bit DH params now part of TLS 1.3

@ -1,3 +1,5 @@
# https://getmail6.org/configuration.html#conf-options
[options]
verbose = 0
read_all = false
@ -5,3 +7,5 @@ delete = false
max_messages_per_session = 500
received = false
delivered_to = false
message_log_syslog = true

@ -0,0 +1,13 @@
# https://getmail6.org/configuration.html
[retriever]
type = SimpleIMAPSSLRetriever
server = imap.gmail.com
username = alice
password = notsecure
[destination]
type = MDA_external
path = /usr/lib/dovecot/deliver
allow_root_commands = true
arguments =("-d","user1@example.com")

@ -0,0 +1,13 @@
# https://getmail6.org/configuration.html
[retriever]
type = SimplePOP3SSLRetriever
server = pop3.gmail.com
username = alice
password = notsecure
[destination]
type = MDA_external
path = /usr/lib/dovecot/deliver
allow_root_commands = true
arguments =("-d","user1@example.com")

@ -10,14 +10,19 @@ environment:
- GETMAIL_POLL=5
```
In your DMS config volume (eg: `docker-data/dms/config/`), create a `getmail-<ID>.cf` file for each remote account that you want to retrieve mail and store into a local DMS account. `<ID>` should be replaced by you, and is just the rest of the filename (eg: `getmail-example.cf`). The contents of each file should be configuration like documented below.
In your DMS config volume (eg: `docker-data/dms/config/`), add a subdirectory `getmail/` for including your getmail config files (eg: `imap-example.cf`) for each remote account that you want to retrieve mail from and deliver to the mailbox of a DMS account.
The directory structure should similar to this:
The content of these config files is documented in the next section with an IMAP and POP3 example to reference.
The directory structure should look similar to this:
```txt
├── docker-data/dms/config
│   ├── dovecot.cf
│   ├── getmail-example.cf
│ ├── getmail
│   │ ├── getmailrc_general.cf
│   │ ├── remote-account1.cf
│   │ ├── remote-account2.cf
│   ├── postfix-accounts.cf
│   └── postfix-virtual.cf
├── docker-compose.yml
@ -42,7 +47,11 @@ received = false
delivered_to = false
```
If you want to use a different base config, mount a file to `/etc/getmailrc_general`. This file will replace the default "Common Options" base config above, that all `getmail-<ID>.cf` files will extend with their configs when used.
The DMS integration for Getmail generates a `getmailrc` config that prepends the common options of the base config to each remote account config file (`*.cf`) found in the DMS Config Volume `getmail/` directory.
!!! tip "Change the base options"
Add your own base config as `getmail/getmailrc_general.cf` into the DMS Config Volume. It will replace the DMS defaults shown above.
??? example "IMAP Configuration"
@ -54,7 +63,7 @@ If you want to use a different base config, mount a file to `/etc/getmailrc_gene
```getmailrc
[retriever]
type = SimpleIMAPRetriever
type = SimpleIMAPSSLRetriever
server = imap.gmail.com
username = alice
password = notsecure
@ -71,7 +80,7 @@ If you want to use a different base config, mount a file to `/etc/getmailrc_gene
```getmailrc
[retriever]
type = SimplePOP3Retriever
type = SimplePOP3SSLRetriever
server = pop3.gmail.com
username = alice
password = notsecure
@ -84,7 +93,7 @@ If you want to use a different base config, mount a file to `/etc/getmailrc_gene
### Polling Interval
By default the `getmail` service checks external mail accounts for new mail every 5 minutes. That polling interval is configurable via the `GETMAIL_POLL` ENV variable, with a value in minutes (_default: 5, min: 1, max: 30_):
By default the `getmail` service checks external mail accounts for new mail every 5 minutes. That polling interval is configurable via the `GETMAIL_POLL` ENV variable, with a value in minutes (_default: 5, min: 1_):
```yaml
environment:

@ -732,7 +732,7 @@ Enable or disable `getmail`.
##### GETMAIL_POLL
- **5** => `getmail` The number of minutes for the interval. Min: 1; Max: 30; Default: 5.
- **5** => `getmail` The number of minutes for the interval. Min: 1; Default: 5.
#### OAUTH2

@ -430,7 +430,7 @@ FETCHMAIL_PARALLEL=0
# - 1 => Enabled
ENABLE_GETMAIL=0
# The number of minutes for the interval. Min: 1; Max: 30.
# The number of minutes for the interval. Min: 1; Default: 5.
GETMAIL_POLL=5
# -----------------------------------------------

@ -5,9 +5,20 @@ source /usr/local/bin/helpers/log.sh
# shellcheck source=../scripts/startup/setup-stack.sh
source /usr/local/bin/setup.d/getmail.sh
_setup_getmail
# Setup getmail, even if not enabled.
ENABLE_GETMAIL=1 _setup_getmail
GETMAILDIR=/tmp/docker-mailserver/getmail
for FILE in /etc/getmailrc.d/getmailrc*; do
getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}" --dump | tail -n +6
# Directory, where "oldmail" files are stored.
# Getmail stores its state - its "memory" of what it has seen in your POP/IMAP account - in the oldmail files.
GETMAIL_DIR=/var/lib/getmail
# If no matching filenames are found, and the shell option nullglob is disabled, the word is left unchanged.
# If the nullglob option is set, and no matches are found, the word is removed.
shopt -s nullglob
# Dump configuration from each RC file.
for RC_FILE in /etc/getmailrc.d/*; do
echo "${RC_FILE##*/}:"
echo
getmail --getmaildir "${GETMAIL_DIR}" --rcfile "${RC_FILE}" --dump | tail -n +6
done

@ -1,8 +0,0 @@
#! /bin/bash
GETMAILDIR=/tmp/docker-mailserver/getmail
for FILE in /etc/getmailrc.d/getmailrc*; do
if ! pgrep -f "${FILE}$" &>/dev/null; then
getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}"
fi
done

@ -0,0 +1,47 @@
#!/bin/bash
# shellcheck source=../scripts/helpers/log.sh
source /usr/local/bin/helpers/log.sh
# Directory, where "oldmail" files are stored.
# getmail stores its state - its "memory" of what it has seen in your POP/IMAP account - in the oldmail files.
GETMAIL_DIR=/var/lib/getmail
# Kill all child processes on EXIT.
# Otherwise 'supervisorctl restart getmail' leads to zombie 'sleep' processes.
trap 'pkill --parent ${$}' EXIT
function _syslog_error() {
logger --priority mail.err --tag getmail "${1}"
}
function _stop_service() {
_syslog_error "Stopping service"
exec supervisorctl stop getmail
}
# Verify the correct value for GETMAIL_POLL. Valid are any numbers greater than 0.
if [[ ! ${GETMAIL_POLL} =~ ^[0-9]+$ ]] || [[ ${GETMAIL_POLL} -lt 1 ]]; then
_syslog_error "Invalid value for GETMAIL_POLL: ${GETMAIL_POLL}"
_stop_service
fi
# If no matching filenames are found, and the shell option nullglob is disabled, the word is left unchanged.
# If the nullglob option is set, and no matches are found, the word is removed.
shopt -s nullglob
# Run each getmailrc periodically.
while :; do
for RC_FILE in /etc/getmailrc.d/*; do
_log 'debug' "Processing ${RC_FILE}"
getmail --getmaildir "${GETMAIL_DIR}" --rcfile "${RC_FILE}"
done
# Stop service if no configuration is found.
if [[ -z ${RC_FILE} ]]; then
_syslog_error 'No configuration found'
_stop_service
fi
sleep "${GETMAIL_POLL}m"
done

@ -0,0 +1,11 @@
# https://getmail6.org/configuration.html#conf-options
[options]
verbose = 0
read_all = false
delete = false
max_messages_per_session = 500
received = false
delivered_to = false
message_log_syslog = true

@ -161,6 +161,7 @@ function _register_functions() {
[[ ${ENABLE_CLAMAV} -eq 1 ]] && _register_start_daemon '_start_daemon_clamav'
[[ ${ENABLE_AMAVIS} -eq 1 ]] && _register_start_daemon '_start_daemon_amavis'
[[ ${ACCOUNT_PROVISIONER} == 'FILE' ]] && _register_start_daemon '_start_daemon_changedetector'
[[ ${ENABLE_GETMAIL} -eq 1 ]] && _register_start_daemon '_start_daemon_getmail'
}
# ------------------------------------------------------------

@ -34,6 +34,7 @@ function _start_daemon_clamav { _default_start_daemon 'clamav' ;
function _start_daemon_cron { _default_start_daemon 'cron' ; }
function _start_daemon_dovecot { _default_start_daemon 'dovecot' ; }
function _start_daemon_fail2ban { _default_start_daemon 'fail2ban' ; }
function _start_daemon_getmail { _default_start_daemon 'getmail' ; }
function _start_daemon_opendkim { _default_start_daemon 'opendkim' ; }
function _start_daemon_opendmarc { _default_start_daemon 'opendmarc' ; }
function _start_daemon_postgrey { _default_start_daemon 'postgrey' ; }

@ -4,38 +4,46 @@ function _setup_getmail() {
if [[ ${ENABLE_GETMAIL} -eq 1 ]]; then
_log 'trace' 'Preparing Getmail configuration'
local GETMAILRC ID CONFIGS
local GETMAIL_RC ID GETMAIL_DIR
GETMAILRC='/etc/getmailrc.d'
CONFIGS=0
local GETMAIL_CONFIG_DIR='/tmp/docker-mailserver/getmail'
local GETMAIL_RC_DIR='/etc/getmailrc.d'
local GETMAIL_RC_GENERAL_CF="${GETMAIL_CONFIG_DIR}/getmailrc_general.cf"
local GETMAIL_RC_GENERAL='/etc/getmailrc_general'
mkdir -p "${GETMAILRC}"
# Create the directory /etc/getmailrc.d to place the user config in later.
mkdir -p "${GETMAIL_RC_DIR}"
# Generate getmailrc configs, starting with the `/etc/getmailrc_general` base config,
# Add a unique `message_log` config, then append users own config to the end.
for FILE in /tmp/docker-mailserver/getmail-*.cf; do
if [[ -f ${FILE} ]]; then
CONFIGS=1
ID=$(cut -d '-' -f 3 <<< "${FILE}" | cut -d '.' -f 1)
local GETMAIL_CONFIG="${GETMAILRC}/getmailrc-${ID}"
cat /etc/getmailrc_general >"${GETMAIL_CONFIG}"
echo -e "message_log = /var/log/mail/getmail-${ID}.log\n" >>"${GETMAIL_CONFIG}"
cat "${FILE}" >>"${GETMAIL_CONFIG}"
fi
done
if [[ ${CONFIGS} -eq 1 ]]; then
cat >/etc/cron.d/getmail << EOF
*/${GETMAIL_POLL} * * * * root /usr/local/bin/getmail-cron
EOF
chmod -R 600 "${GETMAILRC}"
# Check if custom getmailrc_general.cf file is present.
if [[ -f "${GETMAIL_RC_GENERAL_CF}" ]]; then
_log 'debug' "Custom 'getmailrc_general.cf' found"
cp "${GETMAIL_RC_GENERAL_CF}" "${GETMAIL_RC_GENERAL}"
fi
# Both the debug command and cron job (that runs getmail) for getmail
# expect this location to exist.
GETMAILDIR=/tmp/docker-mailserver/getmail
mkdir -p "${GETMAILDIR}"
# If no matching filenames are found, and the shell option nullglob is disabled, the word is left unchanged.
# If the nullglob option is set, and no matches are found, the word is removed.
shopt -s nullglob
# Generate getmailrc configs, starting with the `/etc/getmailrc_general` base config, then appending users own config to the end.
for FILE in "${GETMAIL_CONFIG_DIR}"/*.cf; do
if [[ ${FILE} =~ /getmail/(.+)\.cf ]] && [[ ${FILE} != "${GETMAIL_RC_GENERAL_CF}" ]]; then
ID=${BASH_REMATCH[1]}
_log 'debug' "Processing getmail config '${ID}'"
GETMAIL_RC=${GETMAIL_RC_DIR}/${ID}
cat "${GETMAIL_RC_GENERAL}" "${FILE}" >"${GETMAIL_RC}"
fi
done
# Strip read access from non-root due to files containing secrets:
chmod -R 600 "${GETMAIL_RC_DIR}"
# Directory, where "oldmail" files are stored.
# For more information see: https://getmail6.org/faq.html#faq-about-oldmail
# The debug command for getmail expects this location to exist.
GETMAIL_DIR=/var/lib/getmail
_log 'debug' "Creating getmail state-dir '${GETMAIL_DIR}'"
mkdir -p "${GETMAIL_DIR}"
else
_log 'debug' 'Getmail is disabled'
fi

@ -23,6 +23,7 @@ function _setup_save_states() {
[[ ${ENABLE_CLAMAV} -eq 1 ]] && SERVICEDIRS+=('lib/clamav')
[[ ${ENABLE_FAIL2BAN} -eq 1 ]] && SERVICEDIRS+=('lib/fail2ban')
[[ ${ENABLE_FETCHMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/fetchmail')
[[ ${ENABLE_GETMAIL} -eq 1 ]] && SERVICEDIRS+=('lib/getmail')
[[ ${ENABLE_MTA_STS} -eq 1 ]] && SERVICEDIRS+=('lib/mta-sts')
[[ ${ENABLE_POSTGREY} -eq 1 ]] && SERVICEDIRS+=('lib/postgrey')
[[ ${ENABLE_RSPAMD} -eq 1 ]] && SERVICEDIRS+=('lib/rspamd')

@ -170,3 +170,12 @@ stderr_logfile=/var/log/supervisor/%(program_name)s.log
command=/usr/bin/mta-sts-daemon --config /etc/mta-sts-daemon.yml
user=_mta-sts
environment=HOME=/var/lib/mta-sts
[program:getmail]
startsecs=0
stopwaitsecs=55
autostart=false
autorestart=true
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
command=/bin/bash -l -c /usr/local/bin/getmail-service.sh

@ -8,12 +8,13 @@ function setup_file() {
local CUSTOM_SETUP_ARGUMENTS=(--env 'ENABLE_GETMAIL=1')
_init_with_defaults
mv "${TEST_TMP_CONFIG}/getmail/getmail-user3.cf" "${TEST_TMP_CONFIG}/getmail-user3.cf"
_common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
}
function teardown_file() { _default_teardown ; }
#? The file used in the following tests is placed in test/config/getmail/user3.cf
@test 'default configuration exists and is correct' {
_run_in_container cat /etc/getmailrc_general
assert_success
@ -24,15 +25,16 @@ function teardown_file() { _default_teardown ; }
assert_line 'max_messages_per_session = 500'
assert_line 'received = false'
assert_line 'delivered_to = false'
assert_line 'message_log_syslog = true'
_run_in_container_bash '[[ -f /usr/local/bin/debug-getmail ]]'
assert_success
_run_in_container_bash '[[ -f /usr/local/bin/getmail-cron ]]'
_run_in_container_bash '[[ -f /usr/local/bin/getmail-service.sh ]]'
assert_success
}
@test 'debug-getmail works as expected' {
_run_in_container cat /etc/getmailrc.d/getmailrc-user3
_run_in_container cat /etc/getmailrc.d/user3
assert_success
assert_line '[options]'
assert_line 'verbose = 0'
@ -41,7 +43,7 @@ function teardown_file() { _default_teardown ; }
assert_line 'max_messages_per_session = 500'
assert_line 'received = false'
assert_line 'delivered_to = false'
assert_line 'message_log = /var/log/mail/getmail-user3.log'
assert_line 'message_log_syslog = true'
assert_line '[retriever]'
assert_line 'type = SimpleIMAPSSLRetriever'
assert_line 'server = imap.remote-service.test'
@ -55,19 +57,18 @@ function teardown_file() { _default_teardown ; }
_run_in_container /usr/local/bin/debug-getmail
assert_success
assert_line --regexp 'retriever:.*SimpleIMAPSSLRetriever\(ca_certs="None", certfile="None", getmaildir="\/tmp\/docker-mailserver\/getmail", imap_on_delete="None", imap_search="None", keyfile="None", mailboxes="\(.*INBOX.*\)", move_on_delete="None", password="\*", password_command="\(\)", port="993", record_mailbox="True", server="imap.remote-service.test", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="\(\)", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="False", username="user3"\)'
assert_line --regexp 'retriever:.*SimpleIMAPSSLRetriever\(ca_certs="None", certfile="None", getmaildir="\/var\/lib\/getmail", imap_on_delete="None", imap_search="None", keyfile="None", mailboxes="\(.*INBOX.*\)", move_on_delete="None", password="\*", password_command="\(\)", port="993", record_mailbox="True", server="imap.remote-service.test", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="\(\)", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="False", username="user3"\)'
assert_line --regexp 'destination:.*MDA_external\(allow_root_commands="True", arguments="\(.*-d.*user3@example.test.*\)", command="deliver", group="None", ignore_stderr="False", path="\/usr\/lib\/dovecot\/deliver", pipe_stdout="True", unixfrom="False", user="None"\)'
assert_line ' delete : False'
assert_line ' delete_after : 0'
assert_line ' delete_bigger_than : 0'
assert_line ' delivered_to : False'
assert_line ' fingerprint : False'
assert_line ' logfile : logfile(filename="/var/log/mail/getmail-user3.log")'
assert_line ' max_bytes_per_session : 0'
assert_line ' max_message_size : 0'
assert_line ' max_messages_per_session : 500'
assert_line ' message_log : /var/log/mail/getmail-user3.log'
assert_line ' message_log_syslog : False'
assert_line ' message_log : None'
assert_line ' message_log_syslog : True'
assert_line ' message_log_verbose : False'
assert_line ' netrc_file : None'
assert_line ' read_all : False'

@ -18,6 +18,7 @@ function teardown() { _default_teardown ; }
# dovecot (/usr/sbin/dovecot)
# fetchmail (/usr/bin/fetchmail)
# fail2ban-server (/usr/bin/python3 /usr/bin/fail2ban-server) - NOTE: python3 is due to the shebang
# getmail (/bin/bash /usr/local/bin/getmail-service.sh)
# mta-sts-daemon (/usr/bin/bin/python3 /usr/bin/mta-sts-daemon)
# postgrey (postgrey) - NOTE: This process command uses perl via shebang, but unlike python3 the context is missing
# postsrsd (/usr/sbin/postsrsd)
@ -41,6 +42,7 @@ ENV_PROCESS_LIST=(
dovecot
fail2ban-server
fetchmail
getmail
mta-sts-daemon
opendkim
opendmarc
@ -56,6 +58,7 @@ ENV_PROCESS_LIST=(
--env ENABLE_CLAMAV=0
--env ENABLE_FAIL2BAN=0
--env ENABLE_FETCHMAIL=0
--env ENABLE_GETMAIL=0
--env ENABLE_MTA_STS=0
--env ENABLE_OPENDKIM=0
--env ENABLE_OPENDMARC=0
@ -92,6 +95,7 @@ ENV_PROCESS_LIST=(
--env ENABLE_AMAVIS=1
--env ENABLE_FAIL2BAN=1
--env ENABLE_FETCHMAIL=1
--env ENABLE_GETMAIL=1
--env ENABLE_MTA_STS=1
--env ENABLE_OPENDKIM=1
--env ENABLE_OPENDMARC=1