mirror/ docker-mailserver
1
1
Fork 0
mirror of https://github.com/docker-mailserver/docker-mailserver synced 2024-05-24 13:26:03 +02:00
Code Issues
2,423 Commits 3 Branches 26 Tags 42 MiB
Commit Graph

135 Commits

Author SHA1 Message Date
Casper c2d0b748b2
Change 'while' style (#3365) 2023-05-26 01:39:39 +02:00
Casper 37ca0f9ba9
Change 'function' style (#3364) 2023-05-26 01:01:41 +02:00
Casper cf74127f78
change if style (#3361) 2023-05-24 09:06:59 +02:00
LucidityCrash 7af7546d88
feature: adding `getmail` as an alternative to `fetchmail` (#2803) 
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
 2023-05-23 17:25:08 +02:00
Georg Lauterbach bba72daedf
scripts: add DKIM helper script for Rspamd (#3286) 
Co-authored-by: Casper <casperklein@users.noreply.github.com>
 2023-05-03 08:30:49 +02:00
Casper 423188176f
fail2ban: add 'log' command (#3299) 
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
 2023-05-03 00:13:44 +02:00
Georg Lauterbach 1e20e7c332
Image registry and setup update (#3233) 
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
 2023-04-10 11:37:25 +02:00
Georg Lauterbach dfc2b39308
scripts: housekeeping & cleanup setup (2/2) (#3123) 2023-02-28 10:25:23 +01:00
Georg Lauterbach f35b60042f
scripts: split `setup-stack.sh` (#3115) 2023-02-26 11:42:14 +01:00
Georg Lauterbach f5d325bdc1
fix `restrict-access` (#3067) 2023-02-18 16:52:42 +01:00
Brennan Kinney a7e6439a39
fix: Workaround `postconf` write settling logic (#2998) 
* fix: Workaround `postconf` write settle logic

After updating `main.cf`, to avoid an enforced delay from reading the config by postfix tools, we can ensure the modified time is at least 2 seconds in the past as a workaround. This should be ok with our usage AFAIK.

Shaves off 2+ seconds roughly off each container startup, reduces roughly 2+ minutes off tests.

* chore: Only modify `mtime` if less than 2 seconds ago

- Slight improvement by avoiding unnecessary writes with a conditional check on the util method.
- Can more comfortably call this during `postfix reload` in the change detection cycle now.
- Identified other tests that'd benefit from this, created a helper method to call instead of copy/paste.
- The `setup email restrict` command also did a modification and reload. Added util method here too.

* tests(fix): `mail_smtponly.bats` should wait for Postfix

- `postfix reload` fails if the service is not ready yet.
- `service postfix reload` and `/etc/init.d/postfix reload` presumably wait until it is ready? (as these work regardless)

* chore: Review feedback - Move reload method into utilities
 2023-01-13 10:10:58 +13:00
Casper dcf34fd63b
Fix several typos (#2993) 2023-01-11 13:31:21 +13:00
Casper 69aef2e94f
Bugfix: './setup.sh email list' does not display aliases correctly (#2881) 2022-11-04 22:39:05 +01:00
Casper 897d84f0f9
scripts: Improve error handling, when parameters are missing (#2854) 2022-10-29 11:04:35 +02:00
Casper c4df9c116b
Follow up for #2849 (#2853) 2022-10-23 12:57:07 +02:00
Casper edb3fb1880
setup: fix unbound variable error (#2849) 2022-10-22 14:43:17 +13:00
Casper a930aeb18a
Remove unusual space from shebang line (#2834) 2022-10-17 10:40:09 +02:00
Casper c92a67f91e
Fix typo (#2830) 2022-10-15 11:32:25 +13:00
Brennan Kinney 75a75bfae6
chore: `addmailuser` - Remove delaying completion until `/var/mail` is ready (#2729) 
## Quick Summary

Resolves a `TODO` task with `addmailuser`.

## Overview

The main change is adding three new methods in `common.bash`, which replace the completion delay in `addmailuser` / `setup email add` command.

Other than that:

- I swapped `sh -c 'addmailuser ...'` to `setup email add ...`.
- Improved three tests in `setup-cli.bats` for `setup email add|update|del` (_logic remains effectively the same still_).
- Rewrote the `TODO` comment for `setup-cli.bats` test on `setup email del` to better clarify the concern, but the test itself was no longer affected due to changes prior to this PR, so I enabled the commented out assertion.
- Removed unnecessary waits. The two `skip` tests in `test/tests.bats` could be enabled again after this PR.
- Additional fixes to tests were made during the PR (see discussion comments for details), resolving race conditions.

Individual commit messages of the PR provide additional details if helpful.

---

## Relevant commit messages

* chore: Remove creation delay in `addmailuser`

This was apparently only for supporting tests that need to wait on account creation being ready to test against.

As per the removed inline docs, it should be fine to remove once tests are updated to work correctly without it.

* tests(feat): Add two new common helper methods

`wait_until_account_maildir_exists()` provides the same logic `addmailuser` command was carrying, to wait upon the account dir creation in `/var/mail`.

As this was specifically to support tests, it makes more sense as a test method.

`add_mail_account_then_wait_until_ready()` was added to handle the common pattern of creating account and waiting on it. An internal assert will ensure the account was successfully created first during the test before attempting to wait.

* tests(feat): Add common helper for waiting on change event to be processed

The current helper is more complicated for no real benefit, it only detects when a change is made that would trigger a change event in the `changedetector` service. Our usage of this in tests however is only interested in waiting out the completion of the change event.

Remove unnecessary change event waits. These waits should not be necessary if handled correctly.

* tests: `addmailuser` to `add_mail_account_then_wait_until_ready mail()`

This helper method is used where appropriate.
- A password is not relevant (optional).
- We need to wait on the creation on the account (Dovecot and `/var/mail` directory).

* tests: `setup-cli` revise `add`, `update`, `del` tests

The delete test was failing as the `/var/mail` directory did not yet exist.

There is now a proper delay imposed in the `add` test now shares the same account for both `update` and `del` tests resolving that failure.

Additionally tests use better asserts where appropriate and the wait + sleep logic in `add` has been improved (now takes 10 seconds to complete, approx half the time than before).

The `del` test TODO while not technically addressed is no longer relevant due to the tests being switched to `-c` option (there is a separate `no container` test file, but it doesn't provide a `del` test).

* tests(fix): Ensure Postfix is reachable after waiting on ClamAV

There is not much reason to check before waiting on ClamAV.

It is more helpful to debug failures from `nc` mail send commands if we know that nothing went wrong inbetween the ClamAV wait time.

Additionally added an assertion which should provide more information if this part of the test setup fails again.

* tests(fix): Move health check to the top

This test is a bit fragile. It relies on defaults for the healthcheck with intervals of 30 seconds.

If the check occurs while Postfix is down due a change event from earlier tests and the healthcheck kicks in at that point, then if there is not enough time to refresh the health status from `unhealthy`, the test will fail with a false-positive as Postfix is actually working and up again..

* tests(fix): Wait on directory to be removed

Workaround that tries not to introduce heavier delays by waiting on a full change event to complete in the previous `email update` if possible.

There is a chance that the account has the folder deleted, but restored from an active change event (for password update, then the account delete).
 2022-08-23 10:15:06 +12:00
Casper 11f8cc7ee9
Simplify fail2ban output (#2703) 2022-07-29 18:14:35 +02:00
Brennan Kinney 57aeb6db2a
refactor: CLI commands for database management (#2654) 
See the associated PR for more detailed commentary on specific changes.

### Commands refactored:
- User (**All:** add / list / update / del + _dovecot-master variants_)
- Quota (**All:** set / del)
- Virtual Alias (**All:** add / list /del)
- Relay (**All:** add-relayhost / add-sasl / exclude-domain)

### Overall changes involve:
- **Fairly common structure:**
  - `_main` method at the top provides an overview of logical steps:
    - After all methods are declared beneath it (_and imported from the new `helpers/database/db.sh`_), the `_main` is called at the bottom of the file.
    - `delmailuser` additionally processes option support for `-y` prior to calling `_main`.
  - `__usage` is now consistent with each of these commands, along with the `help` command.
  - Most logic delegated to new helper scripts. Some duplicate content remains on the basis that it's low-risk to maintenance and avoids less hassle to jump between files to check a single line, usually this is arg validation.
  - Error handling should be more consistent, along with var names (_no more `USER`/`EMAIL`/`FULL_EMAIL` to refer to the same expected value_).
- **Three new management scripts** (in `helpers/database/manage/`) using a common structure for managing changes to their respective "Database" config file.
  - `postfix-accounts.sh` unified not only add and update commands, but also all the dovecot-master versions, a single password call for all 4 of them, with a 5th consumer of the password prompt from the relay command `addsaslpassword`.
  - These scripts delegate actual writes to `helpers/database/db.sh` which provides a common API to support the changes made.
     - This is more verbose/complex vs the current inline operations each command currently has, as it provides generic support instead of slightly different variations being maintained, along with handling some edge cases that existed and would lead to bugs (notably substring matches).
     - Centralizing changes here seems wiser than scattered about. I've tried to make it easy to grok, hopefully it's not worse than the current situation.
     - List operations were kept in their respective commands, `db.sh` is only really managing writes. I didn't see a nice way for removing the code duplication for list commands as the duplication was fairly minimal, especially for `listalias` and `listdovecotmasteruser` which were quite simple in their differences in the loop body.
     - `listmailuser` and `delmailuser` also retain methods exclusive to respective commands, I wasn't sure if there was any benefit to move those, but they were refactored.
 2022-07-29 12:10:23 +12:00
Brennan Kinney e3cc627e18
refactor: Share a common helper (vhost builder) for sourcing domains (#2620) 
* chore: Split vhost helper method and use filepath vars

- Helpers `accounts.sh` and `aliases.sh` can move their vhost code into this helper.
- They share duplicate code with `bin/open-dkim` which will also leverage this vhost helper going forward.

* chore: Sync vhost generation logic into helper

- Chunky commit, but mostly copy/paste of logic into a common method.
- `bin/open-dkim` additionally wrapped relevant logic in a function call and revised inline docs.

* chore: Include LDAP vhost support

- Revises notes for LDAP vhost support.
- This now ensures LDAP users get vhost rebuilt to match the startup script for when change detection support is enabled.
- `bin/open-dkim` will additionally be able to support the default `DOMAINNAME` var (set via `helpers/dns.sh`) for LDAP users instead of requiring them to provide one explicitly.

* chore(`bin/open-dkim`): Ensure `DOMAINNAME` is properly set

- This will ensure LDAP users insert the same `DOMAINNAME` value as used during container startup.
- The container itself should panic at startup (during `helpers/dns.sh`) if this isn't configured correctly already, thus it should not introduce any breaking change to users of this utility?

* chore: Set the 2nd value as blank `_`

Line is split by a delimiter such as white-space (or via IFS: `|`), the blank `_` var is to indicate we're not interested in that value, but still leverage how `read -r` works, instead of splitting the var ourselves first thing.

* chore: Remove shellcheck disable lines

No longer applicable with the switch to `_`
 2022-06-10 10:57:10 +12:00
Casper 8e0fd306b7
Check if fail2ban is running (#2626) 2022-06-07 11:42:06 +00:00
Brennan Kinney 54904aa02c
chore(housekeeping): Normalize how config files filter out unwanted lines (#2619) 
* chore(`aliases.sh`): Filepath to local var `DATABASE_VIRTUAL`

* chore(`accounts.sh`): Filepath to local var `DATABASE_ACCOUNTS`

* chore(`accounts.sh`): Filepath to local var `DATABASE_VIRTUAL`

* chore(`accounts.sh`): Filepath to local var `DATABASE_DOVECOT_MASTERS`

* chore(`bin/open-dkim`): Filepaths to local vars (accounts,virtual,vhost)

* chore(`relay.sh`): Filepath to local var `DATABASE_SASL_PASSWD`

* chore: Rename method

Prior PR feedback suggested a better helper method name.

* chore: Normalize filtering config lines as input for iterating

* chore: Remove `_is_comment` helper method

No longer serving a purpose with more appropriate filter method for pre-processing the entire config file.
 2022-06-07 01:07:30 +12:00
Brennan Kinney 1f3e672018
fix(listmailuser): Don't parse comments when reading config (#2617) 
* fix(listmailuser): Don't parse comments

Avoids passing comments to `dovecot_quota_to_hr()` which fails to handle it and would throws errors.

* chore: Move config filter method to `helpers/utils.sh`
 2022-06-06 11:02:52 +12:00
Brennan Kinney 1bd8ef3976
chore: Sync dovecot-master-user commands to use new style syntax (#2611) 
Adopt the new colour vars from `helpers/log.sh` like the other supported commands have.
 2022-06-05 00:05:33 +00:00
Brennan Kinney c862e1451d
chore(housekeeping): Create `helpers/change-detection.sh` (#2610) 
* chore: Extract change-detection method to it's own helper

This doesn't really belong in `helpers/ssl.sh`. Moving to it's own helper script.

* chore: Co-locate related change-detection method from container startup

It seems relevant to migrate the related support during startup for the change detection feature into this helper.

I opted to move the call from `start-mailserver.sh` into the `_setup` call at the end for a more explicit/visible location.

* chore: Move `CHKSUM_FILE` into `helpers/change-detection.sh`

It belongs there, not in `helpers/index.sh`.

* chore: Revise inline documentation

* tests(fix): Ensure correct functionality

Presently `test/test_helper.bats` is using it's own  `CHKSUM_FILE` instead of sourcing the var for the filepath.

`test_helper/common.bash` was calling a method to check for changes, but this helper may not correctly detect letsencrypt related changes as these are not ENV rely on, but global vars handled by `helpers/dns.sh`, so that should be run first like it is for `check-for-changes.sh`.

* tests(chore): Use `CHKSUM_FILE` var from helper

* chore: `addmailuser` should use `CHKSUM_FILE` var

* chore: Update `check-for-changes.sh` log message with correct path
 2022-06-05 11:59:54 +12:00
Casper 372e3fe35e
Use color variables instead of escape codes (#2565) 2022-05-10 17:50:33 +02:00
Andrew Cornford 2f3cbfc144
feat: Support for Dovecot master accounts (#2535) 
Dovecot master accounts can now be configured in DMS via `setup.sh`.
A master account is useful for administration purposes, or to perform mailbox backups of every user account over IMAP.
Upstream Docs: https://doc.dovecot.org/configuration_manual/authentication/master_users/

Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
 2022-05-08 00:28:32 +02:00
Casper 18acd7860b
Run sedfile tests in container (#2569) 2022-05-05 12:58:00 +02:00
Casper 628e902233
Remove unnecessary quotes from command substitutions (#2561) 2022-05-05 10:28:38 +02:00
Brennan Kinney 182b71d33f
chore: Drop `setup.sh` DATABASE fallback ENV (#2556) 
* chore: Remove `DATABASE` fallback ENV

This was introduced without any mention or need, thus removing until a real use-case requires it.

* chore: Remove `USER_DATABASE` fallback ENV

Likewise, nothing requires this to be customizable.

* chore: Consistently use single quote strings
 2022-04-20 12:29:28 +12:00
Casper de61d42e68
Add ban feature to fail2ban script (#2538) 2022-04-19 08:44:51 +00:00
Casper ee0c088b1f
setup.sh/setup: show usage when no argument is given (#2540) 2022-04-18 15:28:12 +02:00
Georg Lauterbach a9305a073f
firewall: replace `iptables` with `nftables` (#2505) 
* first adjustments to use Fail2Ban with nftables

* replace `iptables` -> `nftables` and adjust tests

nftables lists IPs a bit differently , so the order was adjusted for the
tests to be more flexible.

* line correction in mailserver.env

* change from `.conf` -> `.local` and remove redundant config

* revert HEREDOC to `echo`

Co-authored-by: Casper <casperklein@users.noreply.github.com>
 2022-04-05 15:13:59 +02:00
Georg Lauterbach a1ecd781c8
scripts: introduce `_log` to `sedfile` (#2507) 2022-04-02 15:52:30 +02:00
Georg Lauterbach a54d774587
scripts: refactored `daemon-stack.sh` (#2496) 
* refactored `daemon-stack.sh`

A new method was introduced to uniformaly start daemons and log output
accordingly. The methods for daemon start were renamed (plural ->
singular), therefore the adjustments in `start-mailserver.sh`.

* cleaned Fetchmail setup from `daemon-stack.sh`

Not sure why, but the Fetchmail setup was somehow happening in
`daemon-stack.sh` - this is not supposed to be the case. I relocated the
setup into `setup-stack.sh`, where it belong.

* delete old, unnecessary script in `target/bin/`

These are unused leftovers from the last commit, that relocated the
setup of Fetchmail into `setup.stack.sh`.

* corrected changedetector function name

* Apply suggestions from code review

* adjusted `debug-fetchmail` script

It is absolutely fine to source `setup-stack.sh` because sourcing the
script does not execute a single function (by desing of the script).
This way, we retain functionality.

* praise be ShellCheck

* added `log.sh` to `debug-fetchmail` as a dependency

* final cleanup

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
 2022-03-27 09:43:39 +02:00
Georg Lauterbach b9dbec3276
scripts: refactored scripts located under `target/bin/` (#2500) 
* refactored scripts located under `target/bin/`

The scripts under `target/bin/` now use the new log and I replaced some
`""` with `''` on the way. The functionality stays the same, this mostly
style and log.

* corrected fail2ban (script and tests)

* corrected OpenDKIM log output in tests

* reverted (some) changes to `sedfile`

Moreover, a few messages for BATS were streamlined and a regression in
the linting script reverted.

* apple PR feedback

* improve log output from `fail2ban` script

The new output has a single, clear message with the '[  ERROR  ]  '
prefix, and then output that explains the error afterwards. This is
coherent with the logging style which should be used while providing
more information than just a single line about IPTables not functioning.

* simplified `setquota` script

* consistently named the `__usage` function

Before, scripts located under `target/bin/` were using `usage` or
`__usage`. Now, they're using `__usage` as they should.

* improved `sedfile`

With `sedfile`, we cannot use the helper functions in a nice way because
it is used early in the Dockerfile at a stage where the helper scripts
are not yet copied. The script has been adjusted to be canonical with
all the other scripts under `target/bin/`.

* fixed tests

* removed `__usage` from places where it does not belong

`__usage` is to be used on wrong user input, not on other failures as
well. This was fixed in `delquota` and `setquota`.

* apply PR review feedback
 2022-03-26 09:30:09 +01:00
Georg Lauterbach b7bf40ac21
renamed function `_errex` -> `_exit_with_error` (#2497) 2022-03-21 15:01:07 +01:00
Georg Lauterbach 24031ae365
scripts: new log (#2493) 
* added new `_log` function

With `_log`, the `_notify` method wa rendered obsolete. `_notify` was
not completely removed due to test failures in `check-for-changes.sh`.

The new `_log` function properly uses log levels such as `trace`,
`debug`, `info`, `warn` and `error`. It provides a cleaner solution
and renders `DMS_DEBUG` obsolete too (as only `_notify` depends on it).

* converted all helper script to new `_log` function

* converted all startup stacks to new `log` function

* `start-mailserver.sh` now uses new `_log` function

* final test and misc small script adjustments

* updated documentation
 2022-03-21 07:07:52 +01:00
NorseGaud 1ab8ea96b2
acme_extract -> acme_extract.py + F_BIN to avoid .py 2022-03-06 22:17:29 -05:00
Casper 1911f9a4b4
setup.sh: Remove obsolete 'debug inspect' command from usage text (#2454) 2022-03-03 11:38:28 +00:00
Georg Lauterbach b61dfe1e24
refactoring: split helper functions into smaller scripts (#2420) 2022-02-21 11:56:57 +01:00
Georg Lauterbach ed834d50d3
fixed non-number-argument in listmailuser (#2382) 2022-01-27 14:57:40 +01:00
Casper 29c2d975ec
Fix harmless startup errors (#2357) 2022-01-08 00:25:09 +01:00
Casper d46e094280
Remove quotes to meet style guidelines (#2330) 
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
 2021-12-21 17:01:40 +01:00
Georg Lauterbach 99cc9fec2a
Updated ShellCheck to `0.8.0` and Hadolint to `2.8.0` (#2329) 
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
 2021-12-19 11:56:22 +01:00
Robert Schumann b11e5ffd1a
fetchmailrc split: ignore commented lines (#2305) 
Co-authored-by: Robert Schumann <robert@schumann.link>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
 2021-11-29 15:24:38 +01:00
Brennan Kinney e807631a76
refactor: acme.json extraction (#2274) 
Split into scoped commits with messages if further details are needed, view those via the associated PR :)

**Commit Summary:**

**`check-for-changes.sh`**

- Prevent `SSL_DOMAIN` silently skipping when value has wildcard prefix `*.` (_at least this was known as a bugfix when originally committed in linked PR_).
- Improved inlined docs for maintainers.
- Additional logging for debugging.

**`helper-functions.sh:_extract_certs_from_acme`**:

- Fail if the input arg (_`$CERT_DOMAIN`, aka the FQDN_) provided for extraction is empty.
- Use `$CERT_DOMAIN` in place of `$HOSTNAME` and `$1` for a consistent value (_previously could mismatch, eg with `SSL_DOMAIN` defined_).
- The conditional is now only for handling extraction failure (_key or cert value is missing from extraction_).
- Log an actual warning or success (debug) based on outcome.
- Don't use `SSL_DOMAIN` with wildcard value for the `mkdir` letsencrypt directory name (_wildcard prefix `*.` is first stripped instead_).

**`acme_extract`** (_new python utility for `acme.json` handling_):

- Extracted out into a python script that can be treated as a utility in the `$PATH` like other helper scripts. It can now be used and optionally tested directly instead of via `helper-functions.sh`.
-Made compatible with Python 3, as Python 2 is EOL and no longer in newer versions of Debian.
 2021-11-04 09:28:40 +13:00
Casper b117cd1423
Improve VIRUSMAILS_DELETE_DELAY usage (#2281) 2021-11-01 22:13:14 +01:00