Mailing List Archive

exim4 only queues mails sent by systemd service
(This is repost of
https://lists.freedesktop.org/archives/systemd-devel/2018-September/041393.html
as Lennart Poettering suggest this)

It is something strange with sending mails from systemd system service:
assume we have service file /etc/systemd/system/mailtest.service:

--8<---------------cut here---------------start------------->8---
[Unit]
Description="Test maili"
[Service]
#User=kjonca
NoNewPrivileges=false
Type=oneshot
ExecStart=-zsh -c 'echo xxx|mail news'
ExecStart=-zsh -c 'echo xxx|mutt -F /dev/null -s subject -e \'set copy=no\' kjonca'
--8<---------------cut here---------------end--------------->8---

When I call
sudo systemctl start mailtest.service
Two messages are put in exim queue, but not deliveried immediately.
Why? What am I missing?
Morevoer this service run as "--user" service works as expected - mails
are delivered at once.
Does systemd interfere with processes (suid/sgid? file access limitations?)

Moreover;
when I put strace in front of mutt - mails were delivered at once

Beside this
- mails from cron
- mails from fetchmail
- mails from "--user" services
are delivered at once - as expected.

KJ


--
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
kondensator - kondensatorych - kondensatoremu
(odmiana s?owa "kondensator" wed?ug MS Word 6.0)

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
On 24/09/2018 14:25, Kamil Jo?ca via Exim-users wrote:
> assume we have service file /etc/systemd/system/mailtest.service:
>
> --8<---------------cut here---------------start------------->8---
> [Unit]
> Description="Test maili"
> [Service]
> #User=kjonca
> NoNewPrivileges=false
> Type=oneshot
> ExecStart=-zsh -c 'echo xxx|mail news'
> ExecStart=-zsh -c 'echo xxx|mutt -F /dev/null -s subject -e \'set copy=no\' kjonca'
> --8<---------------cut here---------------end--------------->8---
>
> When I call
> sudo systemctl start mailtest.service
> Two messages are put in exim queue, but not deliveried immediately.
> Why? What am I missing?
> Morevoer this service run as "--user" service works as expected - mails
> are delivered at once.
> Does systemd interfere with processes (suid/sgid? file access limitations?)

Possible, but rather difficult for us to guess.
If you could call exim directly, rather than via mutt, you could
add debug flags to the commandline. Then assuming that systemd
collects stderr, we might be able to trace what happens.

http://exim.org/exim-html-current/doc/html/spec_html/ch-the_exim_command_line.html#SECID39

--
Cheers,
Jeremy



--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
Jeremy Harris via Exim-users <exim-users@exim.org> writes:

> On 24/09/2018 14:25, Kamil Jo?ca via Exim-users wrote:
>> assume we have service file /etc/systemd/system/mailtest.service:
>>
>> --8<---------------cut here---------------start------------->8---
>> [Unit]
>> Description="Test maili"
>> [Service]
>> #User=kjonca
>> NoNewPrivileges=false
>> Type=oneshot
>> ExecStart=-zsh -c 'echo xxx|mail news'
>> ExecStart=-zsh -c 'echo xxx|mutt -F /dev/null -s subject -e \'set copy=no\' kjonca'
>> --8<---------------cut here---------------end--------------->8---
>>
>> When I call
>> sudo systemctl start mailtest.service
>> Two messages are put in exim queue, but not deliveried immediately.
>> Why? What am I missing?
>> Morevoer this service run as "--user" service works as expected - mails
>> are delivered at once.
>> Does systemd interfere with processes (suid/sgid? file access limitations?)
[...my mail...]
>
> Possible, but rather difficult for us to guess.
> If you could call exim directly, rather than via mutt, you could
> add debug flags to the commandline. Then assuming that systemd
> collects stderr, we might be able to trace what happens.

After discussion on systemd list we have conclusions:
(https://lists.debian.org/debian-user/2018/10/msg00054.html
and thread
https://lists.freedesktop.org/archives/systemd-devel/2018-September/041395.html
)

1. exim forks and creates background process to deliver mail
2. systemd, after main process exits, kill all remaining proceses, so
...
3. background exim process are killed during delivery, and message remains in
queue.

I think this is important for services sending mails (for example
logrotate with mail option)

There are solutions/workarounds:
1. migrate to postfix :)
2. set KillMode=none in service file.
3. use other init (not systemd)

KJ

--
http://wolnelektury.pl/wesprzyj/teraz/
Sendmail may be safely run set-user-id to root.
-- Eric Allman, "Sendmail Installation Guide"

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
Howdy

Firstly - please subscribe to the list (or post from your subscribed address) so we don’t have to keep allowing your emails out of the mod queue.

On 5 Oct 2018, at 19:11, Kamil Jo?ca via Exim-users <exim-users@exim.org> wrote:
> After discussion on systemd list we have conclusions:
> (https://lists.debian.org/debian-user/2018/10/msg00054.html
> and thread
> https://lists.freedesktop.org/archives/systemd-devel/2018-September/041395.html
> )
>
> 1. exim forks and creates background process to deliver mail
> 2. systemd, after main process exits, kill all remaining proceses, so
> ...
> 3. background exim process are killed during delivery, and message remains in
> queue.

…how often do you run the queue?

It doesn’t address the scenario you raise, but if you’re running it frequently enough then the death of a delivery subprocess from within systemd shouldn’t matter one jot.

That said: setting KillMode=none seems like an easy fix to me.

Graeme
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
Graeme Fowler via Exim-users <exim-users@exim.org> writes:

> Howdy
>
> Firstly - please subscribe to the list (or post from your subscribed address) so we don’t have to keep allowing your emails out of the mod queue.

Sorry, I have read list via gmane. Now I am subscribed.

>
> On 5 Oct 2018, at 19:11, Kamil Jo?ca via Exim-users <exim-users@exim.org> wrote:
>> After discussion on systemd list we have conclusions:
>> (https://lists.debian.org/debian-user/2018/10/msg00054.html
>> and thread
>> https://lists.freedesktop.org/archives/systemd-devel/2018-September/041395.html
>> )
>>
>> 1. exim forks and creates background process to deliver mail
>> 2. systemd, after main process exits, kill all remaining proceses, so
>> ...
>> 3. background exim process are killed during delivery, and message remains in
>> queue.
>
> …how often do you run the queue?
30min

>
> It doesn’t address the scenario you raise, but if you’re running it
> frequently enough then the death of a delivery subprocess from within
> systemd shouldn’t matter one jot.

Not quite. It is possible (and I observed it) another scenario:

1. exim starts deliver message and put in queue (yes?)
2. background process starts deliver message and deliver it, but ...
3. backround process is killed just before it can report delivery
success, so message remains in queue
4. queue process deliver message for second time ...

We end up with duplicate ...
>
> That said: setting KillMode=none seems like an easy fix to me.

Yes, although it should be done for every service which might send mails.

>
> Graeme


KJ

--
http://wolnelektury.pl/wesprzyj/teraz/
I'd probably settle for a vampire if he were romantic enough.
Couldn't be any worse than some of the relationships I've had.
-- Brenda Starr

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
On 05/10/2018 19:11, Kamil Jo?ca via Exim-users wrote:
> 1. exim forks and creates background process to deliver mail
> 2. systemd, after main process exits, kill all remaining proceses, so
> ...
> 3. background exim process are killed during delivery, and message remains in
> queue.
>
> I think this is important for services sending mails (for example
> logrotate with mail option)
>
> There are solutions/workarounds:
> 1. migrate to postfix :)
> 2. set KillMode=none in service file.
> 3. use other init (not systemd)

It does seem somewhat... arrogant of systemd to assume that
when a process it has started terminates, any of its children
that might happen to be doing useful, desired work should
be killed. If that's the default. At least, as you
note, there's a way of telling it not to.

Another option is (if calling Exim directly, or in
a way that you can modify Exim's command line) the -odf
option; locally-generated messages have the delivery
process waited for by the toplevel process.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
Jeremy Harris via Exim-users <exim-users@exim.org> writes:

[..]
> It does seem somewhat... arrogant of systemd to assume that
> when a process it has started terminates, any of its children
Ahem. :)

https://github.com/systemd/systemd/issues/10299

KJ
--
http://wolnelektury.pl/wesprzyj/teraz/
P-K4

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
On 16/10/2018 12:44, Kamil Jo?ca via Exim-users wrote:
> Jeremy Harris via Exim-users <exim-users@exim.org> writes:
>
> [..]
>> It does seem somewhat... arrogant of systemd to assume that
>> when a process it has started terminates, any of its children
> Ahem. :)
>
> https://github.com/systemd/systemd/issues/10299

I think we need slightly better justification for
the "might have been ok design in 1995, but not now"
assertion.
--
Cheers,
Jeremy


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
Jeremy Harris via Exim-users <exim-users@exim.org> writes:

> On 16/10/2018 12:44, Kamil Jo?ca via Exim-users wrote:
>> Jeremy Harris via Exim-users <exim-users@exim.org> writes:
>>
>> [..]
>>> It does seem somewhat... arrogant of systemd to assume that
>>> when a process it has started terminates, any of its children
>> Ahem. :)
>>
>> https://github.com/systemd/systemd/issues/10299
>
> I think we need slightly better justification for
> the "might have been ok design in 1995, but not now"
> assertion.

I do not know if it is clear: this is not my words, neither I agreed
with them (please note, that I opened this issue on github.)

I agreed that systemd should allow exim to work on current rules. But I
don know how can I argue to Lennart Poettering to change his mind.
KJ


--
http://wolnelektury.pl/wesprzyj/teraz/
Are you mentally here at Pizza Hut??

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
On 16 Oct 2018, at 14:56, Kamil Jo?ca via Exim-users <exim-users@exim.org> wrote:
> I do not know if it is clear: this is not my words, neither I agreed
> with them (please note, that I opened this issue on github.)

...and you have a workaround in systemd via the KillMode switch, or the various timeout options. Or you could use Exim in queue-only mode with a frequent (say 30s) queue runner so it doesn't attempt immediate delivery when the local message is generated.

> I agreed that systemd should allow exim to work on current rules. But I
> don know how can I argue to Lennart Poettering to change his mind.

You can't :)

What you've shown us is (in my opinion) an incredibly niche case which has a variety of options available already to fix it. I have *never* heard of a systemd unit being used to send an email itself - I'm not saying it doesn't happen, but it seems a rather esoteric thing to do (to me). Of course, several hundred people will now tell me that they do it too!

I dare say you could probably do something with the runtime configuration too, or with the configuration of the 'mail' command - which you set in /etc/mail.rc, ~/.mailrc or similar:

set sendmail=/usr/sbin/exim -odq

That works for me - message injected but no immediate delivery and no forking of a child subprocess. Message delivered by next queue run.

If you set it in /etc/mail.rc, that works for all users.

Do those help?

Graeme
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
On 2018-10-16 15:40, Graeme Fowler via Exim-users wrote:

> > I agreed that systemd should allow exim to work on current rules. But I
> > don know how can I argue to Lennart Poettering to change his mind.
>
> You can't :)
>
> What you've shown us is (in my opinion) an incredibly niche case which
> has a variety of options available already to fix it. I have *never*
> heard of a systemd unit being used to send an email itself - I'm not
> saying it doesn't happen, but it seems a rather esoteric thing to do
> (to me). Of course, several hundred people will now tell me that they
> do it too!
>
> I dare say you could probably do something with the runtime
> configuration too, or with the configuration of the 'mail' command -
> which you set in /etc/mail.rc, ~/.mailrc or similar

I think this is a misunderstanding both on the part of Lennart and on
the part of Graeme.

The example systemd unit shown by the OP was just _a test_. The real
use case with a problem (or at least a potential problem) is not a human
user sending mail with mutt or /bin/mail or whatever, but a service
sending mail as a means of communicating status. Sometimes such
services have no run time configuration item to change how mail is
sent. I think this is the case with Vixie cron as used in Debian, even
today.

One could argue that the bug is in the service, and I'd tend to agree.

--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
On 16 Oct 2018, at 18:03, Ian Zimmerman via Exim-users <exim-users@exim.org> wrote:
> I think this is a misunderstanding both on the part of Lennart and on
> the part of Graeme.

I’m not so sure that it is; I was addressing the very specific example given.

> The example systemd unit shown by the OP was just _a test_. The real
> use case with a problem (or at least a potential problem) is not a human
> user sending mail with mutt or /bin/mail or whatever, but a service
> sending mail as a means of communicating status. Sometimes such
> services have no run time configuration item to change how mail is
> sent. I think this is the case with Vixie cron as used in Debian, even
> today.

Services spawned from systemd that are long-running don’t suffer this issue. It’s the one-shot units that do, and only those, so far as I can see due to the way systemd reaps them. It simply doesn’t account for a process which may have spawned other background processes when it runs.

> One could argue that the bug is in the service, and I'd tend to agree.

The bug is in the interaction between the choice of init (systemd), the options in the unit file, the configuration of mail (or rmail, or mailx, whichever is being used), the Exim command line, and the Exim configuration. There’s not really a single entity which causes the issue.

There are ways to work around the issue, as I described previously. Making Exim not do instantaneous delivery is but one of them.

Graeme
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
Graeme Fowler via Exim-users <exim-users@exim.org> writes:

> On 16 Oct 2018, at 18:03, Ian Zimmerman via Exim-users <exim-users@exim.org> wrote:
>> I think this is a misunderstanding both on the part of Lennart and on
>> the part of Graeme.
>
> I’m not so sure that it is; I was addressing the very specific example given.

Well. I was alway teach to deliver "simplest" not working example. So I
did. And this example can be seen as unimportant.

The real use case are "oneshot" services started by timers[1]. So far
people still use cron. But when somebody try to migrate its crontab to
*.timer service(s),

KJ
[1] I was hit by this behavior when debian migrate logrotate crontab entry
to logrotate.{timer,service}
--
http://wolnelektury.pl/wesprzyj/teraz/
"?ycie jest wrzodem na tkance wszech?wiata."
Jerzy Vetulani, 21.1.1936 — 7.4.2017

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
On 17 Oct 2018, at 12:38, Kamil Jo?ca via Exim-users <exim-users@exim.org> wrote:
> Well. I was alway teach to deliver "simplest" not working example. So I
> did. And this example can be seen as unimportant.
>
> The real use case are "oneshot" services started by timers[1]. So far
> people still use cron. But when somebody try to migrate its crontab to
> *.timer service(s),

…and the workarounds/configuration options I’ve suggested will work for pretty well all of these instances, however complex. You just have to choose the one that works for you.

The quickest win for us would be to document the use of queue_only and frequent queue runners, I reckon. Changing the fork delivery model is not likely to be something that happens quickly at all!

Graeme
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: exim4 only queues mails sent by systemd service [ In reply to ]
Graeme Fowler via Exim-users <exim-users@exim.org> writes:

> On 17 Oct 2018, at 12:38, Kamil Jo?ca via Exim-users <exim-users@exim.org> wrote:
[...]
>

> frequent queue runners, I reckon. Changing the fork delivery model is
> not likely to be something that happens quickly at all!

I never suggest that. :) L. Poettering did.

KJ

--
http://wolnelektury.pl/wesprzyj/teraz/
Cheese -- milk's leap toward immortality.
-- Clifton Fadiman, "Any Number Can Play"

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/