Mailing List Archive

Exim 4.91 and eximstats
Hi,

it seems that the eximstats script is not up to date with the current logging of Exim 4.91. In my case it does not catch temporary rejects (for grey listing) of the following form

2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de> temporarily rejected after DATA: Please try again later

I am not sure if this reject should be counted and printed in the “Temp Rejects” line but have created a patch for the file src/src/eximstats.src b/src/src/eximstats.src (attached).

Maybe someone can have a look at my patch and tell me whether it needs more improvement. I can also create a pull request on GitHub if this is easier…

Thanks,
Paul
Re: Exim 4.91 and eximstats [ In reply to ]
On 19/04/18 15:59, Paul Hecker via Exim-users wrote:
> it seems that the eximstats script is not up to date with the current logging of Exim 4.91. In my case it does not catch temporary rejects (for grey listing) of the following form
>
> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de> temporarily rejected after DATA: Please try again later

I don't see the words "Please try" (with the capital) nor
"again later" in the source code. This might be your
configuration.

>
> I am not sure if this reject should be counted and printed in the “Temp Rejects” line but have created a patch for the file src/src/eximstats.src b/src/src/eximstats.src (attached).

".*?" - given the *, the ? is redundant. And then so is the
rest of the line.
>
> Maybe someone can have a look at my patch and tell me whether it needs more improvement. I can also create a pull request on GitHub if this is easier…

No pull reqs, please. A bug is fine, here is ok too.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 4.91 and eximstats [ In reply to ]
Hi Jeremy,


> On 19. Apr 2018, at 17:23, Jeremy Harris via Exim-users <exim-users@exim.org> wrote:
>
> On 19/04/18 15:59, Paul Hecker via Exim-users wrote:
>> it seems that the eximstats script is not up to date with the current logging of Exim 4.91. In my case it does not catch temporary rejects (for grey listing) of the following form
>>
>> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de> temporarily rejected after DATA: Please try again later
>
> I don't see the words "Please try" (with the capital) nor
> "again later" in the source code. This might be your
> configuration.

its the word “after” before “DATA” that breaks the regex. The rest of the logging (in my case “Please try again later”) does not matter.

>
>>
>> I am not sure if this reject should be counted and printed in the “Temp Rejects” line but have created a patch for the file src/src/eximstats.src b/src/src/eximstats.src (attached).
>
> ".*?" - given the *, the ? is redundant. And then so is the
> rest of the line.

My change to the old parsing was, that I now allow lowercase characters in the word(s) between “temporarily rejected” and “:” (see the if above my change). I did not want to break anything from the old parsing and therefore added another if with a tweaked regex.

>>
>> Maybe someone can have a look at my patch and tell me whether it needs more improvement. I can also create a pull request on GitHub if this is easier…
>
> No pull reqs, please. A bug is fine, here is ok too.

OK

> --
> Cheers,
> Jeremy

Thanks,
Paul


>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 4.91 and eximstats [ In reply to ]
On 19/04/18 16:38, Paul Hecker via Exim-users wrote:
> Hi Jeremy,
>
>
>> On 19. Apr 2018, at 17:23, Jeremy Harris via Exim-users <exim-users@exim.org> wrote:
>>
>> On 19/04/18 15:59, Paul Hecker via Exim-users wrote:
>>> it seems that the eximstats script is not up to date with the current logging of Exim 4.91. In my case it does not catch temporary rejects (for grey listing) of the following form
>>>
>>> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de> temporarily rejected after DATA: Please try again later
>>
>> I don't see the words "Please try" (with the capital) nor
>> "again later" in the source code. This might be your
>> configuration.
>
> its the word “after” before “DATA” that breaks the regex. The rest of the logging (in my case “Please try again later”) does not matter.
>


Hmm. Do you use cutthrough routing? There's a "after" in that path.

The larger picture is: log-parsing is fragile. The Exim log is
intended for humans, not for automatic parsing. The eximstats
script, or any other parsing, is pretty much certain to get
out-of-date; the logging is not regarded as a stable interface.

You'd be better-off writing stats into a database with explicit
ACL actions.
--
Cheers,
Jeremy



--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 4.91 and eximstats [ In reply to ]
On Thu, 2018-04-19 at 16:23 +0100, Jeremy Harris via Exim-users wrote:
> ".*?" - given the *, the ? is redundant.  And then so is the
> rest of the line.

Apologies if I'm missing some other subtlety, but ".*?" is a non-greedy
match - i.e. it will consume the fewest number of characters possible,
and in particular in this case won't consume the remainder of the line.

Regards,

Adam

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim 4.91 and eximstats [ In reply to ]
Hi,

> On 19. Apr 2018, at 19:09, Jeremy Harris via Exim-users <exim-users@exim.org> wrote:
>
> On 19/04/18 16:38, Paul Hecker via Exim-users wrote:
>> Hi Jeremy,
>>
>>
>>> On 19. Apr 2018, at 17:23, Jeremy Harris via Exim-users <exim-users@exim.org> wrote:
>>>
>>> On 19/04/18 15:59, Paul Hecker via Exim-users wrote:
>>>> it seems that the eximstats script is not up to date with the current logging of Exim 4.91. In my case it does not catch temporary rejects (for grey listing) of the following form
>>>>
>>>> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de> temporarily rejected after DATA: Please try again later
>>>
>>> I don't see the words "Please try" (with the capital) nor
>>> "again later" in the source code. This might be your
>>> configuration.
>>
>> its the word “after” before “DATA” that breaks the regex. The rest of the logging (in my case “Please try again later”) does not matter.
>>
>
>
> Hmm. Do you use cutthrough routing? There's a "after" in that path.

no, I do not use cutthrough routing. But I have found the source, where the “after” is added to the log. It's in smtp_handle_acl_fail() (smtp_in.c:3175).

After some more investigation, the regex should catch the following log lines:

2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de> temporarily rejected during MIME ACL checks: Please try again later
2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de> temporarily rejected DATA: Please try again later
2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de> temporarily rejected after DATA: Please try again later
2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de> temporarily rejected after DATA PRDR: Please try again later

I have improved the code so that all these cases are caught. See the attached patch.

>
> The larger picture is: log-parsing is fragile. The Exim log is
> intended for humans, not for automatic parsing. The eximstats
> script, or any other parsing, is pretty much certain to get
> out-of-date; the logging is not regarded as a stable interface.
>
> You'd be better-off writing stats into a database with explicit
> ACL actions.

I see. I just use the script for some minor monitoring (ACL and DB would be a bit over the top for my case).

You can decide whether you want to apply my patch or not. At least it is working for me again ;-)

> --
> Cheers,
> Jeremy

Thanks,
Paul



>
>
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users <https://lists.exim.org/mailman/listinfo/exim-users>
> ## Exim details at http://www.exim.org/ <http://www.exim.org/>
> ## Please use the Wiki with this list - http://wiki.exim.org/ <http://wiki.exim.org/>
Re: Exim 4.91 and eximstats [ In reply to ]
Hi,

sorry about the confusion, this is the right patch:




Regards,
Paul



> On 23. Apr 2018, at 11:58, Paul Hecker <paul@iwascoding.com> wrote:
>
> Hi,
>
>> On 19. Apr 2018, at 19:09, Jeremy Harris via Exim-users <exim-users@exim.org <mailto:exim-users@exim.org>> wrote:
>>
>> On 19/04/18 16:38, Paul Hecker via Exim-users wrote:
>>> Hi Jeremy,
>>>
>>>
>>>> On 19. Apr 2018, at 17:23, Jeremy Harris via Exim-users <exim-users@exim.org <mailto:exim-users@exim.org>> wrote:
>>>>
>>>> On 19/04/18 15:59, Paul Hecker via Exim-users wrote:
>>>>> it seems that the eximstats script is not up to date with the current logging of Exim 4.91. In my case it does not catch temporary rejects (for grey listing) of the following form
>>>>>
>>>>> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected after DATA: Please try again later
>>>>
>>>> I don't see the words "Please try" (with the capital) nor
>>>> "again later" in the source code. This might be your
>>>> configuration.
>>>
>>> its the word “after” before “DATA” that breaks the regex. The rest of the logging (in my case “Please try again later”) does not matter.
>>>
>>
>>
>> Hmm. Do you use cutthrough routing? There's a "after" in that path.
>
> no, I do not use cutthrough routing. But I have found the source, where the “after” is added to the log. It's in smtp_handle_acl_fail() (smtp_in.c:3175).
>
> After some more investigation, the regex should catch the following log lines:
>
> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected during MIME ACL checks: Please try again later
> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected DATA: Please try again later
> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected after DATA: Please try again later
> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected after DATA PRDR: Please try again later
>
> I have improved the code so that all these cases are caught. See the attached patch.
>
>>
>> The larger picture is: log-parsing is fragile. The Exim log is
>> intended for humans, not for automatic parsing. The eximstats
>> script, or any other parsing, is pretty much certain to get
>> out-of-date; the logging is not regarded as a stable interface.
>>
>> You'd be better-off writing stats into a database with explicit
>> ACL actions.
>
> I see. I just use the script for some minor monitoring (ACL and DB would be a bit over the top for my case).
>
> You can decide whether you want to apply my patch or not. At least it is working for me again ;-)
>
>> --
>> Cheers,
>> Jeremy
>
> Thanks,
> Paul
>
> <eximstats.patch>
>
>>
>>
>>
>> --
>> ## List details at https://lists.exim.org/mailman/listinfo/exim-users <https://lists.exim.org/mailman/listinfo/exim-users>
>> ## Exim details at http://www.exim.org/ <http://www.exim.org/>
>> ## Please use the Wiki with this list - http://wiki.exim.org/ <http://wiki.exim.org/>
Re: Exim 4.91 and eximstats [ In reply to ]
Hi,

as the list strips attachments (thanks Kurt) here is a link to the patch:

https://gist.github.com/lluuaapp/a939d33abe0a773175dd1bd1e1fd672e <https://gist.github.com/lluuaapp/a939d33abe0a773175dd1bd1e1fd672e>

Thanks!



> On 23. Apr 2018, at 12:04, Paul Hecker <paul@iwascoding.com> wrote:
>
> Hi,
>
> sorry about the confusion, this is the right patch:
>
>
> <eximstats.patch>
>
> Regards,
> Paul
>
>
>
>> On 23. Apr 2018, at 11:58, Paul Hecker <paul@iwascoding.com <mailto:paul@iwascoding.com>> wrote:
>>
>> Hi,
>>
>>> On 19. Apr 2018, at 19:09, Jeremy Harris via Exim-users <exim-users@exim.org <mailto:exim-users@exim.org>> wrote:
>>>
>>> On 19/04/18 16:38, Paul Hecker via Exim-users wrote:
>>>> Hi Jeremy,
>>>>
>>>>
>>>>> On 19. Apr 2018, at 17:23, Jeremy Harris via Exim-users <exim-users@exim.org <mailto:exim-users@exim.org>> wrote:
>>>>>
>>>>> On 19/04/18 15:59, Paul Hecker via Exim-users wrote:
>>>>>> it seems that the eximstats script is not up to date with the current logging of Exim 4.91. In my case it does not catch temporary rejects (for grey listing) of the following form
>>>>>>
>>>>>> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected after DATA: Please try again later
>>>>>
>>>>> I don't see the words "Please try" (with the capital) nor
>>>>> "again later" in the source code. This might be your
>>>>> configuration.
>>>>
>>>> its the word “after” before “DATA” that breaks the regex. The rest of the logging (in my case “Please try again later”) does not matter.
>>>>
>>>
>>>
>>> Hmm. Do you use cutthrough routing? There's a "after" in that path.
>>
>> no, I do not use cutthrough routing. But I have found the source, where the “after” is added to the log. It's in smtp_handle_acl_fail() (smtp_in.c:3175).
>>
>> After some more investigation, the regex should catch the following log lines:
>>
>> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected during MIME ACL checks: Please try again later
>> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected DATA: Please try again later
>> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected after DATA: Please try again later
>> 2018-04-19 14:38:21 1f9AYC-0000P7-Ev H=from.mail.example.com <http://from.mail.example.com/> [123.128.22.119] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<blah@lol.de <mailto:blah@lol.de>> temporarily rejected after DATA PRDR: Please try again later
>>
>> I have improved the code so that all these cases are caught. See the attached patch.
>>
>>>
>>> The larger picture is: log-parsing is fragile. The Exim log is
>>> intended for humans, not for automatic parsing. The eximstats
>>> script, or any other parsing, is pretty much certain to get
>>> out-of-date; the logging is not regarded as a stable interface.
>>>
>>> You'd be better-off writing stats into a database with explicit
>>> ACL actions.
>>
>> I see. I just use the script for some minor monitoring (ACL and DB would be a bit over the top for my case).
>>
>> You can decide whether you want to apply my patch or not. At least it is working for me again ;-)
>>
>>> --
>>> Cheers,
>>> Jeremy
>>
>> Thanks,
>> Paul
>>
>> <eximstats.patch>
>>
>>>
>>>
>>>
>>> --
>>> ## List details at https://lists.exim.org/mailman/listinfo/exim-users <https://lists.exim.org/mailman/listinfo/exim-users>
>>> ## Exim details at http://www.exim.org/ <http://www.exim.org/>
>>> ## Please use the Wiki with this list - http://wiki.exim.org/ <http://wiki.exim.org/>