Mailing List Archive

GROK/regex-Patterns for mainlog
Hi there,

I was wondering if somebody has any experience with extracting Patterns
from the mainlog for example for graylog or elastic.
The only thing I found were these patterns
https://github.com/vjeantet/grok/blob/master/patterns/exim
but I'd like to have a more extensive, more complete list. For example
vjeantet's GROK misses the "message fakereject" documented here Ch52.5 (
https://www.exim.org/exim-html-current/doc/html/spec_html/ch-log_files.html)
The basic idea would be to have one or a few giant GROK patterns to match
every possible field in any log line.

I have a few problems with doing this myself:
1. Different kinds of log lines:
If you exigrep some log lines there are those grouped together under one
message id and those which come before message delivery and thus cannot be
grouped to a specific message. No idea how many different "syntaxes" there
are for such lines.
2. Doubly used fields:
At the moment I only know of one field which is used for different purposed
on different log lines. "T" for "Topic" or "Transport" inbound and
outbound, which you can identify by the log line flags (Ch52.5). You can
find most Fields in Cha52.13 but it is incomplete because at least the
second "T" is not explained. So there could be more undocumented fields.
3. Complex Fields:
If you look at the H-Field it contains several pieces of information:
Hostname, HELO, IP and Port; separated by spaces and a colon. Sometimes you
can have every one of those Fields, sometimes there is no Hostname because
the PTR and A records doe not match(at least I think this is the reason),
sometimes there is no HELO because the server sent his hostname as HELO
(which is great) The linked GROK patterns do account for this correctly.
But if I do not know of other fields that could have a similar behavior.
4. Different keywords and extra messages:
Most fields have a clear "F=" or "H=" but there are also keywords like
"from" and "for"

I can only do so much to generate the most complete log lines. It would be
helpful to have a set of every field in the right order that might or might
not be displayed in a particular type of log line. Or at least know what
different kind of log lines there are would be helpful, there are at least
inbound ( <=, (= ), outbound (**, =>, *>, >>, etc.), completed, rejects
before delivering/submitting a mail.

Does anybody know of a complete pattern for mainlog (or even reject and
panic log as well)? Or are there maybe already GROK/Regex patterns I did
not stumble upon that satisfy my requirements or at least come close?

Cheers and have a nice weekend,
Christian Kugler
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: GROK/regex-Patterns for mainlog [ In reply to ]
On 13/04/18 19:52, Christian K via Exim-users wrote:
> Does anybody know of a complete pattern for mainlog (or even reject and
> panic log as well)? Or are there maybe already GROK/Regex patterns I did
> not stumble upon that satisfy my requirements or at least come close?

Any such list will necessarily become out-of-date as Exim advances. We
do not regard the logging as a stable interface; nor is it intended to
be suited for machine parsing (though see bug 2142, which aims in that
direction). You would be better off deliberately writing the particular
details you want, under the control of expansions in ACLs and
transports, to your required output channel.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: GROK/regex-Patterns for mainlog [ In reply to ]
On 13 Apr 2018, at 19:52, Christian K via Exim-users <exim-users@exim.org> wrote:
> I was wondering if somebody has any experience with extracting Patterns
> from the mainlog for example for graylog or elastic.

Logstash has a set of basic patterns created by yours truly:

https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/exim

So far as I am aware, they are included in the current release of logstash.

Graeme
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/