Mailing List Archive

How to change field after mmnormalize parsing
Hi!

I'm using rsyslog to create gelf messages and send them to graylog server.

PHP-FPM Slow Logs has special multiline format.


[02-Jan-2019 14:04:28] [pool kosmetika_proff_ru] pid 18139
script_filename = /path/to/slow/script.php
[0x00007fe5ae417430] function1() /path/to/file1.php:301
[0x00007fe5ae417380] function2() /path/to/file2.php:160
[0x00007fe5ae4171f0] function3() /path/to/file3.php:164

So new line, two lines of diagnostic information and stacktrace.

I'm using this rulebase to parse it

version=2

rule=php-fpm,info,gelf:\\n[%date:char-to:]%] [pool
%pool_name:char-to:]%] pid %-:number%\\nscript_filename =
%script_filename:string-to{"extradata":"\\n"}%\\n%full_message:rest%

And this config to forward messages to graylog

template(name="gelf-ext" type="list") {
constant(value="{\"version\":\"1.1\",")
constant(value="\"host\":\"") property(name="hostname")
constant(value="\",\"short_message\":\"") property(name="msg"
format="json")
constant(value="\",\"timestamp\":\"") property(name="timegenerated"
dateformat="unixtimestamp")
constant(value="\",\"_application_name \":\"") property(name="app-name")
constant(value="\",\"level\":\"") property(name="syslogseverity")
constant(value="\",") property(name="$!all-json" position.from="2")
}

ruleset(name="graylog-ext") {
action(
type="omfwd"
Target="graylog"
Port="12201"
Protocol="tcp"
KeepAlive="on"
template="gelf-ext"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="graylog"
TCP_FrameDelimiter="0"
)
}

input(
type="imfile"
file="/var/www/*/logs/*log.slow"
startmsg.regex="^$"
tag="php-fpm"
readTimeout="10"
ruleset="php-fpm-slow"
)

ruleset(name="php-fpm-slow") {
action(type="mmnormalize" rulebase="/path/to/php_fpm_slow.rb")
call graylog-ext
}

Everything works fine, messages are forwarded to graylog, indexed to
elasticsearch etc, except one thing.

Stacktrace is one huge line with \\n symbol between lines.

Is there a way to alter full_message field in $!all-json variable before
send gelf message to graylog?

--

Best Regards, Ilya Rassadin.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: How to change field after mmnormalize parsing [ In reply to ]
look at the set and unset functions, they let you change any variable.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: How to change field after mmnormalize parsing [ In reply to ]
Thanks, it helped.

Just curious, is there a way to use mmnormalize and multiline imfile
with switched off escapeLF option directly?

On 02/01/2019 15:21, David Lang wrote:
> look at the set and unset functions, they let you change any variable.
>
> David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: How to change field after mmnormalize parsing [ In reply to ]
No, mmnormalize doesn't support multi-line rules

David Lang

On Fri, 4 Jan 2019, ???? ???????? via rsyslog wrote:

> Just curious, is there a way to use mmnormalize and multiline imfile with
> switched off escapeLF option directly?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: How to change field after mmnormalize parsing [ In reply to ]
For the record, my final working config to parse php-fpm slowlog into
gelf with rsyslog.

input(
? type="imfile"
? file="/var/www/*/logs/*log.slow"
? startmsg.regex="^$"
? tag="php-fpm"
? readTimeout="10"
? ruleset="php-fpm-slow"
? addMetadata="off"
)

template(name="gelf-ext" type="list") {
? constant(value="{\"version\":\"1.1\",")
? constant(value="\"host\":\"") property(name="hostname")
? constant(value="\",\"short_message\":\"") property(name="msg"
format="json")
? constant(value="\",\"timestamp\":") property(name="timegenerated"
dateformat="unixtimestamp")
? constant(value=",\"_application_name \":\"") property(name="app-name")
? constant(value="\",\"level\":\"") property(name="syslogseverity")
? constant(value="\",") property(name="$!all-json" position.from="2")
}


ruleset(name="php-fpm-slow") {
? action(type="mmnormalize"
rulebase="/etc/rsyslog.d/rules/php_fpm_slow.rb")
? set $!full_message = replace($!full_message, '\\n', "\n");
? action(
??? type="omfwd"
??? Target="graylog"
??? Port="12201"
??? Protocol="tcp"
??? KeepAlive="on"
??? template="gelf-ext"
??? StreamDriver="gtls"
??? StreamDriverMode="1"
??? StreamDriverAuthMode="x509/name"
??? StreamDriverPermittedPeers="graylog"
??? TCP_FrameDelimiter="0"
? )
}

rulebase file

version=2
rule=:\\n[%_date:char-to:]%]? [pool %_pool_name:char-to:]%] pid
%-:number%\\nscript_filename =
%_script_filename:string-to{"extradata":"\\n"}%\\n%full_message:rest%

It can be improved further:

1. clean up short message in template

2. set up timestamp from php-fpm slowlog

Many Thanks to David for helping.

On 04/01/2019 03:12, David Lang wrote:
> No, mmnormalize doesn't support multi-line rules
>
> David Lang
>
> On Fri, 4 Jan 2019, ???? ???????? via rsyslog wrote:
>
>> Just curious, is there a way to use mmnormalize and multiline imfile
>> with switched off escapeLF option directly?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.