Mailing List Archive

json properties
Sorry, some more questions, this time relating to JSON properties.

1. I found these:

jsonmesg
$!
$!all-json
$!all-json-plain

https://www.rsyslog.com/doc/master/configuration/properties.html
documents only the first of these.
https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmjsonparse.html
documents only the second.  I found the third from some blog postings,
and grepping rsyslog-doc I only find one mention of it - tangentially,
in an config example for imjournal
<https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html>.

Just to confirm what I *think* is going on:

$! is only the content which was parsed by mmjsonparse (otherwise empty
or does not exist)

$!all-json and $!all-json-plain are $! re-serialized as a JSON string

jsonmesg is all of the internal syslog properties (including $! if it
exists) serialized as a JSON string.

Is that right?


2. My second question is around preserving message metadata, and whether
JSON is the right way to do this.  Consider the following setup:

            imudp         omkafka imkafka          ,-->...
[message] ------> rsyslog ----------> [kafka] ---------> rsyslog --->...
`-->...

The idea is that kafka becomes the canonical store of messages, which
can be picked up and processed by multiple systems, including additional
instances of rsyslog.  However, I don't want to lose message metadata
like the source IP of the UDP packets (property "fromhost-ip").  How
should this best be done?

For omkafka, I could make a template using property "jsonmesg", which I
think will preserve everything.

However, I don't see the best way to get this back in on the second
rsyslog instance.  Should I use pmnull and mmjsonparse?  As far as I can
see, mmjsonparse puts everything in the "$!" property; and it says here
<https://www.rsyslog.com/doc/v8-stable/rainerscript/variable_property_types.html>
that "Only message json (CEE/Lumberjack) properties can be modified by
the set, unset and reset statements, not any other message property."

In other words, I can't see how to copy them back into the normal
rsyslog metadata.  I guess it's not a problem, as long as I remember
everywhere that the property is in the "wrong" place, e.g. use
"$!.rawmsg" instead of "rawmsg"

Am I missing a trick here?  Or is there a potential use case for a
"pmjsonmesg" parser module?

Thanks again,

Brian.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: json properties [ In reply to ]
On 16/09/2019 16:23, Brian Candler wrote:
> imudp         omkafka               imkafka          ,-->...
> [message] ------> rsyslog ----------> [kafka] ---------> rsyslog --->...
> `-->...

Supplementary question.

Suppose I were to write to kafka* just the rawmsg plus minimal captured
metadata, e.g.

{"fromhost-ip":"192.0.2.1", "rawmsg":"<134>Sep 16 15:33:53 root: Hello
world"}

Then I want to pick that up using rsyslog/imkafka, extract the supplied
metadata, and then continue to parse rawmsg with the normal parsing
chain as if it were a newly arrived message.  Is that possible?

Cheers,

Brian.

*actually redis. There's no imhiredis, but I was prepared to prototype
something with improg.  Except that improg is missing from Ubuntu's
packaging of rsyslog :-(

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: json properties [ In reply to ]
On Mon, 16 Sep 2019, Brian Candler via rsyslog wrote:

> 1. I found these:
>
> jsonmesg
> $!
> $!all-json
> $!all-json-plain
>
> https://www.rsyslog.com/doc/master/configuration/properties.html
> documents only the first of these.
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmjsonparse.html
> documents only the second.  I found the third from some blog postings,
> and grepping rsyslog-doc I only find one mention of it - tangentially,
> in an config example for imjournal
> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html>.
>
> Just to confirm what I *think* is going on:
>
> $! is only the content which was parsed by mmjsonparse (otherwise empty
> or does not exist)
>
> $!all-json and $!all-json-plain are $! re-serialized as a JSON string

I believe that $! and $!all-json and $!all-json-plain should be identical

mmjsonparse is one of the things that can add value in $! but not all

RSYSLOG_DebugFormat is your friend here. it's output shows the $! , $. and $\
namespaces (the last three lines), lay around with set, mmjsonparse,
mmnromalize, etc and see how they can populate these namespaces.

$! is just the shortcut way of saying 'everything in this namespace as a single
json string'

> jsonmesg is all of the internal syslog properties (including $! if it
> exists) serialized as a JSON string.

I don't think it's going to be _all_ of them (there is quite a bit of
duplication)

> Is that right?
>
>
> 2. My second question is around preserving message metadata, and whether
> JSON is the right way to do this.  Consider the following setup:
>
>             imudp         omkafka imkafka          ,-->...
> [message] ------> rsyslog ----------> [kafka] ---------> rsyslog --->...
> `-->...
>
> The idea is that kafka becomes the canonical store of messages, which
> can be picked up and processed by multiple systems, including additional
> instances of rsyslog.  However, I don't want to lose message metadata
> like the source IP of the UDP packets (property "fromhost-ip").  How
> should this best be done?

What I typically do is reserve a chunk of namespace $!trusted and use the set
command to populate it with metadata (frequently in multiple sets, such as
origin, relay, core), then when I send the message, I use a template that has $!
instead of $msg (note, you need to set something in the $!namespace to contain
the message, I typically use $!msg)

> For omkafka, I could make a template using property "jsonmesg", which I
> think will preserve everything.
>
> However, I don't see the best way to get this back in on the second
> rsyslog instance.  Should I use pmnull and mmjsonparse?  As far as I can
> see, mmjsonparse puts everything in the "$!" property; and it says here
> <https://www.rsyslog.com/doc/v8-stable/rainerscript/variable_property_types.html>
> that "Only message json (CEE/Lumberjack) properties can be modified by
> the set, unset and reset statements, not any other message property."
>
> In other words, I can't see how to copy them back into the normal
> rsyslog metadata.  I guess it's not a problem, as long as I remember
> everywhere that the property is in the "wrong" place, e.g. use
> "$!.rawmsg" instead of "rawmsg"

you cannot change the default properties, you just have to use the $! versions.

I would suggest using mmnormalize instead of mmjsonparse, mmjsonparse is not any
better, and is much more limiting.


> Am I missing a trick here?  Or is there a potential use case for a
> "pmjsonmesg" parser module?

you can use mmnormalize at that point, but I think that you are going to be
better off just accepting that after the round trip, everything will be in $!
variables (I put them there as early in the pipeline as possible and just always
use the $! namespace)

David Lang

> Thanks again,
>
> Brian.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: json properties [ In reply to ]
On Mon, 16 Sep 2019, Brian Candler via rsyslog wrote:

> On 16/09/2019 16:23, Brian Candler wrote:
>> imudp         omkafka               imkafka          ,-->...
>> [message] ------> rsyslog ----------> [kafka] ---------> rsyslog --->...
>> `-->...
>
> Supplementary question.
>
> Suppose I were to write to kafka* just the rawmsg plus minimal captured
> metadata, e.g.
>
> {"fromhost-ip":"192.0.2.1", "rawmsg":"<134>Sep 16 15:33:53 root: Hello
> world"}
>
> Then I want to pick that up using rsyslog/imkafka, extract the supplied
> metadata, and then continue to parse rawmsg with the normal parsing
> chain as if it were a newly arrived message.  Is that possible?

you could do it with a pm configuration, but I think you end up working harder
than you need to. I also prefer to parse as early as is practical, to spread the
parsing over more cores/machine, making it's cost invisible.

David Lang

> Cheers,
>
> Brian.
>
> *actually redis. There's no imhiredis, but I was prepared to prototype
> something with improg.  Except that improg is missing from Ubuntu's
> packaging of rsyslog :-(
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: json properties [ In reply to ]
On 17/09/2019 00:42, David Lang wrote:
>> Just to confirm what I *think* is going on:
>>
>> $! is only the content which was parsed by mmjsonparse (otherwise
>> empty or does not exist)
>>
>> $!all-json and $!all-json-plain are $! re-serialized as a JSON string
>
> I believe that $! and $!all-json and $!all-json-plain should be identical

I'd better test it then.

template(name="test-json" type="list") {
  constant(value="<")
  property(name="$!")
  constant(value="|")
  property(name="$!all-json")
  constant(value="|")
  property(name="$!all-json-plain")
  constant(value=">\n")
}

local1.* action(type="omfile"
  file="/var/tmp/test.log"
  template="test-json"
)

With this and nothing in $! I get:

<|{}|{}>

After adding

module(load="mmjsonparse")
local1.* action(type="mmjsonparse" cookie="")

then I get:

<{ "wibble": "bibble" }|{ "wibble": "bibble" }|{"wibble":"bibble"}>

So:

- if $! is empty, $! as a property gives empty string but $!all-json and
$!all-json-plain give {}

- if $! is not empty, $! as a property gives the same as $!all-json;
$!all-json-plain is similar but without whitespace padding

> I would suggest using mmnormalize instead of mmjsonparse, mmjsonparse
is not any better, and is much more limiting.

You can parse a json line using mmnormalize? How?

Cheers,

Brian.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: json properties [ In reply to ]
On Tue, 17 Sep 2019, Brian Candler wrote:

> - if $! is empty, $! as a property gives empty string but $!all-json and
> $!all-json-plain give {}
>
> - if $! is not empty, $! as a property gives the same as $!all-json;
> $!all-json-plain is similar but without whitespace padding

so not exactly the same but close. could you update the docs to make this clear
for everyone?

>> I would suggest using mmnormalize instead of mmjsonparse, mmjsonparse is
> not any better, and is much more limiting.
>
> You can parse a json line using mmnormalize? How?

there is a json data type

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: json properties [ In reply to ]
On 9/17/19 7:48 AM, Brian Candler via rsyslog wrote:
> On 17/09/2019 00:42, David Lang wrote:
>>> Just to confirm what I *think* is going on:
>>>
>>> $! is only the content which was parsed by mmjsonparse (otherwise
>>> empty or does not exist)
>>>
>>> $!all-json and $!all-json-plain are $! re-serialized as a JSON string
>>
>> I believe that $! and $!all-json and $!all-json-plain should be identical
>
> I'd better test it then.
>
> template(name="test-json" type="list") {
>   constant(value="<")
>   property(name="$!")
>   constant(value="|")
>   property(name="$!all-json")
>   constant(value="|")
>   property(name="$!all-json-plain")
>   constant(value=">\n")
> }
>
> local1.* action(type="omfile"
>   file="/var/tmp/test.log"
>   template="test-json"
> )
>
> With this and nothing in $! I get:
>
> <|{}|{}>
>
> After adding
>
> module(load="mmjsonparse")
> local1.* action(type="mmjsonparse" cookie="")
>
> then I get:
>
> <{ "wibble": "bibble" }|{ "wibble": "bibble" }|{"wibble":"bibble"}>
>
> So:
>
> - if $! is empty, $! as a property gives empty string but $!all-json and
> $!all-json-plain give {}
>
> - if $! is not empty, $! as a property gives the same as $!all-json;
> $!all-json-plain is similar but without whitespace padding
>
> > I would suggest using mmnormalize instead of mmjsonparse, mmjsonparse
> is not any better, and is much more limiting.
>
> You can parse a json line using mmnormalize? How?

https://github.com/rsyslog/liblognorm/blob/master/doc/configuration.rst#json

>
> Cheers,
>
> Brian.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: json properties [ In reply to ]
On 17/09/2019 14:55, David Lang wrote:
> On Tue, 17 Sep 2019, Brian Candler wrote:
>
>> - if $! is empty, $! as a property gives empty string but $!all-json
>> and $!all-json-plain give {}
>>
>> - if $! is not empty, $! as a property gives the same as $!all-json;
>> $!all-json-plain is similar but without whitespace padding
>
> so not exactly the same but close. could you update the docs to make
> this clear for everyone?

$!all-json isn't mentioned in the docs at all.  Where would you say it
belongs?  Perhaps at
https://www.rsyslog.com/doc/v8-stable/configuration/properties.html

There it says that system properties start with $ but are not related to
the message.

Maybe it belongs wherever $/ is documented.  But I can't find that
anywhere either.

AFAICS, $!all-json was originally added here:

commit 471f07f45a382c29f74e1c676bd081c3b304d7db
Author: Rainer Gerhards <rgerhards@adiscon.com>
Date:   Wed Dec 1 10:19:50 2010 +0100

    milestone: template supports CEE output via %$!all-json%

diff --git a/runtime/msg.c b/runtime/msg.c
index 5318cb751..479b2ba31 100644
--- a/runtime/msg.c
+++ b/runtime/msg.c
@@ -445,6 +445,10 @@ rsRetVal propNameToID(cstr_t *pCSPropName, propid_t
*pPropID)
                *pPropID = PROP_SYS_MINUTE;
        } else if(!strcmp((char*) pName, "$myhostname")) {
                *pPropID = PROP_SYS_MYHOSTNAME;
+       } else if(!strcmp((char*) pName, "$!all-json")) {
+               *pPropID = PROP_CEE_ALL_JSON;
+       } else if(!strncmp((char*) pName, "$!", 2)) {
+               *pPropID = PROP_CEE;

... but I'm still unclear of the intended difference between PROP_CEE
and PROP_CEE_ALL_JSON.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.