Mailing List Archive

Parse & use date
Hi,

tl;dr
How can I parse and use a date?

Long story:
An application is writing a log file (/var/log/application.log).

Each message starts with a date format like

[%d/%b/%Y:%H:%M:%S %z]

which resolves to

[21/Jan/2019:12:20:41 +0100]

for example.

I am reading that file using imfile module (and then the rsyslog client
will send that message to the centralized rsyslog server doing all the
processing but this part doesn't matter for my question).

For processing, I am using mmnormalize and my rule will start like

version=2
rule=:[%timestamp:char-to:]%]

Once the message was successfully parsed I want to write it to a 'dynamic'
target (could be just a dynamic file, e.g. /logs/%YEAR/%MONTH...). In my
case I have to specify a dynamic elasticsearch index, e.g. I would do
something like

template(name="es-index-name" type="string" string="myindex-%$YEAR%.%$MONTH%.%$DAY%")
template(name="es-json" type="list") { ... }

action(type="omelasticsearch" template="es-json" searchIndex="es-index-name" dynSearchIndex="on")

However, $YEAR, $MONTH, $DAY will refer to a default message property,
e.g. a value reflecting either the date when imfile read and created a
syslog message from application log or when the centralized rsyslog
server received or processed that message. But I need to access
year/month/day value *from parsed* $!timestamp value (=the value from the
originating log file).

How can I do that?

property(name="$!timestamp" dateformat="year")

would be too easy ;-)

I am using rsyslog-8.40 version.


--
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Parse & use date [ In reply to ]
If I understand the need correctly, you could chop your $!timestamp (or
$msg) into pieces with the substring function, and then use the values
contained in the pieces.  It's a way of "parsing" it yourself.

Regards,



On 1/22/19 2:44 PM, Thomas Deutschmann via rsyslog wrote:
> Hi,
>
> tl;dr
> How can I parse and use a date?
>
> Long story:
> An application is writing a log file (/var/log/application.log).
>
> Each message starts with a date format like
>
> [%d/%b/%Y:%H:%M:%S %z]
>
> which resolves to
>
> [21/Jan/2019:12:20:41 +0100]
>
> for example.
>
> I am reading that file using imfile module (and then the rsyslog client
> will send that message to the centralized rsyslog server doing all the
> processing but this part doesn't matter for my question).
>
> For processing, I am using mmnormalize and my rule will start like
>
> version=2
> rule=:[%timestamp:char-to:]%]
>
> Once the message was successfully parsed I want to write it to a 'dynamic'
> target (could be just a dynamic file, e.g. /logs/%YEAR/%MONTH...). In my
> case I have to specify a dynamic elasticsearch index, e.g. I would do
> something like
>
> template(name="es-index-name" type="string" string="myindex-%$YEAR%.%$MONTH%.%$DAY%")
> template(name="es-json" type="list") { ... }
>
> action(type="omelasticsearch" template="es-json" searchIndex="es-index-name" dynSearchIndex="on")
>
> However, $YEAR, $MONTH, $DAY will refer to a default message property,
> e.g. a value reflecting either the date when imfile read and created a
> syslog message from application log or when the centralized rsyslog
> server received or processed that message. But I need to access
> year/month/day value *from parsed* $!timestamp value (=the value from the
> originating log file).
>
> How can I do that?
>
> property(name="$!timestamp" dateformat="year")
>
> would be too easy ;-)
>
> I am using rsyslog-8.40 version.
>
>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Parse & use date [ In reply to ]
Hi,

On 2019-01-22 23:37, John Chivian wrote:
> If I understand the need correctly, you could chop your $!timestamp (or
> $msg) into pieces with the substring function, and then use the values
> contained in the pieces.  It's a way of "parsing" it yourself.

This could be a workaround, yes. How would I translate %b value, i.e.
the abbreviated month name into its numeric representation? 12 if
clauses or is there a better way?


--
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Parse & use date [ In reply to ]
Use a lookup table - https://www.rsyslog.com/doc/v8-stable/rainerscript/lookup_tables.html

here are some examples of using a lookup table to map priority numeric values to log level string values, and log level values to their canonical representation:

https://github.com/openshift/cluster-logging-operator/blob/master/files/rsyslog/normalize_level.json
https://github.com/openshift/cluster-logging-operator/blob/master/files/rsyslog/prio_to_level.json

This is how you declare and use them

https://github.com/openshift/cluster-logging-operator/blob/master/files/rsyslog/65-viaq-formatting.conf#L2

https://github.com/openshift/cluster-logging-operator/blob/master/files/rsyslog/65-viaq-formatting.conf#L191
https://github.com/openshift/cluster-logging-operator/blob/master/files/rsyslog/65-viaq-formatting.conf#L236



On 1/22/19 4:42 PM, Thomas Deutschmann via rsyslog wrote:
> Hi,
>
> On 2019-01-22 23:37, John Chivian wrote:
>> If I understand the need correctly, you could chop your $!timestamp (or
>> $msg) into pieces with the substring function, and then use the values
>> contained in the pieces.  It's a way of "parsing" it yourself.
> This could be a workaround, yes. How would I translate %b value, i.e.
> the abbreviated month name into its numeric representation? 12 if
> clauses or is there a better way?
>
>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Parse & use date [ In reply to ]
On Wed, 23 Jan 2019, Thomas Deutschmann via rsyslog wrote:

> This could be a workaround, yes. How would I translate %b value, i.e.
> the abbreviated month name into its numeric representation? 12 if
> clauses or is there a better way?

Rsyslog does not yet have good date manipulation capabilities. We have started
it with parse_time() and format_time(), but they are currently very limited.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Parse & use date [ In reply to ]
Hi,

thank you, the links to the openshift repository were very helpful!

I am now doing something like

> lookup_table(name="normalize_month" file="/etc/rsyslog.d/normalize_month.json")
> action(
> type="mmnormalize"
> rulebase="/etc/rsyslog.d/my-application.rulebase"
> )
>
> if ($parsesuccess == "OK") then {
> # if parsed, we extracted $!year, $!month, $!day ...
> set $.lcmonth = tolower($!month);
> set $.normmonth = lookup("normalize_month", $.lcmonth);
> if $.normmonth == "unknown" then {
> stop
> } else {
> reset $!month = $.normmonth;
> }
> unset $.lcmonth;
> unset $.normmonth;
>
> # ...
> }

...and the lookup table looks like

> { "version" : 1,
> "nomatch" : "unknown",
> "type" : "string",
> "table" : [.
> {"index": "jan", "value": "01"},
> {"index": "feb", "value": "02"},
> {"index": "mar", "value": "03"},
> {"index": "apr", "value": "04"},
> {"index": "may", "value": "05"},
> {"index": "jun", "value": "06"},
> {"index": "jul", "value": "07"},
> {"index": "aug", "value": "08"},
> {"index": "sep", "value": "09"},
> {"index": "oct", "value": "10"},
> {"index": "nov", "value": "11"},
> {"index": "dec", "value": "12"},
> ]
> }


--
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Parse & use date [ In reply to ]
On 2019-01-23 09:33, David Lang wrote:
> Rsyslog does not yet have good date manipulation capabilities. We have
> started it with parse_time() and format_time(), but they are currently
> very limited.

Yeah, extracting each value and creating a new date format like

> template(name="iso8601date" type="list") {
> property(name="$!year")
> constant(value="-")
> property(name="$!month")
> constant(value="-")
> property(name="$!day")
> constant(value="T")
> property(name="$!hours")
> constant(value=":")
> property(name="$!minutes")
> constant(value=":")
> property(name="$!seconds")
> property(name="$!timezone")
> }
>
> if ($parsesuccess == "OK") then {
> # ...values are now extracted
>
> set $!timestamp = exec_template("iso8601date");
>
> # ...
> }

works but...

However I realized that even if I would receive messages with
a date value supported by any date-* parser in liblognorm, I
would be unable to transform the matched value just via

> property(name="$!timestmap" dateFormat="rfc3339")

later in rsyslog because the extracted value would just be string,
not a date object, right?


--
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Parse & use date [ In reply to ]
On Thu, 24 Jan 2019, Thomas Deutschmann via rsyslog wrote:

>
> However I realized that even if I would receive messages with
> a date value supported by any date-* parser in liblognorm, I
> would be unable to transform the matched value just via
>
>> property(name="$!timestmap" dateFormat="rfc3339")
>
> later in rsyslog because the extracted value would just be string,
> not a date object, right?

correct, but parse_time() is intended to convert a string into a unix timestamp,
and then format_time() would be able to convert it to rfc3339 or other format.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.