Mailing List Archive

Filtering syslogs on server and writing to specific files
Dear Experts,

I have implemented syslog server on CentOS and filtering all *NATing *events
from Juniper nodes into a single file using below template (*rsyslog.conf*)
-

*##For redirecting the NAT/FW logs to specific file/directory*
*#$template TmplcpFW, "/var/log/NIPFW/%HOSTNAME%.log"*
*if ($hostname == ["CP1FW1", "CP1FW2", "CP2FW1", "CP2FW2","CP1CGNAT"]) then
?TmplcpFW*
*& ~*

Now I would like to further segregate these logs based on IP-pools used (*e.g-
100.70.0.0*) as this is the only unique data available in all event logs.

Below is the sample logs and info available in various events (*for one of
the ip-pool*)

*<150>1 2018-12-20T16:07:00.482369+01:00 2018-12-20 15 - - - 06:59:
CP1CGNAT_O{OFR_NAT}[jservices-nat]: JSERVICES_NAT_RULE_MATCH: proto 6 (TCP)
application: any, xe-1/0/1.1718:100.70.0.2:59794 -> 109.32.8.15:80
<http://109.32.8.15:80>, Match NAT rule-set: (null), rule: O_NAT_XX, term:
t1*
*<150>1 2018-12-20T16:07:00.482369+01:00 2018-12-20 15 - - - 06:59:
CP1CGNAT_O{OFR_NAT}JSERVICES_SESSION_OPEN: application:none, xe-1/0/1.1718
100.70.0.2:59794 [55.93.69.53:26620 <http://55.93.69.53:26620>] ->
109.32.8.15:80 <http://109.32.8.15:80> (TCP) *
*<150>1 2018-12-20T16:07:11.091313+01:00 2018-12-20 15 - - - 07:10:
CP1CGNAT_O{OFR_NAT}JSERVICES_SESSION_CLOSE: application:none, xe-1/0/1.1718
100.70.0.2:59778 [55.93.69.60:40136 <http://55.93.69.60:40136>] ->
109.32.8.15:80 <http://109.32.8.15:80> (TCP) *

Can you please suggest/help to achieve this in rsyslog?


Thanks a lot for your great help.

--

Regards

Sarjit
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Filtering syslogs on server and writing to specific files [ In reply to ]
The first thing to do is to parse the logs apart using mmnormalize, then you can
either use startswith matches, or lookup_table to decide hat to put where.

but until you parse the logs apart so that you have the IP address you want to
filter by in a variable of it's own, it's too messy to deal with.

David Lang

On Tue, 8 Jan 2019, sarjit yadav via rsyslog wrote:

> Date: Tue, 8 Jan 2019 16:44:25 +0530
> From: sarjit yadav via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: sarjit yadav <sarjit.ymca@gmail.com>
> Subject: [rsyslog] Filtering syslogs on server and writing to specific files
>
> Dear Experts,
>
> I have implemented syslog server on CentOS and filtering all *NATing *events
> from Juniper nodes into a single file using below template (*rsyslog.conf*)
> -
>
> *##For redirecting the NAT/FW logs to specific file/directory*
> *#$template TmplcpFW, "/var/log/NIPFW/%HOSTNAME%.log"*
> *if ($hostname == ["CP1FW1", "CP1FW2", "CP2FW1", "CP2FW2","CP1CGNAT"]) then
> ?TmplcpFW*
> *& ~*
>
> Now I would like to further segregate these logs based on IP-pools used (*e.g-
> 100.70.0.0*) as this is the only unique data available in all event logs.
>
> Below is the sample logs and info available in various events (*for one of
> the ip-pool*)
>
> *<150>1 2018-12-20T16:07:00.482369+01:00 2018-12-20 15 - - - 06:59:
> CP1CGNAT_O{OFR_NAT}[jservices-nat]: JSERVICES_NAT_RULE_MATCH: proto 6 (TCP)
> application: any, xe-1/0/1.1718:100.70.0.2:59794 -> 109.32.8.15:80
> <http://109.32.8.15:80>, Match NAT rule-set: (null), rule: O_NAT_XX, term:
> t1*
> *<150>1 2018-12-20T16:07:00.482369+01:00 2018-12-20 15 - - - 06:59:
> CP1CGNAT_O{OFR_NAT}JSERVICES_SESSION_OPEN: application:none, xe-1/0/1.1718
> 100.70.0.2:59794 [55.93.69.53:26620 <http://55.93.69.53:26620>] ->
> 109.32.8.15:80 <http://109.32.8.15:80> (TCP) *
> *<150>1 2018-12-20T16:07:11.091313+01:00 2018-12-20 15 - - - 07:10:
> CP1CGNAT_O{OFR_NAT}JSERVICES_SESSION_CLOSE: application:none, xe-1/0/1.1718
> 100.70.0.2:59778 [55.93.69.60:40136 <http://55.93.69.60:40136>] ->
> 109.32.8.15:80 <http://109.32.8.15:80> (TCP) *
>
> Can you please suggest/help to achieve this in rsyslog?
>
>
> Thanks a lot for your great help.
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Filtering syslogs on server and writing to specific files [ In reply to ]
Thanks David, I have used the different log-prefixs (configured in NAT
device) as filter conditions and used in templates as (total 10 similar
templates) -

*$template TmplcpFW_P,
"/var/log/NIPFW/Poland/Poland_%HOSTNAME%_%$year%.%$month%.%$day%.log"*
*if ($hostname == ["CP1CGNAT","CP2CGNAT"]) and $msg contains
'CP1CGNAT_O_Poland' then ?TmplcpFW_P*

*$template TmplcpFW_F,
"/var/log/NIPFW/France/France_%HOSTNAME%_%$year%.%$month%.%$day%.log"*
*if ($hostname == ["CP1CGNAT","CP2CGNAT"]) and $msg contains
'CP1CGNAT_O_France' then ?TmplcpFW_F*

Now I want only matching logs from above conditions to go into
corresponding directories/files and remaining logs in /var/log/messages.

Can you please suggest how to do this ?

Other suggestions/recommendation are most welcome.


On Thu, Jan 10, 2019 at 2:18 PM David Lang <david@lang.hm> wrote:

> The first thing to do is to parse the logs apart using mmnormalize, then
> you can
> either use startswith matches, or lookup_table to decide hat to put where.
>
> but until you parse the logs apart so that you have the IP address you
> want to
> filter by in a variable of it's own, it's too messy to deal with.
>
> David Lang
>
> On Tue, 8 Jan 2019, sarjit yadav via rsyslog wrote:
>
> > Date: Tue, 8 Jan 2019 16:44:25 +0530
> > From: sarjit yadav via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog@lists.adiscon.com
> > Cc: sarjit yadav <sarjit.ymca@gmail.com>
> > Subject: [rsyslog] Filtering syslogs on server and writing to specific
> files
> >
> > Dear Experts,
> >
> > I have implemented syslog server on CentOS and filtering all *NATing
> *events
> > from Juniper nodes into a single file using below template
> (*rsyslog.conf*)
> > -
> >
> > *##For redirecting the NAT/FW logs to specific file/directory*
> > *#$template TmplcpFW, "/var/log/NIPFW/%HOSTNAME%.log"*
> > *if ($hostname == ["CP1FW1", "CP1FW2", "CP2FW1", "CP2FW2","CP1CGNAT"])
> then
> > ?TmplcpFW*
> > *& ~*
> >
> > Now I would like to further segregate these logs based on IP-pools used
> (*e.g-
> > 100.70.0.0*) as this is the only unique data available in all event logs.
> >
> > Below is the sample logs and info available in various events (*for one
> of
> > the ip-pool*)
> >
> > *<150>1 2018-12-20T16:07:00.482369+01:00 2018-12-20 15 - - - 06:59:
> > CP1CGNAT_O{OFR_NAT}[jservices-nat]: JSERVICES_NAT_RULE_MATCH: proto 6
> (TCP)
> > application: any, xe-1/0/1.1718:100.70.0.2:59794 -> 109.32.8.15:80
> > <http://109.32.8.15:80>, Match NAT rule-set: (null), rule: O_NAT_XX,
> term:
> > t1*
> > *<150>1 2018-12-20T16:07:00.482369+01:00 2018-12-20 15 - - - 06:59:
> > CP1CGNAT_O{OFR_NAT}JSERVICES_SESSION_OPEN: application:none,
> xe-1/0/1.1718
> > 100.70.0.2:59794 [55.93.69.53:26620 <http://55.93.69.53:26620>] ->
> > 109.32.8.15:80 <http://109.32.8.15:80> (TCP) *
> > *<150>1 2018-12-20T16:07:11.091313+01:00 2018-12-20 15 - - - 07:10:
> > CP1CGNAT_O{OFR_NAT}JSERVICES_SESSION_CLOSE: application:none,
> xe-1/0/1.1718
> > 100.70.0.2:59778 [55.93.69.60:40136 <http://55.93.69.60:40136>] ->
> > 109.32.8.15:80 <http://109.32.8.15:80> (TCP) *
> >
> > Can you please suggest/help to achieve this in rsyslog?
> >
> >
> > Thanks a lot for your great help.
> >
> >
>


--

Regards

Sarjit
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Filtering syslogs on server and writing to specific files [ In reply to ]
after the 'then' you can put multiple statements, just inclose them in []

so you can do
{
?TmplcpFW_P
stop
}

the 'stop' tells rsyslog to not process any more actions, so if the write to
/var/log/messages is after this, it won't be processed for that log message.

David Lang



On Fri, 11 Jan 2019, sarjit yadav wrote:

> Date: Fri, 11 Jan 2019 11:41:49 +0530
> From: sarjit yadav <sarjit.ymca@gmail.com>
> To: David Lang <david@lang.hm>
> Cc: sarjit yadav via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Filtering syslogs on server and writing to specific
> files
>
> Thanks David, I have used the different log-prefixs (configured in NAT
> device) as filter conditions and used in templates as (total 10 similar
> templates) -
>
> *$template TmplcpFW_P,
> "/var/log/NIPFW/Poland/Poland_%HOSTNAME%_%$year%.%$month%.%$day%.log"*
> *if ($hostname == ["CP1CGNAT","CP2CGNAT"]) and $msg contains
> 'CP1CGNAT_O_Poland' then ?TmplcpFW_P*
>
> *$template TmplcpFW_F,
> "/var/log/NIPFW/France/France_%HOSTNAME%_%$year%.%$month%.%$day%.log"*
> *if ($hostname == ["CP1CGNAT","CP2CGNAT"]) and $msg contains
> 'CP1CGNAT_O_France' then ?TmplcpFW_F*
>
> Now I want only matching logs from above conditions to go into
> corresponding directories/files and remaining logs in /var/log/messages.
>
> Can you please suggest how to do this ?
>
> Other suggestions/recommendation are most welcome.
>
>
> On Thu, Jan 10, 2019 at 2:18 PM David Lang <david@lang.hm> wrote:
>
>> The first thing to do is to parse the logs apart using mmnormalize, then
>> you can
>> either use startswith matches, or lookup_table to decide hat to put where.
>>
>> but until you parse the logs apart so that you have the IP address you
>> want to
>> filter by in a variable of it's own, it's too messy to deal with.
>>
>> David Lang
>>
>> On Tue, 8 Jan 2019, sarjit yadav via rsyslog wrote:
>>
>>> Date: Tue, 8 Jan 2019 16:44:25 +0530
>>> From: sarjit yadav via rsyslog <rsyslog@lists.adiscon.com>
>>> To: rsyslog@lists.adiscon.com
>>> Cc: sarjit yadav <sarjit.ymca@gmail.com>
>>> Subject: [rsyslog] Filtering syslogs on server and writing to specific
>> files
>>>
>>> Dear Experts,
>>>
>>> I have implemented syslog server on CentOS and filtering all *NATing
>> *events
>>> from Juniper nodes into a single file using below template
>> (*rsyslog.conf*)
>>> -
>>>
>>> *##For redirecting the NAT/FW logs to specific file/directory*
>>> *#$template TmplcpFW, "/var/log/NIPFW/%HOSTNAME%.log"*
>>> *if ($hostname == ["CP1FW1", "CP1FW2", "CP2FW1", "CP2FW2","CP1CGNAT"])
>> then
>>> ?TmplcpFW*
>>> *& ~*
>>>
>>> Now I would like to further segregate these logs based on IP-pools used
>> (*e.g-
>>> 100.70.0.0*) as this is the only unique data available in all event logs.
>>>
>>> Below is the sample logs and info available in various events (*for one
>> of
>>> the ip-pool*)
>>>
>>> *<150>1 2018-12-20T16:07:00.482369+01:00 2018-12-20 15 - - - 06:59:
>>> CP1CGNAT_O{OFR_NAT}[jservices-nat]: JSERVICES_NAT_RULE_MATCH: proto 6
>> (TCP)
>>> application: any, xe-1/0/1.1718:100.70.0.2:59794 -> 109.32.8.15:80
>>> <http://109.32.8.15:80>, Match NAT rule-set: (null), rule: O_NAT_XX,
>> term:
>>> t1*
>>> *<150>1 2018-12-20T16:07:00.482369+01:00 2018-12-20 15 - - - 06:59:
>>> CP1CGNAT_O{OFR_NAT}JSERVICES_SESSION_OPEN: application:none,
>> xe-1/0/1.1718
>>> 100.70.0.2:59794 [55.93.69.53:26620 <http://55.93.69.53:26620>] ->
>>> 109.32.8.15:80 <http://109.32.8.15:80> (TCP) *
>>> *<150>1 2018-12-20T16:07:11.091313+01:00 2018-12-20 15 - - - 07:10:
>>> CP1CGNAT_O{OFR_NAT}JSERVICES_SESSION_CLOSE: application:none,
>> xe-1/0/1.1718
>>> 100.70.0.2:59778 [55.93.69.60:40136 <http://55.93.69.60:40136>] ->
>>> 109.32.8.15:80 <http://109.32.8.15:80> (TCP) *
>>>
>>> Can you please suggest/help to achieve this in rsyslog?
>>>
>>>
>>> Thanks a lot for your great help.
>>>
>>>
>>
>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.