Mailing List Archive

Missing messages
List,

Our firewall guys reported that they're missing messages.
They've got devices pointed at a dedicated rsyslog server ( they are the only folks using it ) and they are seeing cases were they are sending messages, I am receiving the messages, but the messages aren't getting written to the logs.
The missing messages aren't from any particular source, the issue seems to occur all throughout the day, and in the case of the messages, we expect to see when sessions are built and torn down.
There are times the entire session is captured, times when part of the session is missed, and times when the entire session is missed.

I know the message is reaching the server, because I ran a packet capture and see the data.
I turned on debug and captured a pretty healthy dump of data.
BTW, I set: RSYSLOG_DEBUG to "Debug".

Messages that are getting logged, are visible in the debug logs.
Messages that aren't getting logged leave no trace anywhere.

I'm running:
Name : rsyslog
Version : 8.24.0
Release : 16.el7
Architecture: x86_64

On RHEL 7.5.

Do you guys have any ideas of things I might try to get more info?

Thank you,

Radesh
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Missing messages [ In reply to ]
Just to ensure that I don't too much confusion.
I made the following statement:

Messages that aren't getting logged leave no trace anywhere.

I should have said:
Messages that aren't getting logged leave no trace anywhere, EXCEPT the packet capture.

The missing messages are being captured in the tcpdump, so when my firewall guy sends me a sample of what he sent, if I'm running a packet capture, I've validated that the server has received the data, even if rsyslog isn't writing it down.

Thanks again.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Singh, Radesh via rsyslog
Sent: Thursday, August 01, 2019 3:08 PM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Singh, Radesh <Radesh_Singh@csx.com>
Subject: [E][rsyslog] Missing messages

List,

Our firewall guys reported that they're missing messages.
They've got devices pointed at a dedicated rsyslog server ( they are the only folks using it ) and they are seeing cases were they are sending messages, I am receiving the messages, but the messages aren't getting written to the logs.
The missing messages aren't from any particular source, the issue seems to occur all throughout the day, and in the case of the messages, we expect to see when sessions are built and torn down.
There are times the entire session is captured, times when part of the session is missed, and times when the entire session is missed.

I know the message is reaching the server, because I ran a packet capture and see the data.
I turned on debug and captured a pretty healthy dump of data.
BTW, I set: RSYSLOG_DEBUG to "Debug".

Messages that are getting logged, are visible in the debug logs.
Messages that aren't getting logged leave no trace anywhere.

I'm running:
Name : rsyslog
Version : 8.24.0
Release : 16.el7
Architecture: x86_64

On RHEL 7.5.

Do you guys have any ideas of things I might try to get more info?

Thank you,

Radesh
_______________________________________________
rsyslog mailing list
http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-MO5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi4n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH2ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr5D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT09bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6KklJgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPobE8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJsscL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5q7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNVgpZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26vzubCvnjmiVvAjt-g/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
__________________________________________
E - EXTERNAL SENDER
Use discretion when clicking links, opening attachments, or replying.
__________________________________________

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Missing messages [ In reply to ]
how are you rotating your logs? are you sending rsyslog a HUP or are you doing a
/etc/init.d/reload (which is a full restart)?

are you using copytruncate as you rotate your logs?

these are the most common issues.

if you are sending via UDP, check your OS UDP buffers, if they are filling up,
the packets will be dropped before they get to rsyslog.

8.24 is pretty old (~2.5 years), but it's unlikely to be the root cause of the
problem

David Lang


On Thu, 1 Aug 2019, Singh, Radesh via rsyslog wrote:

> Date: Thu, 1 Aug 2019 19:13:11 +0000
> From: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: "Singh, Radesh" <Radesh_Singh@csx.com>
> Subject: Re: [rsyslog] Missing messages
>
> Just to ensure that I don't too much confusion.
> I made the following statement:
>
> Messages that aren't getting logged leave no trace anywhere.
>
> I should have said:
> Messages that aren't getting logged leave no trace anywhere, EXCEPT the packet capture.
>
> The missing messages are being captured in the tcpdump, so when my firewall guy sends me a sample of what he sent, if I'm running a packet capture, I've validated that the server has received the data, even if rsyslog isn't writing it down.
>
> Thanks again.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Singh, Radesh via rsyslog
> Sent: Thursday, August 01, 2019 3:08 PM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Singh, Radesh <Radesh_Singh@csx.com>
> Subject: [E][rsyslog] Missing messages
>
> List,
>
> Our firewall guys reported that they're missing messages.
> They've got devices pointed at a dedicated rsyslog server ( they are the only folks using it ) and they are seeing cases were they are sending messages, I am receiving the messages, but the messages aren't getting written to the logs.
> The missing messages aren't from any particular source, the issue seems to occur all throughout the day, and in the case of the messages, we expect to see when sessions are built and torn down.
> There are times the entire session is captured, times when part of the session is missed, and times when the entire session is missed.
>
> I know the message is reaching the server, because I ran a packet capture and see the data.
> I turned on debug and captured a pretty healthy dump of data.
> BTW, I set: RSYSLOG_DEBUG to "Debug".
>
> Messages that are getting logged, are visible in the debug logs.
> Messages that aren't getting logged leave no trace anywhere.
>
> I'm running:
> Name : rsyslog
> Version : 8.24.0
> Release : 16.el7
> Architecture: x86_64
>
> On RHEL 7.5.
>
> Do you guys have any ideas of things I might try to get more info?
>
> Thank you,
>
> Radesh
> _______________________________________________
> rsyslog mailing list
> http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-MO5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi4n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH2ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr5D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT09bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
> http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6KklJgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPobE8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJsscL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5q7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNVgpZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26vzubCvnjmiVvAjt-g/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> __________________________________________
> E - EXTERNAL SENDER
> Use discretion when clicking links, opening attachments, or replying.
> __________________________________________
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
David,

Thank you for your response.

The firewall logs don't get rotated.
The logs from the firewall get this rule applied to them:
$template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
:inputname , isequal , "imudp" ?RemoteClient
:inputname , isequal , "imtcp" ?RemoteClient

So we usually have some messages from the previous day, but that isn't a concern to us.

At first I suspected that logrotation might be the culprit, but the issue is occurring all throughout the day, and our logs usually rotate between 0000 and 0500 each day.
The only logrotation config file that interacts with rsyslog is our syslog config.
In that config file (/etc/logrotate.d/syslog), I see the following config:

syslog
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
/var/log/boot.log
{
missingok
sharedscripts
notifempty
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}

So, we are sending HUP, not using the copytruncate option.

I wonder if we are having issue with UDP buffers.
I haven't had to investigate that before.

Could you provide any suggestions on how you'd go about that on RHEL7?
A quick google return several results... one of which suggests watching /proc/net/udp, and comparing the tx_queue and rx_queue, but as I haven't run across this before, wondering if that is the best approach.

Thanks,

Radesh


-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Thursday, August 01, 2019 4:45 PM
To: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Cc: Singh, Radesh <Radesh_Singh@csx.com>
Subject: [E]Re: [rsyslog] Missing messages

how are you rotating your logs? are you sending rsyslog a HUP or are you doing a /etc/init.d/reload (which is a full restart)?

are you using copytruncate as you rotate your logs?

these are the most common issues.

if you are sending via UDP, check your OS UDP buffers, if they are filling up, the packets will be dropped before they get to rsyslog.

8.24 is pretty old (~2.5 years), but it's unlikely to be the root cause of the problem

David Lang


On Thu, 1 Aug 2019, Singh, Radesh via rsyslog wrote:

> Date: Thu, 1 Aug 2019 19:13:11 +0000
> From: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: "Singh, Radesh" <Radesh_Singh@csx.com>
> Subject: Re: [rsyslog] Missing messages
>
> Just to ensure that I don't too much confusion.
> I made the following statement:
>
> Messages that aren't getting logged leave no trace anywhere.
>
> I should have said:
> Messages that aren't getting logged leave no trace anywhere, EXCEPT the packet capture.
>
> The missing messages are being captured in the tcpdump, so when my firewall guy sends me a sample of what he sent, if I'm running a packet capture, I've validated that the server has received the data, even if rsyslog isn't writing it down.
>
> Thanks again.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Singh,
> Radesh via rsyslog
> Sent: Thursday, August 01, 2019 3:08 PM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Singh, Radesh <Radesh_Singh@csx.com>
> Subject: [E][rsyslog] Missing messages
>
> List,
>
> Our firewall guys reported that they're missing messages.
> They've got devices pointed at a dedicated rsyslog server ( they are the only folks using it ) and they are seeing cases were they are sending messages, I am receiving the messages, but the messages aren't getting written to the logs.
> The missing messages aren't from any particular source, the issue seems to occur all throughout the day, and in the case of the messages, we expect to see when sessions are built and torn down.
> There are times the entire session is captured, times when part of the session is missed, and times when the entire session is missed.
>
> I know the message is reaching the server, because I ran a packet capture and see the data.
> I turned on debug and captured a pretty healthy dump of data.
> BTW, I set: RSYSLOG_DEBUG to "Debug".
>
> Messages that are getting logged, are visible in the debug logs.
> Messages that aren't getting logged leave no trace anywhere.
>
> I'm running:
> Name : rsyslog
> Version : 8.24.0
> Release : 16.el7
> Architecture: x86_64
>
> On RHEL 7.5.
>
> Do you guys have any ideas of things I might try to get more info?
>
> Thank you,
>
> Radesh
> _______________________________________________
> rsyslog mailing list
> http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-MO
> 5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi4
> n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH2
> ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr5
> D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT09
> bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%3
> A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
> http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6Kkl
> JgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPobE
> 8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJs
> scL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5q
> 7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNVg
> pZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26vz
> ubCvnjmiVvAjt-g/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2
> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> __________________________________________
> E - EXTERNAL SENDER
> Use discretion when clicking links, opening attachments, or replying.
> __________________________________________
>
> _______________________________________________
> rsyslog mailing list
> http://secure-web.cisco.com/1Yd3N88k4kLwQnn6wm8nV_N7J2OaVx0bD2BdIWsKLr
> FE0ZaOj3W01mHVLaElEBwN-8H0WiHag1JSIym27W1OuyVKNfLmf77yYJ9Cm9CnWBOSzaAe
> 9UjGSiTaiNydxElV2Iltc8-n0H1asSF4KQUugSLV1rCOlmbvJjUX_LEYNlwfdmoIboUPDM
> taUxedojI9XGrqfeIJk2Ei_Z4mznxnej5Fe4wiWN6JTwCWh6zBK1dNGX52BnSB6s6dHOS9
> ejGwTi09PSEQBztWHHFXKX2Naid30ed_iPiKc5i38dAiJ4bFTI0E808MttZdgwmUCv7JMc
> jfDTojUSOR0_CVK_0m6TYXm--LmFgc2Rizd305L7j3iOHuCQJmnJ47G0BhV1aLlpQpYc1Y
> pIi4K2rpXsUafcA/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2F
> rsyslog
> http://secure-web.cisco.com/1IHJGgE_2T4pU7cw6lpFJqUNH3yko3ST-aOsNhS-rn
> tkkdSonwR7ZoNduMbgJuf3Xkhx8LpgB3iAC68nYf95aAkbLmNmxK86nUypZwYR_y6E2xjm
> GnV39qiCcdcikUxq8ShscO2fAm3PKpXBwO8jUWSyMhgHgmffHR1p28C7eHvOQ3sf_78ljG
> sHyMDHHbi0nVCqWAhXRCDNWKloDLWUIEqFIGE1jMADGyxFf7u-51jPUcNUbRQtigP2aezh
> jhm1DNftQu_kBqVJ9-yU5DQqIq8dshZZYuJgXXg-GuCawlr17h82XWWjYKv10RTFu1EVUT
> j1s6wWhVU_3iKRwq3U8uaZ8H57YjkkALH-M0OHSj6WGQda82Zoq1gNO7IMSBb3bjexSiLj
> UUXUdy0ikBSRjDg/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2
> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
__________________________________________
E - EXTERNAL SENDER
Use discretion when clicking links, opening attachments, or replying.
__________________________________________

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
David,

I may have found more clues...

The number of errors related to UDP are really high...

netstat -su
IcmpMsg:
InType0: 6
InType3: 980584
InType8: 28959
InType11: 688
OutType0: 28959
OutType3: 987041
OutType8: 6
Udp:
140571327944 packets received
332780 packets to unknown port received.
151716349245 packet receive errors
207450667309 packets sent
151716213631 receive buffer errors
287 send buffer errors
InCsumErrors: 135614
UdpLite:
IpExt:
InBcastPkts: 446852
InOctets: 73907716648604
OutOctets: 103321724708971
InBcastOctets: 130548312
InNoECTPkts: 356643654014
InECT0Pkts: 810

I compared this to another log host, and while the other log host has errors, the numbers are nowhere as large.

Radesh

-----Original Message-----
From: Singh, Radesh
Sent: Thursday, August 01, 2019 5:07 PM
To: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

David,

Thank you for your response.

The firewall logs don't get rotated.
The logs from the firewall get this rule applied to them:
$template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
:inputname , isequal , "imudp" ?RemoteClient :inputname , isequal , "imtcp" ?RemoteClient

So we usually have some messages from the previous day, but that isn't a concern to us.

At first I suspected that logrotation might be the culprit, but the issue is occurring all throughout the day, and our logs usually rotate between 0000 and 0500 each day.
The only logrotation config file that interacts with rsyslog is our syslog config.
In that config file (/etc/logrotate.d/syslog), I see the following config:

syslog
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
/var/log/boot.log
{
missingok
sharedscripts
notifempty
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript
}

So, we are sending HUP, not using the copytruncate option.

I wonder if we are having issue with UDP buffers.
I haven't had to investigate that before.

Could you provide any suggestions on how you'd go about that on RHEL7?
A quick google return several results... one of which suggests watching /proc/net/udp, and comparing the tx_queue and rx_queue, but as I haven't run across this before, wondering if that is the best approach.

Thanks,

Radesh


-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Thursday, August 01, 2019 4:45 PM
To: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Cc: Singh, Radesh <Radesh_Singh@csx.com>
Subject: [E]Re: [rsyslog] Missing messages

how are you rotating your logs? are you sending rsyslog a HUP or are you doing a /etc/init.d/reload (which is a full restart)?

are you using copytruncate as you rotate your logs?

these are the most common issues.

if you are sending via UDP, check your OS UDP buffers, if they are filling up, the packets will be dropped before they get to rsyslog.

8.24 is pretty old (~2.5 years), but it's unlikely to be the root cause of the problem

David Lang


On Thu, 1 Aug 2019, Singh, Radesh via rsyslog wrote:

> Date: Thu, 1 Aug 2019 19:13:11 +0000
> From: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: "Singh, Radesh" <Radesh_Singh@csx.com>
> Subject: Re: [rsyslog] Missing messages
>
> Just to ensure that I don't too much confusion.
> I made the following statement:
>
> Messages that aren't getting logged leave no trace anywhere.
>
> I should have said:
> Messages that aren't getting logged leave no trace anywhere, EXCEPT the packet capture.
>
> The missing messages are being captured in the tcpdump, so when my firewall guy sends me a sample of what he sent, if I'm running a packet capture, I've validated that the server has received the data, even if rsyslog isn't writing it down.
>
> Thanks again.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Singh,
> Radesh via rsyslog
> Sent: Thursday, August 01, 2019 3:08 PM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Singh, Radesh <Radesh_Singh@csx.com>
> Subject: [E][rsyslog] Missing messages
>
> List,
>
> Our firewall guys reported that they're missing messages.
> They've got devices pointed at a dedicated rsyslog server ( they are the only folks using it ) and they are seeing cases were they are sending messages, I am receiving the messages, but the messages aren't getting written to the logs.
> The missing messages aren't from any particular source, the issue seems to occur all throughout the day, and in the case of the messages, we expect to see when sessions are built and torn down.
> There are times the entire session is captured, times when part of the session is missed, and times when the entire session is missed.
>
> I know the message is reaching the server, because I ran a packet capture and see the data.
> I turned on debug and captured a pretty healthy dump of data.
> BTW, I set: RSYSLOG_DEBUG to "Debug".
>
> Messages that are getting logged, are visible in the debug logs.
> Messages that aren't getting logged leave no trace anywhere.
>
> I'm running:
> Name : rsyslog
> Version : 8.24.0
> Release : 16.el7
> Architecture: x86_64
>
> On RHEL 7.5.
>
> Do you guys have any ideas of things I might try to get more info?
>
> Thank you,
>
> Radesh
> _______________________________________________
> rsyslog mailing list
> http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-MO
> 5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi4
> n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH2
> ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr5
> D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT09
> bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%3
> A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
> http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6Kkl
> JgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPobE
> 8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJs
> scL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5q
> 7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNVg
> pZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26vz
> ubCvnjmiVvAjt-g/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2
> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> __________________________________________
> E - EXTERNAL SENDER
> Use discretion when clicking links, opening attachments, or replying.
> __________________________________________
>
> _______________________________________________
> rsyslog mailing list
> http://secure-web.cisco.com/1Yd3N88k4kLwQnn6wm8nV_N7J2OaVx0bD2BdIWsKLr
> FE0ZaOj3W01mHVLaElEBwN-8H0WiHag1JSIym27W1OuyVKNfLmf77yYJ9Cm9CnWBOSzaAe
> 9UjGSiTaiNydxElV2Iltc8-n0H1asSF4KQUugSLV1rCOlmbvJjUX_LEYNlwfdmoIboUPDM
> taUxedojI9XGrqfeIJk2Ei_Z4mznxnej5Fe4wiWN6JTwCWh6zBK1dNGX52BnSB6s6dHOS9
> ejGwTi09PSEQBztWHHFXKX2Naid30ed_iPiKc5i38dAiJ4bFTI0E808MttZdgwmUCv7JMc
> jfDTojUSOR0_CVK_0m6TYXm--LmFgc2Rizd305L7j3iOHuCQJmnJ47G0BhV1aLlpQpYc1Y
> pIi4K2rpXsUafcA/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2F
> rsyslog
> http://secure-web.cisco.com/1IHJGgE_2T4pU7cw6lpFJqUNH3yko3ST-aOsNhS-rn
> tkkdSonwR7ZoNduMbgJuf3Xkhx8LpgB3iAC68nYf95aAkbLmNmxK86nUypZwYR_y6E2xjm
> GnV39qiCcdcikUxq8ShscO2fAm3PKpXBwO8jUWSyMhgHgmffHR1p28C7eHvOQ3sf_78ljG
> sHyMDHHbi0nVCqWAhXRCDNWKloDLWUIEqFIGE1jMADGyxFf7u-51jPUcNUbRQtigP2aezh
> jhm1DNftQu_kBqVJ9-yU5DQqIq8dshZZYuJgXXg-GuCawlr17h82XWWjYKv10RTFu1EVUT
> j1s6wWhVU_3iKRwq3U8uaZ8H57YjkkALH-M0OHSj6WGQda82Zoq1gNO7IMSBb3bjexSiLj
> UUXUdy0ikBSRjDg/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2
> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
__________________________________________
E - EXTERNAL SENDER
Use discretion when clicking links, opening attachments, or replying.
__________________________________________

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
yep, each of those errors is probably a lost log
in sysctl you can increase the buffer size, but you probably also need to tune
rsyslog to handle logs faster

if you can post your config it would help

enabling impstats would help understand what's going on inside rsyslog
(including showing the counts of the number of messages it sees)

David Lang

On Thu, 1 Aug 2019, Singh, Radesh wrote:

> Date: Thu, 1 Aug 2019 22:48:40 +0000
> From: "Singh, Radesh" <Radesh_Singh@csx.com>
> To: David Lang <david@lang.hm>,
> "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> I may have found more clues...
>
> The number of errors related to UDP are really high...
>
> netstat -su
> IcmpMsg:
> InType0: 6
> InType3: 980584
> InType8: 28959
> InType11: 688
> OutType0: 28959
> OutType3: 987041
> OutType8: 6
> Udp:
> 140571327944 packets received
> 332780 packets to unknown port received.
> 151716349245 packet receive errors
> 207450667309 packets sent
> 151716213631 receive buffer errors
> 287 send buffer errors
> InCsumErrors: 135614
> UdpLite:
> IpExt:
> InBcastPkts: 446852
> InOctets: 73907716648604
> OutOctets: 103321724708971
> InBcastOctets: 130548312
> InNoECTPkts: 356643654014
> InECT0Pkts: 810
>
> I compared this to another log host, and while the other log host has errors, the numbers are nowhere as large.
>
> Radesh
>
> -----Original Message-----
> From: Singh, Radesh
> Sent: Thursday, August 01, 2019 5:07 PM
> To: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> Thank you for your response.
>
> The firewall logs don't get rotated.
> The logs from the firewall get this rule applied to them:
> $template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
> :inputname , isequal , "imudp" ?RemoteClient :inputname , isequal , "imtcp" ?RemoteClient
>
> So we usually have some messages from the previous day, but that isn't a concern to us.
>
> At first I suspected that logrotation might be the culprit, but the issue is occurring all throughout the day, and our logs usually rotate between 0000 and 0500 each day.
> The only logrotation config file that interacts with rsyslog is our syslog config.
> In that config file (/etc/logrotate.d/syslog), I see the following config:
>
> syslog
> /var/log/cron
> /var/log/maillog
> /var/log/messages
> /var/log/secure
> /var/log/spooler
> /var/log/boot.log
> {
> missingok
> sharedscripts
> notifempty
> postrotate
> /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
> endscript
> }
>
> So, we are sending HUP, not using the copytruncate option.
>
> I wonder if we are having issue with UDP buffers.
> I haven't had to investigate that before.
>
> Could you provide any suggestions on how you'd go about that on RHEL7?
> A quick google return several results... one of which suggests watching /proc/net/udp, and comparing the tx_queue and rx_queue, but as I haven't run across this before, wondering if that is the best approach.
>
> Thanks,
>
> Radesh
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Thursday, August 01, 2019 4:45 PM
> To: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Singh, Radesh <Radesh_Singh@csx.com>
> Subject: [E]Re: [rsyslog] Missing messages
>
> how are you rotating your logs? are you sending rsyslog a HUP or are you doing a /etc/init.d/reload (which is a full restart)?
>
> are you using copytruncate as you rotate your logs?
>
> these are the most common issues.
>
> if you are sending via UDP, check your OS UDP buffers, if they are filling up, the packets will be dropped before they get to rsyslog.
>
> 8.24 is pretty old (~2.5 years), but it's unlikely to be the root cause of the problem
>
> David Lang
>
>
> On Thu, 1 Aug 2019, Singh, Radesh via rsyslog wrote:
>
>> Date: Thu, 1 Aug 2019 19:13:11 +0000
>> From: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: "Singh, Radesh" <Radesh_Singh@csx.com>
>> Subject: Re: [rsyslog] Missing messages
>>
>> Just to ensure that I don't too much confusion.
>> I made the following statement:
>>
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I should have said:
>> Messages that aren't getting logged leave no trace anywhere, EXCEPT the packet capture.
>>
>> The missing messages are being captured in the tcpdump, so when my firewall guy sends me a sample of what he sent, if I'm running a packet capture, I've validated that the server has received the data, even if rsyslog isn't writing it down.
>>
>> Thanks again.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Singh,
>> Radesh via rsyslog
>> Sent: Thursday, August 01, 2019 3:08 PM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Singh, Radesh <Radesh_Singh@csx.com>
>> Subject: [E][rsyslog] Missing messages
>>
>> List,
>>
>> Our firewall guys reported that they're missing messages.
>> They've got devices pointed at a dedicated rsyslog server ( they are the only folks using it ) and they are seeing cases were they are sending messages, I am receiving the messages, but the messages aren't getting written to the logs.
>> The missing messages aren't from any particular source, the issue seems to occur all throughout the day, and in the case of the messages, we expect to see when sessions are built and torn down.
>> There are times the entire session is captured, times when part of the session is missed, and times when the entire session is missed.
>>
>> I know the message is reaching the server, because I ran a packet capture and see the data.
>> I turned on debug and captured a pretty healthy dump of data.
>> BTW, I set: RSYSLOG_DEBUG to "Debug".
>>
>> Messages that are getting logged, are visible in the debug logs.
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I'm running:
>> Name : rsyslog
>> Version : 8.24.0
>> Release : 16.el7
>> Architecture: x86_64
>>
>> On RHEL 7.5.
>>
>> Do you guys have any ideas of things I might try to get more info?
>>
>> Thank you,
>>
>> Radesh
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-MO
>> 5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi4
>> n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH2
>> ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr5
>> D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT09
>> bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%3
>> A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
>> http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6Kkl
>> JgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPobE
>> 8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJs
>> scL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5q
>> 7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNVg
>> pZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26vz
>> ubCvnjmiVvAjt-g/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> __________________________________________
>> E - EXTERNAL SENDER
>> Use discretion when clicking links, opening attachments, or replying.
>> __________________________________________
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1Yd3N88k4kLwQnn6wm8nV_N7J2OaVx0bD2BdIWsKLr
>> FE0ZaOj3W01mHVLaElEBwN-8H0WiHag1JSIym27W1OuyVKNfLmf77yYJ9Cm9CnWBOSzaAe
>> 9UjGSiTaiNydxElV2Iltc8-n0H1asSF4KQUugSLV1rCOlmbvJjUX_LEYNlwfdmoIboUPDM
>> taUxedojI9XGrqfeIJk2Ei_Z4mznxnej5Fe4wiWN6JTwCWh6zBK1dNGX52BnSB6s6dHOS9
>> ejGwTi09PSEQBztWHHFXKX2Naid30ed_iPiKc5i38dAiJ4bFTI0E808MttZdgwmUCv7JMc
>> jfDTojUSOR0_CVK_0m6TYXm--LmFgc2Rizd305L7j3iOHuCQJmnJ47G0BhV1aLlpQpYc1Y
>> pIi4K2rpXsUafcA/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2F
>> rsyslog
>> http://secure-web.cisco.com/1IHJGgE_2T4pU7cw6lpFJqUNH3yko3ST-aOsNhS-rn
>> tkkdSonwR7ZoNduMbgJuf3Xkhx8LpgB3iAC68nYf95aAkbLmNmxK86nUypZwYR_y6E2xjm
>> GnV39qiCcdcikUxq8ShscO2fAm3PKpXBwO8jUWSyMhgHgmffHR1p28C7eHvOQ3sf_78ljG
>> sHyMDHHbi0nVCqWAhXRCDNWKloDLWUIEqFIGE1jMADGyxFf7u-51jPUcNUbRQtigP2aezh
>> jhm1DNftQu_kBqVJ9-yU5DQqIq8dshZZYuJgXXg-GuCawlr17h82XWWjYKv10RTFu1EVUT
>> j1s6wWhVU_3iKRwq3U8uaZ8H57YjkkALH-M0OHSj6WGQda82Zoq1gNO7IMSBb3bjexSiLj
>> UUXUdy0ikBSRjDg/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
> __________________________________________
> E - EXTERNAL SENDER
> Use discretion when clicking links, opening attachments, or replying.
> __________________________________________
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state


#### RULES ####

### DEAL WITH REMOTE MESSAGES ########################################
$DirGroup secopsr
$FileGroup secopsr
$umask 0000
$DirCreateMode 0754
$FileCreateMode 0644
$template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
:inputname , isequal , "imudp" ?RemoteClient
:inputname , isequal , "imtcp" ?RemoteClient
:fromhost-ip , !isequal , "127.0.0.1" stop
########################################################################


# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

### PUSH LOCAL MESSAGES TO REMOTE ####################################
kern.* @unixloghost
*.info;mail.none;authpriv.none;cron.none @unixloghost
authpriv.* @unixloghost
*.emerg @unixloghost
local7.* @unixloghost
########################################################################

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Thursday, August 01, 2019 7:12 PM
To: Singh, Radesh <Radesh_Singh@csx.com>
Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

yep, each of those errors is probably a lost log in sysctl you can increase the buffer size, but you probably also need to tune rsyslog to handle logs faster

if you can post your config it would help

enabling impstats would help understand what's going on inside rsyslog (including showing the counts of the number of messages it sees)

David Lang

On Thu, 1 Aug 2019, Singh, Radesh wrote:

> Date: Thu, 1 Aug 2019 22:48:40 +0000
> From: "Singh, Radesh" <Radesh_Singh@csx.com>
> To: David Lang <david@lang.hm>,
> "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> I may have found more clues...
>
> The number of errors related to UDP are really high...
>
> netstat -su
> IcmpMsg:
> InType0: 6
> InType3: 980584
> InType8: 28959
> InType11: 688
> OutType0: 28959
> OutType3: 987041
> OutType8: 6
> Udp:
> 140571327944 packets received
> 332780 packets to unknown port received.
> 151716349245 packet receive errors
> 207450667309 packets sent
> 151716213631 receive buffer errors
> 287 send buffer errors
> InCsumErrors: 135614
> UdpLite:
> IpExt:
> InBcastPkts: 446852
> InOctets: 73907716648604
> OutOctets: 103321724708971
> InBcastOctets: 130548312
> InNoECTPkts: 356643654014
> InECT0Pkts: 810
>
> I compared this to another log host, and while the other log host has errors, the numbers are nowhere as large.
>
> Radesh
>
> -----Original Message-----
> From: Singh, Radesh
> Sent: Thursday, August 01, 2019 5:07 PM
> To: David Lang <david@lang.hm>; Singh, Radesh via rsyslog
> <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> Thank you for your response.
>
> The firewall logs don't get rotated.
> The logs from the firewall get this rule applied to them:
> $template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
> :inputname , isequal , "imudp" ?RemoteClient :inputname , isequal ,
> "imtcp" ?RemoteClient
>
> So we usually have some messages from the previous day, but that isn't a concern to us.
>
> At first I suspected that logrotation might be the culprit, but the issue is occurring all throughout the day, and our logs usually rotate between 0000 and 0500 each day.
> The only logrotation config file that interacts with rsyslog is our syslog config.
> In that config file (/etc/logrotate.d/syslog), I see the following config:
>
> syslog
> /var/log/cron
> /var/log/maillog
> /var/log/messages
> /var/log/secure
> /var/log/spooler
> /var/log/boot.log
> {
> missingok
> sharedscripts
> notifempty
> postrotate
> /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
> endscript
> }
>
> So, we are sending HUP, not using the copytruncate option.
>
> I wonder if we are having issue with UDP buffers.
> I haven't had to investigate that before.
>
> Could you provide any suggestions on how you'd go about that on RHEL7?
> A quick google return several results... one of which suggests watching /proc/net/udp, and comparing the tx_queue and rx_queue, but as I haven't run across this before, wondering if that is the best approach.
>
> Thanks,
>
> Radesh
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Thursday, August 01, 2019 4:45 PM
> To: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Singh, Radesh <Radesh_Singh@csx.com>
> Subject: [E]Re: [rsyslog] Missing messages
>
> how are you rotating your logs? are you sending rsyslog a HUP or are you doing a /etc/init.d/reload (which is a full restart)?
>
> are you using copytruncate as you rotate your logs?
>
> these are the most common issues.
>
> if you are sending via UDP, check your OS UDP buffers, if they are filling up, the packets will be dropped before they get to rsyslog.
>
> 8.24 is pretty old (~2.5 years), but it's unlikely to be the root
> cause of the problem
>
> David Lang
>
>
> On Thu, 1 Aug 2019, Singh, Radesh via rsyslog wrote:
>
>> Date: Thu, 1 Aug 2019 19:13:11 +0000
>> From: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: "Singh, Radesh" <Radesh_Singh@csx.com>
>> Subject: Re: [rsyslog] Missing messages
>>
>> Just to ensure that I don't too much confusion.
>> I made the following statement:
>>
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I should have said:
>> Messages that aren't getting logged leave no trace anywhere, EXCEPT the packet capture.
>>
>> The missing messages are being captured in the tcpdump, so when my firewall guy sends me a sample of what he sent, if I'm running a packet capture, I've validated that the server has received the data, even if rsyslog isn't writing it down.
>>
>> Thanks again.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Singh,
>> Radesh via rsyslog
>> Sent: Thursday, August 01, 2019 3:08 PM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Singh, Radesh <Radesh_Singh@csx.com>
>> Subject: [E][rsyslog] Missing messages
>>
>> List,
>>
>> Our firewall guys reported that they're missing messages.
>> They've got devices pointed at a dedicated rsyslog server ( they are the only folks using it ) and they are seeing cases were they are sending messages, I am receiving the messages, but the messages aren't getting written to the logs.
>> The missing messages aren't from any particular source, the issue seems to occur all throughout the day, and in the case of the messages, we expect to see when sessions are built and torn down.
>> There are times the entire session is captured, times when part of the session is missed, and times when the entire session is missed.
>>
>> I know the message is reaching the server, because I ran a packet capture and see the data.
>> I turned on debug and captured a pretty healthy dump of data.
>> BTW, I set: RSYSLOG_DEBUG to "Debug".
>>
>> Messages that are getting logged, are visible in the debug logs.
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I'm running:
>> Name : rsyslog
>> Version : 8.24.0
>> Release : 16.el7
>> Architecture: x86_64
>>
>> On RHEL 7.5.
>>
>> Do you guys have any ideas of things I might try to get more info?
>>
>> Thank you,
>>
>> Radesh
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-M
>> O
>> 5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi
>> 4
>> n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH
>> 2
>> ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr
>> 5
>> D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT0
>> 9
>> bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%
>> 3 A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
>> http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6Kk
>> l
>> JgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPob
>> E
>> 8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJ
>> s
>> scL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5
>> q
>> 7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNV
>> g
>> pZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26v
>> z
>> ubCvnjmiVvAjt-g/http%3A%2F%2Fhttp://secure-web.cisco.com/1xffTXMgG0Vt
>> WNgZbupUoRsbPPrAfrm5lifHCWFevtpUT3zeJvSOW5aaz7Ji1smzNB5Vt3pSo5tlI4iMv
>> 493fJcpD-BZHVNVWQffWPp3Zx3U9UtXnZemw6mzc_2B7THmkomCt_JkQLKAaXuakVeep1
>> z3z3Qc_Zl-qxbNsjiqAX4S1_m_Sx-UW2AZhQArme95hSYyOisK0IXxHsQ9Zb02S9Lq1Xs
>> DWpiDTGV0l94ccHi2cg4MHaV0ZUh4y6eLFIqxrn_a-W4HoFoOue20Y5CAJKZv_gCrQXu0
>> eepObC0CFWhsKiGIGYk6PAgZDXY9sE-KEQtMpbZLoGN-0abhk11n6rYJcaH2oyKlwpCWC
>> 5qcHZEFRA6P0ldZWA0-Kc80TPjG-4H_Ye6OE8u87H39pelfjzg/http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> __________________________________________
>> E - EXTERNAL SENDER
>> Use discretion when clicking links, opening attachments, or replying.
>> __________________________________________
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1Yd3N88k4kLwQnn6wm8nV_N7J2OaVx0bD2BdIWsKL
>> r
>> FE0ZaOj3W01mHVLaElEBwN-8H0WiHag1JSIym27W1OuyVKNfLmf77yYJ9Cm9CnWBOSzaA
>> e
>> 9UjGSiTaiNydxElV2Iltc8-n0H1asSF4KQUugSLV1rCOlmbvJjUX_LEYNlwfdmoIboUPD
>> M
>> taUxedojI9XGrqfeIJk2Ei_Z4mznxnej5Fe4wiWN6JTwCWh6zBK1dNGX52BnSB6s6dHOS
>> 9
>> ejGwTi09PSEQBztWHHFXKX2Naid30ed_iPiKc5i38dAiJ4bFTI0E808MttZdgwmUCv7JM
>> c
>> jfDTojUSOR0_CVK_0m6TYXm--LmFgc2Rizd305L7j3iOHuCQJmnJ47G0BhV1aLlpQpYc1
>> Y
>> pIi4K2rpXsUafcA/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2
>> F
>> rsyslog
>> http://secure-web.cisco.com/1IHJGgE_2T4pU7cw6lpFJqUNH3yko3ST-aOsNhS-r
>> n
>> tkkdSonwR7ZoNduMbgJuf3Xkhx8LpgB3iAC68nYf95aAkbLmNmxK86nUypZwYR_y6E2xj
>> m
>> GnV39qiCcdcikUxq8ShscO2fAm3PKpXBwO8jUWSyMhgHgmffHR1p28C7eHvOQ3sf_78lj
>> G
>> sHyMDHHbi0nVCqWAhXRCDNWKloDLWUIEqFIGE1jMADGyxFf7u-51jPUcNUbRQtigP2aez
>> h
>> jhm1DNftQu_kBqVJ9-yU5DQqIq8dshZZYuJgXXg-GuCawlr17h82XWWjYKv10RTFu1EVU
>> T
>> j1s6wWhVU_3iKRwq3U8uaZ8H57YjkkALH-M0OHSj6WGQda82Zoq1gNO7IMSBb3bjexSiL
>> j
>> UUXUdy0ikBSRjDg/http%3A%2F%2Fhttp://secure-web.cisco.com/1xffTXMgG0Vt
>> WNgZbupUoRsbPPrAfrm5lifHCWFevtpUT3zeJvSOW5aaz7Ji1smzNB5Vt3pSo5tlI4iMv
>> 493fJcpD-BZHVNVWQffWPp3Zx3U9UtXnZemw6mzc_2B7THmkomCt_JkQLKAaXuakVeep1
>> z3z3Qc_Zl-qxbNsjiqAX4S1_m_Sx-UW2AZhQArme95hSYyOisK0IXxHsQ9Zb02S9Lq1Xs
>> DWpiDTGV0l94ccHi2cg4MHaV0ZUh4y6eLFIqxrn_a-W4HoFoOue20Y5CAJKZv_gCrQXu0
>> eepObC0CFWhsKiGIGYk6PAgZDXY9sE-KEQtMpbZLoGN-0abhk11n6rYJcaH2oyKlwpCWC
>> 5qcHZEFRA6P0ldZWA0-Kc80TPjG-4H_Ye6OE8u87H39pelfjzg/http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
> __________________________________________
> E - EXTERNAL SENDER
> Use discretion when clicking links, opening attachments, or replying.
> __________________________________________
>
>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
Here's a look at how much memory we've got:
total used free shared buff/cache available
Mem: 65687464 1753788 9512004 336 54421672 63298172
Swap: 4194300 85080 4109220

Based on what I've been reading online I bumped up the following sysctls.

sysctl net.core.rmem_max=8388608
sysctl net.core.wmem_max=8388608
sysctl net.ipv4.udp_mem='2051962 3077940 8388608'
sysctl net.core.netdev_max_backlog=5000

Our send errors, don't look bad, especially in comparison to the rx errors, so guessing I don't need to mess with that one, but, not sure what to set rmem_max to, and how much memory to allow for udp traffic.

And then of course, once those values are tuned, what changes to make in my rsyslog.conf.

Radesh

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Thursday, August 01, 2019 7:12 PM
To: Singh, Radesh <Radesh_Singh@csx.com>
Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

yep, each of those errors is probably a lost log in sysctl you can increase the buffer size, but you probably also need to tune rsyslog to handle logs faster

if you can post your config it would help

enabling impstats would help understand what's going on inside rsyslog (including showing the counts of the number of messages it sees)

David Lang

On Thu, 1 Aug 2019, Singh, Radesh wrote:

> Date: Thu, 1 Aug 2019 22:48:40 +0000
> From: "Singh, Radesh" <Radesh_Singh@csx.com>
> To: David Lang <david@lang.hm>,
> "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> I may have found more clues...
>
> The number of errors related to UDP are really high...
>
> netstat -su
> IcmpMsg:
> InType0: 6
> InType3: 980584
> InType8: 28959
> InType11: 688
> OutType0: 28959
> OutType3: 987041
> OutType8: 6
> Udp:
> 140571327944 packets received
> 332780 packets to unknown port received.
> 151716349245 packet receive errors
> 207450667309 packets sent
> 151716213631 receive buffer errors
> 287 send buffer errors
> InCsumErrors: 135614
> UdpLite:
> IpExt:
> InBcastPkts: 446852
> InOctets: 73907716648604
> OutOctets: 103321724708971
> InBcastOctets: 130548312
> InNoECTPkts: 356643654014
> InECT0Pkts: 810
>
> I compared this to another log host, and while the other log host has errors, the numbers are nowhere as large.
>
> Radesh
>
> -----Original Message-----
> From: Singh, Radesh
> Sent: Thursday, August 01, 2019 5:07 PM
> To: David Lang <david@lang.hm>; Singh, Radesh via rsyslog
> <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> Thank you for your response.
>
> The firewall logs don't get rotated.
> The logs from the firewall get this rule applied to them:
> $template RemoteClient,"/var/remote/logs/%HOSTNAME%/%HOSTNAME%-%$NOW%.log"
> :inputname , isequal , "imudp" ?RemoteClient :inputname , isequal ,
> "imtcp" ?RemoteClient
>
> So we usually have some messages from the previous day, but that isn't a concern to us.
>
> At first I suspected that logrotation might be the culprit, but the issue is occurring all throughout the day, and our logs usually rotate between 0000 and 0500 each day.
> The only logrotation config file that interacts with rsyslog is our syslog config.
> In that config file (/etc/logrotate.d/syslog), I see the following config:
>
> syslog
> /var/log/cron
> /var/log/maillog
> /var/log/messages
> /var/log/secure
> /var/log/spooler
> /var/log/boot.log
> {
> missingok
> sharedscripts
> notifempty
> postrotate
> /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
> endscript
> }
>
> So, we are sending HUP, not using the copytruncate option.
>
> I wonder if we are having issue with UDP buffers.
> I haven't had to investigate that before.
>
> Could you provide any suggestions on how you'd go about that on RHEL7?
> A quick google return several results... one of which suggests watching /proc/net/udp, and comparing the tx_queue and rx_queue, but as I haven't run across this before, wondering if that is the best approach.
>
> Thanks,
>
> Radesh
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Thursday, August 01, 2019 4:45 PM
> To: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Singh, Radesh <Radesh_Singh@csx.com>
> Subject: [E]Re: [rsyslog] Missing messages
>
> how are you rotating your logs? are you sending rsyslog a HUP or are you doing a /etc/init.d/reload (which is a full restart)?
>
> are you using copytruncate as you rotate your logs?
>
> these are the most common issues.
>
> if you are sending via UDP, check your OS UDP buffers, if they are filling up, the packets will be dropped before they get to rsyslog.
>
> 8.24 is pretty old (~2.5 years), but it's unlikely to be the root
> cause of the problem
>
> David Lang
>
>
> On Thu, 1 Aug 2019, Singh, Radesh via rsyslog wrote:
>
>> Date: Thu, 1 Aug 2019 19:13:11 +0000
>> From: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: "Singh, Radesh" <Radesh_Singh@csx.com>
>> Subject: Re: [rsyslog] Missing messages
>>
>> Just to ensure that I don't too much confusion.
>> I made the following statement:
>>
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I should have said:
>> Messages that aren't getting logged leave no trace anywhere, EXCEPT the packet capture.
>>
>> The missing messages are being captured in the tcpdump, so when my firewall guy sends me a sample of what he sent, if I'm running a packet capture, I've validated that the server has received the data, even if rsyslog isn't writing it down.
>>
>> Thanks again.
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Singh,
>> Radesh via rsyslog
>> Sent: Thursday, August 01, 2019 3:08 PM
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Cc: Singh, Radesh <Radesh_Singh@csx.com>
>> Subject: [E][rsyslog] Missing messages
>>
>> List,
>>
>> Our firewall guys reported that they're missing messages.
>> They've got devices pointed at a dedicated rsyslog server ( they are the only folks using it ) and they are seeing cases were they are sending messages, I am receiving the messages, but the messages aren't getting written to the logs.
>> The missing messages aren't from any particular source, the issue seems to occur all throughout the day, and in the case of the messages, we expect to see when sessions are built and torn down.
>> There are times the entire session is captured, times when part of the session is missed, and times when the entire session is missed.
>>
>> I know the message is reaching the server, because I ran a packet capture and see the data.
>> I turned on debug and captured a pretty healthy dump of data.
>> BTW, I set: RSYSLOG_DEBUG to "Debug".
>>
>> Messages that are getting logged, are visible in the debug logs.
>> Messages that aren't getting logged leave no trace anywhere.
>>
>> I'm running:
>> Name : rsyslog
>> Version : 8.24.0
>> Release : 16.el7
>> Architecture: x86_64
>>
>> On RHEL 7.5.
>>
>> Do you guys have any ideas of things I might try to get more info?
>>
>> Thank you,
>>
>> Radesh
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1vnFnxbBxOsIs-qeTrS7MFa0Z0uJNdSgOaLs-O7-M
>> O
>> 5uwxTkNAicu4oJdX2Mqi7xjZvw9J7JIrCAPM3h53jFpQ31tX5j-1Lrjtg6JqfldwX_tCi
>> 4
>> n7YEc01LtwF6nXi1ObQvmdOyjeRFpX7cMAoRWX35LZvi6CPKRvP1g_pFDO-EN4_Mym6kH
>> 2
>> ADHl0QriRHR_cgPZs4gWlk5qTWSMlvpTR47RKtVRJMmH2Me-YCYheg00UOaUO9Sa_0VGr
>> 5
>> D0U6nPGkZ7LG9LC1kYZAd0tDZRgjiO78IHgwBrFnIf06ZMPtpVxm1jQ4ubf6_H89L6tT0
>> 9
>> bqjRDqUn5z3xAHLiJQln4oCDnjtpbYCYS2fKXqF91zfReNSsW3KKTEejN7RNNzT/http%
>> 3 A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
>> http://secure-web.cisco.com/1HJ44XxbxuDGgvam6LOZz_49mTfJDEXAnxRB9c6Kk
>> l
>> JgvITjRkFjUVPJs_9b_iDBlaYP2OsIL-LBnQJffJXQYEKj5byoQtTwxxwmgkJCNdtPPob
>> E
>> 8ocW1cD6kKfQhIkYibv3xgk4bd3tZka4c7EwYr2ugD0QzKVd8noCGdF7tdmfZqXHNg-XJ
>> s
>> scL_HVy_Jyk9iT4F3OJmPPO94YJBmQzBiLYnffQjeymz2SpFhYnrcOq52HWYu09K1aOZ5
>> q
>> 7e8lwKOk4u77uA0D64pqVBLVD35quCUkIwTWCWs6syxbvqwJx8iHPo1qg6SN6hstQmxNV
>> g
>> pZdGABF2VMaU0zGdUzBzAYNfZS74q3CmkbGWBsOWOBxP3ZbZQDfbiwFovjkY1n_svh26v
>> z
>> ubCvnjmiVvAjt-g/http%3A%2F%2Fhttp://secure-web.cisco.com/1xffTXMgG0Vt
>> WNgZbupUoRsbPPrAfrm5lifHCWFevtpUT3zeJvSOW5aaz7Ji1smzNB5Vt3pSo5tlI4iMv
>> 493fJcpD-BZHVNVWQffWPp3Zx3U9UtXnZemw6mzc_2B7THmkomCt_JkQLKAaXuakVeep1
>> z3z3Qc_Zl-qxbNsjiqAX4S1_m_Sx-UW2AZhQArme95hSYyOisK0IXxHsQ9Zb02S9Lq1Xs
>> DWpiDTGV0l94ccHi2cg4MHaV0ZUh4y6eLFIqxrn_a-W4HoFoOue20Y5CAJKZv_gCrQXu0
>> eepObC0CFWhsKiGIGYk6PAgZDXY9sE-KEQtMpbZLoGN-0abhk11n6rYJcaH2oyKlwpCWC
>> 5qcHZEFRA6P0ldZWA0-Kc80TPjG-4H_Ye6OE8u87H39pelfjzg/http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> __________________________________________
>> E - EXTERNAL SENDER
>> Use discretion when clicking links, opening attachments, or replying.
>> __________________________________________
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://secure-web.cisco.com/1Yd3N88k4kLwQnn6wm8nV_N7J2OaVx0bD2BdIWsKL
>> r
>> FE0ZaOj3W01mHVLaElEBwN-8H0WiHag1JSIym27W1OuyVKNfLmf77yYJ9Cm9CnWBOSzaA
>> e
>> 9UjGSiTaiNydxElV2Iltc8-n0H1asSF4KQUugSLV1rCOlmbvJjUX_LEYNlwfdmoIboUPD
>> M
>> taUxedojI9XGrqfeIJk2Ei_Z4mznxnej5Fe4wiWN6JTwCWh6zBK1dNGX52BnSB6s6dHOS
>> 9
>> ejGwTi09PSEQBztWHHFXKX2Naid30ed_iPiKc5i38dAiJ4bFTI0E808MttZdgwmUCv7JM
>> c
>> jfDTojUSOR0_CVK_0m6TYXm--LmFgc2Rizd305L7j3iOHuCQJmnJ47G0BhV1aLlpQpYc1
>> Y
>> pIi4K2rpXsUafcA/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2
>> F
>> rsyslog
>> http://secure-web.cisco.com/1IHJGgE_2T4pU7cw6lpFJqUNH3yko3ST-aOsNhS-r
>> n
>> tkkdSonwR7ZoNduMbgJuf3Xkhx8LpgB3iAC68nYf95aAkbLmNmxK86nUypZwYR_y6E2xj
>> m
>> GnV39qiCcdcikUxq8ShscO2fAm3PKpXBwO8jUWSyMhgHgmffHR1p28C7eHvOQ3sf_78lj
>> G
>> sHyMDHHbi0nVCqWAhXRCDNWKloDLWUIEqFIGE1jMADGyxFf7u-51jPUcNUbRQtigP2aez
>> h
>> jhm1DNftQu_kBqVJ9-yU5DQqIq8dshZZYuJgXXg-GuCawlr17h82XWWjYKv10RTFu1EVU
>> T
>> j1s6wWhVU_3iKRwq3U8uaZ8H57YjkkALH-M0OHSj6WGQda82Zoq1gNO7IMSBb3bjexSiL
>> j
>> UUXUdy0ikBSRjDg/http%3A%2F%2Fhttp://secure-web.cisco.com/1xffTXMgG0Vt
>> WNgZbupUoRsbPPrAfrm5lifHCWFevtpUT3zeJvSOW5aaz7Ji1smzNB5Vt3pSo5tlI4iMv
>> 493fJcpD-BZHVNVWQffWPp3Zx3U9UtXnZemw6mzc_2B7THmkomCt_JkQLKAaXuakVeep1
>> z3z3Qc_Zl-qxbNsjiqAX4S1_m_Sx-UW2AZhQArme95hSYyOisK0IXxHsQ9Zb02S9Lq1Xs
>> DWpiDTGV0l94ccHi2cg4MHaV0ZUh4y6eLFIqxrn_a-W4HoFoOue20Y5CAJKZv_gCrQXu0
>> eepObC0CFWhsKiGIGYk6PAgZDXY9sE-KEQtMpbZLoGN-0abhk11n6rYJcaH2oyKlwpCWC
>> 5qcHZEFRA6P0ldZWA0-Kc80TPjG-4H_Ye6OE8u87H39pelfjzg/http%3A%2F%2Fwww.r
>> syslog.com%2Fprofessional-services%2
>> F What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
> __________________________________________
> E - EXTERNAL SENDER
> Use discretion when clicking links, opening attachments, or replying.
> __________________________________________
>
>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
these buffers are small compared to the rest of your system (I think you bumped
them to 8M If I remember the units correctly), which is peanuts on your system

so enable impstats, look for rsyslog error messages in your log output, and
watch these buffer errors to see if they really go away.

Then with a little more data, we can look to tune the rsyslog.conf a bit.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
I enabled stats, by adding the following:

module(load="impstats"
interval="600"
severity="7")

# to actually gather the data:
syslog.=debug /dumps/rsyslogd/stats

I've got a bunch of data, but not really sure what to look for.
As you see I'm sampling in 10 minute intervals, and I took severity from an example provided... not sure if that was the right move, pls let me know if I should choose some other value.

I've provided the full output of the stats as an attachment.

Thanks again!

Radesh

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Thursday, August 01, 2019 8:48 PM
To: Singh, Radesh <Radesh_Singh@csx.com>
Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

these buffers are small compared to the rest of your system (I think you bumped them to 8M If I remember the units correctly), which is peanuts on your system

so enable impstats, look for rsyslog error messages in your log output, and watch these buffer errors to see if they really go away.

Then with a little more data, we can look to tune the rsyslog.conf a bit.

David Lang
Re: [E]Re: Missing messages [ In reply to ]
David,

I also see these in /var/log/messages...

Aug 1 04:55:56 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 1 04:55:56 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 1 14:18:49 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
Aug 1 14:18:49 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 1 14:18:49 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 1 14:20:23 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
Aug 1 14:20:23 lnx21648 rsyslogd: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 1 14:20:23 lnx21648 rsyslogd: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 1 14:21:26 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
Aug 1 14:21:26 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 1 14:21:26 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 1 21:14:14 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
Aug 1 21:14:14 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 1 21:14:14 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 2 04:09:59 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
Aug 2 04:09:59 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 2 04:09:59 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 2 06:49:47 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
Aug 2 06:49:47 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
Aug 2 06:49:47 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]

I need to read the info referred to in the log...

Shawn

-----Original Message-----
From: Singh, Radesh
Sent: Friday, August 02, 2019 5:19 AM
To: David Lang <david@lang.hm>
Cc: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

I enabled stats, by adding the following:

module(load="impstats"
interval="600"
severity="7")

# to actually gather the data:
syslog.=debug /dumps/rsyslogd/stats

I've got a bunch of data, but not really sure what to look for.
As you see I'm sampling in 10 minute intervals, and I took severity from an example provided... not sure if that was the right move, pls let me know if I should choose some other value.

I've provided the full output of the stats as an attachment.

Thanks again!

Radesh

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Thursday, August 01, 2019 8:48 PM
To: Singh, Radesh <Radesh_Singh@csx.com>
Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

these buffers are small compared to the rest of your system (I think you bumped them to 8M If I remember the units correctly), which is peanuts on your system

so enable impstats, look for rsyslog error messages in your log output, and watch these buffer errors to see if they really go away.

Then with a little more data, we can look to tune the rsyslog.conf a bit.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
David,

Turns out there isn't any info at the link referenced in the error message :(.

Radesh

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Singh, Radesh via rsyslog
Sent: Friday, August 02, 2019 9:48 AM
To: David Lang <david@lang.hm>
Cc: Singh, Radesh <Radesh_Singh@csx.com>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] [E]Re: Missing messages

David,

I also see these in /var/log/messages...

Aug 1 04:55:56 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1 04:55:56 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmY
XcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1 14:18:49 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://secure-web.cisco.com/19xVgvyi2VCnj9LExtq0fDrJftnNyKY4L5lN_nA3yjGfaNmrlQfL8z2E1xnrJnFnO6XeS3hVC_Yx8Kw1U97-pQO_EIU2B8Xzj7804avMc0REgPfufRxDmtTau-UFnQYx6LF2Qm9qy8i_7kXvxLW3twnDSBEFrtur3To9BY1LsiA671SmzRd0wia6cNaVBLd_4EqNJDLfT0j2i6N0rU7Q0saTwaEZ4yBvljjIgmhOF9tMXwMUQB1AOZWLRNF26NoSRqrCc1VNzg5kwf6Ddyf9sLDRb0nAQcqIUOvcsul8mcg9BhU8r-0LTs71JNGAXZTC3Pt3tf22r1-t1q8vMq0L2A0kft7pjph_7F9s48MTdwhOx8OUJyLdP5ZP82MirJkrB/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 1 14:18:49 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4
274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1 14:18:49 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1 14:20:23 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://secure-web.cisco.com/19xVgvyi2VCnj9LExtq0fDrJftnNyKY4L5lN_nA3yjGfaNmrlQfL8z2E1xnr
JnFnO6XeS3hVC_Yx8Kw1U97-pQO_EIU2B8Xzj7804avMc0REgPfufRxDmtTau-UFnQYx6LF2Qm9qy8i_7kXvxLW3twnDSBEFrtur3To9BY1LsiA671SmzRd0wia6cNaVBLd_4EqNJDLfT0j2i6N0rU7Q0saTwaEZ4yBvljjIgmhOF9tMXwMUQB1AOZWLRNF26NoSRqrCc1VNzg5kwf6Ddyf9sLDRb0nAQcqIUOvcsul8mcg9BhU8r-0LTs71JNGAXZTC3Pt3tf22r1-t1q8vMq0L2A0kft7pjph_7F9s48MTdwhOx8OUJyLdP5ZP82MirJkrB/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 1 14:20:23 lnx21648 rsyslogd: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1 14:20:23 lnx21648 rsyslogd: action 'action 0' resumed (module 'buil
tin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1 14:21:26 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://secure-web.cisco.com/19xVgvyi2VCnj9LExtq0fDrJftnNyKY4L5lN_nA3yjGfaNmrlQfL8z2E1xnrJnFnO6XeS3hVC_Yx8Kw1U97-pQO_EIU2B8Xzj7804avMc0REgPfufRxDmtTau-UFnQYx6LF2Qm9qy8i_7kXvxLW3twnDSBEFrtur3To9BY1LsiA671SmzRd0wia6cNaVBLd_4EqNJDLfT0j2i6N0rU7Q0saTwaEZ4yBvljjIgmhOF9tMXwMUQB1AOZWLRNF26NoSRqrCc1VNzg5kwf6Ddyf9sLDRb0nAQcqIUOvcsul8mcg9BhU8r-0LTs71JNGAXZTC3Pt3tf22r1-t1q8vMq0L2A0kft7pjph_7F9s48MTdwhOx8OUJyLdP5
ZP82MirJkrB/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 1 14:21:26 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1 14:21:26 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OH
n-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1 21:14:14 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://secure-web.cisco.com/19xVgvyi2VCnj9LExtq0fDrJftnNyKY4L5lN_nA3yjGfaNmrlQfL8z2E1xnrJnFnO6XeS3hVC_Yx8Kw1U97-pQO_EIU2B8Xzj7804avMc0REgPfufRxDmtTau-UFnQYx6LF2Qm9qy8i_7kXvxLW3twnDSBEFrtur3To9BY1LsiA671SmzRd0wia6cNaVBLd_4EqNJDLfT0j2i6N0rU7Q0saTwaEZ4yBvljjIgmhOF9tMXwMUQB1AOZWLRNF26NoSRqrCc1VNzg5kwf6Ddyf9sLDRb0nAQcqIUOvcsul8mcg9BhU8r-0LTs71JNGAXZTC3Pt3tf22r1-t1q8vMq0L2A0kft7pjph_7F9s48MTdwhOx8OUJyLdP5ZP82MirJkrB/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 1 21:14:14 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7k
uoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1 21:14:14 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 2 04:09:59 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://secure-web.cisco.com/19xVg
vyi2VCnj9LExtq0fDrJftnNyKY4L5lN_nA3yjGfaNmrlQfL8z2E1xnrJnFnO6XeS3hVC_Yx8Kw1U97-pQO_EIU2B8Xzj7804avMc0REgPfufRxDmtTau-UFnQYx6LF2Qm9qy8i_7kXvxLW3twnDSBEFrtur3To9BY1LsiA671SmzRd0wia6cNaVBLd_4EqNJDLfT0j2i6N0rU7Q0saTwaEZ4yBvljjIgmhOF9tMXwMUQB1AOZWLRNF26NoSRqrCc1VNzg5kwf6Ddyf9sLDRb0nAQcqIUOvcsul8mcg9BhU8r-0LTs71JNGAXZTC3Pt3tf22r1-t1q8vMq0L2A0kft7pjph_7F9s48MTdwhOx8OUJyLdP5ZP82MirJkrB/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 2 04:09:59 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 2 04:09:59 lnx
21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 2 06:49:47 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://secure-web.cisco.com/19xVgvyi2VCnj9LExtq0fDrJftnNyKY4L5lN_nA3yjGfaNmrlQfL8z2E1xnrJnFnO6XeS3hVC_Yx8Kw1U97-pQO_EIU2B8Xzj7804avMc0REgPfufRxDmtTau-UFnQYx6LF2Qm9qy8i_7kXvxLW3twnDSBEFrtur3To9BY1LsiA671SmzRd0wia6cNaVBLd_4EqNJDLfT0j2i6N0rU7Q0saTwaEZ4yBvljjIgmhOF9tMXwMUQB1AOZWLRNF26NoSRqrCc1VNzg5kwf6Ddyf9sLDRb0nAQcqIUOvcsul8mcg9BhU8r-0LTs71JNGAXZT
C3Pt3tf22r1-t1q8vMq0L2A0kft7pjph_7F9s48MTdwhOx8OUJyLdP5ZP82MirJkrB/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 2 06:49:47 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKhJH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 2 06:49:47 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://secure-web.cisco.com/1GNuOE0Q_oBxVt06bWgJYA_FwU-BalNpxQtfkRPc4dK_H09V5bcyfUzOK3LB5F9UPIr36YiqUEgRE0-qBKfW9CgGPmCUpUQ8OROF02fNA4MG3ou3uOpUPjTQ6vxGASbuYiC7kuoByCT6WfOu8TApSJZjDJ0sRGFLPQKySyoerwk8bsYpsDO3pWXusCr4274mVuAIolUIAC8oKh
JH1_pq02bBbPgtF9HVq9c0X9l83OhGWVx2A33FfOBosO2PG6xsrC8OHn-4D3ufqkPI9PqwQIOd7FZ4muva9cI3_N2PDGAVXgqbOGJqbahMNNmYXcunDrxpbtyk0qvKRMYPVsKGyGAx1CeosdcCJZf2ebt1e97LGyCUPJUwF4wk8GoDEWUR2/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ]

I need to read the info referred to in the log...

Shawn

-----Original Message-----
From: Singh, Radesh
Sent: Friday, August 02, 2019 5:19 AM
To: David Lang <david@lang.hm>
Cc: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

I enabled stats, by adding the following:

module(load="impstats"
interval="600"
severity="7")

# to actually gather the data:
syslog.=debug /dumps/rsyslogd/stats

I've got a bunch of data, but not really sure what to look for.
As you see I'm sampling in 10 minute intervals, and I took severity from an example provided... not sure if that was the right move, pls let me know if I should choose some other value.

I've provided the full output of the stats as an attachment.

Thanks again!

Radesh

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Thursday, August 01, 2019 8:48 PM
To: Singh, Radesh <Radesh_Singh@csx.com>
Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

these buffers are small compared to the rest of your system (I think you bumped them to 8M If I remember the units correctly), which is peanuts on your system

so enable impstats, look for rsyslog error messages in your log output, and watch these buffer errors to see if they really go away.

Then with a little more data, we can look to tune the rsyslog.conf a bit.

David Lang
_______________________________________________
rsyslog mailing list
http://secure-web.cisco.com/1HcFngiWSaZiYuTIaqTliQpyGZkCCco_rgvdGPQtp1SeGG9RiObC7A3FI7hP7RmyLjEhbfOkN5022J0UC8WUq0xRxidDkKdESf12IaSv6nV-EHf96n3Q_QvjUJgr59ysESBPjbTKIQlrL7Si3w7c6TO0wOFOGBBbq0fDEIei1yScAMppIHBifixXs3FVaNtZsGFjt3MSFKCrdL8XNCjNEtNZWPDyTbwAejbBl652PmAmo70XoJPYz6MRvj0edkIajS_8XZyhpPMThKDDIfDYr1R_fMzQgmTf9VmkeyhtevY_dua1tUuaC9kE0GQr4QXd8MVTCFDOKbv9ySJPRsKW_OS3dUbTamEEaNbgfaz7xbPNMedQIc_M-e5oWbDlL1THm/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
http://secure-web.cisco.com/1Z2ev_N-EpW-AlsFKzT2EtaCy1CCpRU59srPldBFcH_1uGyHfFgVgDhM36ZyFuY31oPIduymQTGZYI9_P-gFs9jrj12J-wv2_xmY3fRyYoAK5JFIG3FNfUjrWtu_a5SPJY6Sj7lNFnqmD-9-6wJ8RTkWGnwEDBgdkHv9kTLcfwJ6SsO_ZGv--SQ74AHsNS7fcgpwzvZD3cDO8fghAoBIKMVl8HXq5DEN0Kre2oVh3MHxV4drmWHKUxGTof5Vemw9wqQxk_WRxEK1sSew-8FZ8IzO3hJPp0HtVfo5If4WRVuPhQYvwfLIWdX0dRO66QL_7CAGEvvK5JNg-uFUkTyEtWAm5fBWEL4iGa4BFz_vBfVuFHlbV8AMcTs0U3M7-Cwlg/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
> Turns out there isn't any info at the link referenced in the error message :(.

I have updated the description:

https://www.rsyslog.com/rsyslog-error-2354/

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
Thanks Rainer!

-----Original Message-----
From: Rainer Gerhards <rgerhards@hq.adiscon.com>
Sent: Friday, August 02, 2019 10:16 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: David Lang <david@lang.hm>; Singh, Radesh <Radesh_Singh@csx.com>
Subject: Re: [rsyslog] [E]Re: Missing messages

> Turns out there isn't any info at the link referenced in the error message :(.

I have updated the description:

https://secure-web.cisco.com/1ljbkttuf9WCPcd5-RVxhKETv_fcYO122sXETKVpZTsRg80d2cWuWoFKkNz0DAIX9rVf9qMZwEK-oYQTJYmARYISyf-BJj7hvx0lyfyK9lsJPP-F9X_CQaGvpviY5LhCj25Ld_bb2LPGfVyiOXmCCmlFb0Sc0NkOphrb86zWXdYabKJtOm8W9WJH8S0VCMCepXSZCmupQ5c9raGkW3-4cNG5g_-Jh6iQ8RoHT7XCsfxcTbUkhFsJdeLMNOZPLJwuf60wCA1UNQB0f8bVypuPQClVEN1AzXkbLPL-gOZakNNXhHTGDkrohzu74tHle1pH4DvmkZKbTZ3lR1QjHwNrgsThU6n0SkVQUw-wqzlUOEnC4lqlsa8rz1-02RMtNcTz7/https%3A%2F%2Fwww.rsyslog.com%2Frsyslog-error-2354%2F

Rainer

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
Much better :).
Funny, my old install (8.24) is getting called out in the documentation... :P

The message above most probably indicates that the system is out of UDP buffer space. But it may have different causes (e.g. out of sockets). If you can't find an easy solution, you need to monitor your system.
Note that old versions (like 8.24) always failed to send messages close to or larger than 64KiB.

I need to figure out if I'm running out of buffer space or sockets.

I ran netstat -nap | grep ... and I see:
tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 10634/rsyslogd
tcp6 0 0 :::514 :::* LISTEN 10634/rsyslogd
udp 215040 0 0.0.0.0:514 0.0.0.0:* 10634/rsyslogd
udp6 0 0 :::514 :::* 10634/rsyslogd

That 215040 is kinda interesting, because I see it drop ... sometimes to 0, then go right back up, so guessing that's a limit of some sort.
Trying to figure out if that is the case, and if so, what tunable(s) to modify...

Radesh

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Singh, Radesh via rsyslog
Sent: Friday, August 02, 2019 10:22 AM
To: Rainer Gerhards <rgerhards@hq.adiscon.com>; rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Singh, Radesh <Radesh_Singh@csx.com>
Subject: Re: [rsyslog] [E]Re: Missing messages

Thanks Rainer!

-----Original Message-----
From: Rainer Gerhards <rgerhards@hq.adiscon.com>
Sent: Friday, August 02, 2019 10:16 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: David Lang <david@lang.hm>; Singh, Radesh <Radesh_Singh@csx.com>
Subject: Re: [rsyslog] [E]Re: Missing messages

> Turns out there isn't any info at the link referenced in the error message :(.

I have updated the description:

https://secure-web.cisco.com/1ljbkttuf9WCPcd5-RVxhKETv_fcYO122sXETKVpZTsRg80d2cWuWoFKkNz0DAIX9rVf9qMZwEK-oYQTJYmARYISyf-BJj7hvx0lyfyK9lsJPP-F9X_CQaGvpviY5LhCj25Ld_bb2LPGfVyiOXmCCmlFb0Sc0NkOphrb86zWXdYabKJtOm8W9WJH8S0VCMCepXSZCmupQ5c9raGkW3-4cNG5g_-Jh6iQ8RoHT7XCsfxcTbUkhFsJdeLMNOZPLJwuf60wCA1UNQB0f8bVypuPQClVEN1AzXkbLPL-gOZakNNXhHTGDkrohzu74tHle1pH4DvmkZKbTZ3lR1QjHwNrgsThU6n0SkVQUw-wqzlUOEnC4lqlsa8rz1-02RMtNcTz7/https%3A%2F%2Fwww.rsyslog.com%2Frsyslog-error-2354%2F

Rainer

_______________________________________________
rsyslog mailing list
http://secure-web.cisco.com/1gnrQ4eEeNMkuPib5sR-s0XNTeozGHiUiazPq4w42TYNYxQ91r9HOvO9pgDi8DMrcn7uATl5U6Lx6Kps_QNfFxMjZnc_bEKm_WrvdJvWOiaga2t3nMc8Zaw0tIGTF0qej7pYdOvh6QhZtAzumwewCjyyu1YjbB2VcJPHxQ4goXe1qtOeAQ0xdYhFUOdsZTgx63D_hwpmOzZMaCRcr4XIkRpXR6ZHtbjKvMwf0tqNOUfoo4f-0Jgky7GRN2gJivhIp4E1EaGHL8BKdmGKvP72cEuxvQ-N3Ldo3MThqDInkfauTFiYM7w0PYsQLoGfjltKQjuncRWzBdV3TgdE-n9_egI5hDDLb8pU9d6o2S55TolLhytgIGvuwpiBJ5mTL4DtJ/http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog
http://secure-web.cisco.com/1G2lziPaBr2JHycj7RSMQobPSugYu15Mt7pPD5HjW_WmTnUjbJRMFZvE9Aw0cObTaFUmsS-fx8cT4vmKjNgm89bFFqIyIuOzFTlartUpUFxffUXx1ioMZ-9DNM6-EbX-GRTAnVETsGEC3ulF3RQdIGdnl2tTL44lE3sz9TYCWSgFMdc2miNGp64ppzaDtZ4V8tI6SyY78FP2vqsV0dtaywl1oJIyME6FRA5dpdpQWRneM5GPe7TUW_wnOSZYTkTUvITluaC4vTY8jyzcnoeTQsL4juXtGqpaBN_5s7pOGFgI3-orPNglAcAIu2YZJgM8IGrEi84mME29Lhafoaned9bXG_dVKjj5fSqTwWGXzWdnLynnKrHyR_lpYE2-9BRCj/http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
El vie., 2 ago. 2019 a las 16:28, Singh, Radesh
(<Radesh_Singh@csx.com>) escribió:
>
> Much better :).
> Funny, my old install (8.24) is getting called out in the documentation... :P

unfortunately, there is a lot of 8.24 in the real world - with some
very bad bugs long solved... Thus I tend to name it whenever it makes
sense.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
On Fri, 2 Aug 2019, Rainer Gerhards via rsyslog wrote:

>> Funny, my old install (8.24) is getting called out in the documentation... :P
>
> unfortunately, there is a lot of 8.24 in the real world - with some
> very bad bugs long solved... Thus I tend to name it whenever it makes
> sense.

it's the default that RHEL/CentOS 7 ship with, and we have no idea what fixes
they backport (we know they don't backport anything close to all of them)

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
so this is indicating that you are unable to send messages as fast as you want
to, so rsyslog is periodically pausing (which causes more grief)

almost always this is going to be due to buffer space, not sockets. you only use
additional sockets for additional connections, so unless you are sending to
thousands of different machines, you aren't going to run out of sockets

increase your outbound UDP buffers and see if that clears things up.

also, double check your network utilization, just to be sure it's nowhere near
100%

David Lang


On Fri,
2 Aug 2019, Singh, Radesh wrote:

> Date: Fri, 2 Aug 2019 13:47:34 +0000
> From: "Singh, Radesh" <Radesh_Singh@csx.com>
> To: David Lang <david@lang.hm>
> Cc: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> I also see these in /var/log/messages...
>
> Aug 1 04:55:56 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 1 04:55:56 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 1 14:18:49 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
> Aug 1 14:18:49 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 1 14:18:49 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 1 14:20:23 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
> Aug 1 14:20:23 lnx21648 rsyslogd: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 1 14:20:23 lnx21648 rsyslogd: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 1 14:21:26 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
> Aug 1 14:21:26 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 1 14:21:26 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 1 21:14:14 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
> Aug 1 21:14:14 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 1 21:14:14 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 2 04:09:59 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
> Aug 2 04:09:59 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 2 04:09:59 lnx21648 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 2 06:49:47 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource temporarily unavailable [v8.24.0 try http://www.rsyslog.com/e/2354 ]
> Aug 2 06:49:47 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
> Aug 2 06:49:47 lnx21648 rsyslogd: action 'action 2' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
>
> I need to read the info referred to in the log...
>
> Shawn
>
> -----Original Message-----
> From: Singh, Radesh
> Sent: Friday, August 02, 2019 5:19 AM
> To: David Lang <david@lang.hm>
> Cc: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> I enabled stats, by adding the following:
>
> module(load="impstats"
> interval="600"
> severity="7")
>
> # to actually gather the data:
> syslog.=debug /dumps/rsyslogd/stats
>
> I've got a bunch of data, but not really sure what to look for.
> As you see I'm sampling in 10 minute intervals, and I took severity from an example provided... not sure if that was the right move, pls let me know if I should choose some other value.
>
> I've provided the full output of the stats as an attachment.
>
> Thanks again!
>
> Radesh
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Thursday, August 01, 2019 8:48 PM
> To: Singh, Radesh <Radesh_Singh@csx.com>
> Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> these buffers are small compared to the rest of your system (I think you bumped them to 8M If I remember the units correctly), which is peanuts on your system
>
> so enable impstats, look for rsyslog error messages in your log output, and watch these buffer errors to see if they really go away.
>
> Then with a little more data, we can look to tune the rsyslog.conf a bit.
>
> David Lang
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
I just patched and this is what yum info shows...

[root@lnx21648 rsyslogd]# yum info rsyslog
Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
Installed Packages
Name : rsyslog
Arch : x86_64
Version : 8.24.0
Release : 16.el7_5.4

Not sure what (or if) that corelates to in your development cycle.

The issue still persists though, trying some different things to see if can make any head way.

Radesh

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Friday, August 02, 2019 7:39 PM
To: Rainer Gerhards via rsyslog <rsyslog@lists.adiscon.com>
Cc: Singh, Radesh <Radesh_Singh@csx.com>; Rainer Gerhards <rgerhards@hq.adiscon.com>
Subject: Re: [rsyslog] [E]Re: Missing messages

On Fri, 2 Aug 2019, Rainer Gerhards via rsyslog wrote:

>> Funny, my old install (8.24) is getting called out in the
>> documentation... :P
>
> unfortunately, there is a lot of 8.24 in the real world - with some
> very bad bugs long solved... Thus I tend to name it whenever it makes
> sense.

it's the default that RHEL/CentOS 7 ship with, and we have no idea what fixes they backport (we know they don't backport anything close to all of them)

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
Outbound buffers, would that be net.core.wmem_max?

Radesh

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Friday, August 02, 2019 7:41 PM
To: Singh, Radesh <Radesh_Singh@csx.com>
Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

so this is indicating that you are unable to send messages as fast as you want to, so rsyslog is periodically pausing (which causes more grief)

almost always this is going to be due to buffer space, not sockets. you only use additional sockets for additional connections, so unless you are sending to thousands of different machines, you aren't going to run out of sockets

increase your outbound UDP buffers and see if that clears things up.

also, double check your network utilization, just to be sure it's nowhere near 100%

David Lang


On Fri,
2 Aug 2019, Singh, Radesh wrote:

> Date: Fri, 2 Aug 2019 13:47:34 +0000
> From: "Singh, Radesh" <Radesh_Singh@csx.com>
> To: David Lang <david@lang.hm>
> Cc: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> David,
>
> I also see these in /var/log/messages...
>
> Aug 1 04:55:56 lnx21648 rsyslogd: action 'action 1' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1
> 04:55:56 lnx21648 rsyslogd: action 'action 1' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1
> 14:18:49 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource
> temporarily unavailable [v8.24.0 try
> http://secure-web.cisco.com/1PXboNJKwZC09dSvokjXCLZBLbv0VQ5VR-qZAEjKeW
> z-2HBYNARTcCBhCoWVEpambEPfpcUYg0qurT3j7BgkPBLAVw9Qeytsy5tavcsgKSBl2gVB
> UmDeaG4sQ5fpqE7fdxWtEMAKWjQlfYQMhdjDy8lfqRSBChhEZfjvufNKR3_cHvnBwtWpoW
> kKghLJXBM3MF14oJ-gUmYDeBKpLDYrZvjCha79K9IZq3tQXDIjmJWmoCVgBbhkdDCGebLv
> S7wqdibNEdg1YjAi3HVgM4QVYALixZR64n7u3QtljXqMWXf7pcNy1JbHrqxknMdyVbSF8K
> jqd3qWMgpiChXn1EY_29k6n0m_kZyvU-Np_FlcQgrMO9FFyNfCsrlVUBy2nN3NYy2p_VeP
> eWdtBs0GgufoefQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 1
> 14:18:49 lnx21648 rsyslogd: action 'action 1' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1
> 14:18:49 lnx21648 rsyslogd: action 'action 1' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1
> 14:20:23 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource
> temporarily unavailable [v8.24.0 try
> http://secure-web.cisco.com/1PXboNJKwZC09dSvokjXCLZBLbv0VQ5VR-qZAEjKeW
> z-2HBYNARTcCBhCoWVEpambEPfpcUYg0qurT3j7BgkPBLAVw9Qeytsy5tavcsgKSBl2gVB
> UmDeaG4sQ5fpqE7fdxWtEMAKWjQlfYQMhdjDy8lfqRSBChhEZfjvufNKR3_cHvnBwtWpoW
> kKghLJXBM3MF14oJ-gUmYDeBKpLDYrZvjCha79K9IZq3tQXDIjmJWmoCVgBbhkdDCGebLv
> S7wqdibNEdg1YjAi3HVgM4QVYALixZR64n7u3QtljXqMWXf7pcNy1JbHrqxknMdyVbSF8K
> jqd3qWMgpiChXn1EY_29k6n0m_kZyvU-Np_FlcQgrMO9FFyNfCsrlVUBy2nN3NYy2p_VeP
> eWdtBs0GgufoefQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 1
> 14:20:23 lnx21648 rsyslogd: action 'action 0' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1
> 14:20:23 lnx21648 rsyslogd: action 'action 0' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1
> 14:21:26 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource
> temporarily unavailable [v8.24.0 try
> http://secure-web.cisco.com/1PXboNJKwZC09dSvokjXCLZBLbv0VQ5VR-qZAEjKeW
> z-2HBYNARTcCBhCoWVEpambEPfpcUYg0qurT3j7BgkPBLAVw9Qeytsy5tavcsgKSBl2gVB
> UmDeaG4sQ5fpqE7fdxWtEMAKWjQlfYQMhdjDy8lfqRSBChhEZfjvufNKR3_cHvnBwtWpoW
> kKghLJXBM3MF14oJ-gUmYDeBKpLDYrZvjCha79K9IZq3tQXDIjmJWmoCVgBbhkdDCGebLv
> S7wqdibNEdg1YjAi3HVgM4QVYALixZR64n7u3QtljXqMWXf7pcNy1JbHrqxknMdyVbSF8K
> jqd3qWMgpiChXn1EY_29k6n0m_kZyvU-Np_FlcQgrMO9FFyNfCsrlVUBy2nN3NYy2p_VeP
> eWdtBs0GgufoefQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 1
> 14:21:26 lnx21648 rsyslogd: action 'action 1' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1
> 14:21:26 lnx21648 rsyslogd: action 'action 1' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1
> 21:14:14 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource
> temporarily unavailable [v8.24.0 try
> http://secure-web.cisco.com/1PXboNJKwZC09dSvokjXCLZBLbv0VQ5VR-qZAEjKeW
> z-2HBYNARTcCBhCoWVEpambEPfpcUYg0qurT3j7BgkPBLAVw9Qeytsy5tavcsgKSBl2gVB
> UmDeaG4sQ5fpqE7fdxWtEMAKWjQlfYQMhdjDy8lfqRSBChhEZfjvufNKR3_cHvnBwtWpoW
> kKghLJXBM3MF14oJ-gUmYDeBKpLDYrZvjCha79K9IZq3tQXDIjmJWmoCVgBbhkdDCGebLv
> S7wqdibNEdg1YjAi3HVgM4QVYALixZR64n7u3QtljXqMWXf7pcNy1JbHrqxknMdyVbSF8K
> jqd3qWMgpiChXn1EY_29k6n0m_kZyvU-Np_FlcQgrMO9FFyNfCsrlVUBy2nN3NYy2p_VeP
> eWdtBs0GgufoefQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 1
> 21:14:14 lnx21648 rsyslogd: action 'action 2' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 1
> 21:14:14 lnx21648 rsyslogd: action 'action 2' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 2
> 04:09:59 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource
> temporarily unavailable [v8.24.0 try
> http://secure-web.cisco.com/1PXboNJKwZC09dSvokjXCLZBLbv0VQ5VR-qZAEjKeW
> z-2HBYNARTcCBhCoWVEpambEPfpcUYg0qurT3j7BgkPBLAVw9Qeytsy5tavcsgKSBl2gVB
> UmDeaG4sQ5fpqE7fdxWtEMAKWjQlfYQMhdjDy8lfqRSBChhEZfjvufNKR3_cHvnBwtWpoW
> kKghLJXBM3MF14oJ-gUmYDeBKpLDYrZvjCha79K9IZq3tQXDIjmJWmoCVgBbhkdDCGebLv
> S7wqdibNEdg1YjAi3HVgM4QVYALixZR64n7u3QtljXqMWXf7pcNy1JbHrqxknMdyVbSF8K
> jqd3qWMgpiChXn1EY_29k6n0m_kZyvU-Np_FlcQgrMO9FFyNfCsrlVUBy2nN3NYy2p_VeP
> eWdtBs0GgufoefQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 2
> 04:09:59 lnx21648 rsyslogd: action 'action 1' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 2
> 04:09:59 lnx21648 rsyslogd: action 'action 1' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 2
> 06:49:47 lnx21648 rsyslogd: omfwd: error 11 sending via udp: Resource
> temporarily unavailable [v8.24.0 try
> http://secure-web.cisco.com/1PXboNJKwZC09dSvokjXCLZBLbv0VQ5VR-qZAEjKeW
> z-2HBYNARTcCBhCoWVEpambEPfpcUYg0qurT3j7BgkPBLAVw9Qeytsy5tavcsgKSBl2gVB
> UmDeaG4sQ5fpqE7fdxWtEMAKWjQlfYQMhdjDy8lfqRSBChhEZfjvufNKR3_cHvnBwtWpoW
> kKghLJXBM3MF14oJ-gUmYDeBKpLDYrZvjCha79K9IZq3tQXDIjmJWmoCVgBbhkdDCGebLv
> S7wqdibNEdg1YjAi3HVgM4QVYALixZR64n7u3QtljXqMWXf7pcNy1JbHrqxknMdyVbSF8K
> jqd3qWMgpiChXn1EY_29k6n0m_kZyvU-Np_FlcQgrMO9FFyNfCsrlVUBy2nN3NYy2p_VeP
> eWdtBs0GgufoefQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2354 ] Aug 2
> 06:49:47 lnx21648 rsyslogd: action 'action 2' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ] Aug 2
> 06:49:47 lnx21648 rsyslogd: action 'action 2' resumed (module
> 'builtin:omfwd') [v8.24.0 try
> http://secure-web.cisco.com/1mIbPnsiMC10WWo4g4iaktgcASi6rwnjQK9p0KKZPR
> Q1Mv5lOU-xOGuQF8CYjrqwAImfwTcV94r14xhIlQvCbaUSo-RNstb3tlegcPoLsEOIsTRO
> ZuzYg6BrZxTxSXIgpUxMXK53pmqdvU6bLp6XxrqXWDrXyG_XlNIWvgjVdkAp_gmAvef467
> PWil0_LB7KCSSHasj0xlUdWs5NN7gGlE6KJHlajy1hUUmiT8a-7_rU-avW9hXN8zdrg1Rq
> c5LE0krXQkVNq-GO1d8yuhv1BUZS6yfj9-MZ_xwTDsZ3g8Vns55aWpwVaU72W7VkDXlcrg
> THYrdHBvnGL66q1VJsNeXT92kto2-BXGgtAHBeqSwzyBZJs0f4mfNmHd2p00m0wgJwTRNs
> ZMM12n0nUWGLsAQ/http%3A%2F%2Fwww.rsyslog.com%2Fe%2F2359 ]
>
> I need to read the info referred to in the log...
>
> Shawn
>
> -----Original Message-----
> From: Singh, Radesh
> Sent: Friday, August 02, 2019 5:19 AM
> To: David Lang <david@lang.hm>
> Cc: Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> I enabled stats, by adding the following:
>
> module(load="impstats"
> interval="600"
> severity="7")
>
> # to actually gather the data:
> syslog.=debug /dumps/rsyslogd/stats
>
> I've got a bunch of data, but not really sure what to look for.
> As you see I'm sampling in 10 minute intervals, and I took severity from an example provided... not sure if that was the right move, pls let me know if I should choose some other value.
>
> I've provided the full output of the stats as an attachment.
>
> Thanks again!
>
> Radesh
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Thursday, August 01, 2019 8:48 PM
> To: Singh, Radesh <Radesh_Singh@csx.com>
> Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog
> <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> these buffers are small compared to the rest of your system (I think
> you bumped them to 8M If I remember the units correctly), which is
> peanuts on your system
>
> so enable impstats, look for rsyslog error messages in your log output, and watch these buffer errors to see if they really go away.
>
> Then with a little more data, we can look to tune the rsyslog.conf a bit.
>
> David Lang
>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
On Fri, 2 Aug 2019, Singh, Radesh wrote:

> I just patched and this is what yum info shows...
>
> [root@lnx21648 rsyslogd]# yum info rsyslog
> Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-manager
> This system is receiving updates from RHN Classic or Red Hat Satellite.
> Installed Packages
> Name : rsyslog
> Arch : x86_64
> Version : 8.24.0
> Release : 16.el7_5.4
>
> Not sure what (or if) that corelates to in your development cycle.

it really doesn't.

RedHat took the 8.24 that we released about the end of 2016 (nov/dec IIRC) and
they have made 16 updates to it, with changes they developed or backported from
our updates.

In the meantime, every 6 weeks we made a new release up through 8.40 at the end
of 2018, and then changed our version numbering scheme to be 8.YYMM showing the
date of the release rather than just a sequence. we released 8.1907 on July 2
and will release 8.1908 on August 13

> The issue still persists though, trying some different things to see if can make any head way.

to get onto a version we can really support, you would need to update to one of
the 8.19* releases.

But before you worry about that, we have other system tuning to do (buffers
being first)

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
On Fri, 2 Aug 2019, Singh, Radesh wrote:

> Outbound buffers, would that be net.core.wmem_max?

yes, and I think it's in bytes, set is something nice and large, several tens of
MB

from a random web page, for 25MB 26214400

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
:) I went big...
sysctl net.core.wmem_max
net.core.wmem_max = 134217728

then cycled rsyslog.


-----Original Message-----
From: David Lang <david@lang.hm>
Sent: Friday, August 02, 2019 7:54 PM
To: Singh, Radesh <Radesh_Singh@csx.com>
Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
Subject: RE: [E]Re: [rsyslog] Missing messages

On Fri, 2 Aug 2019, Singh, Radesh wrote:

> Outbound buffers, would that be net.core.wmem_max?

yes, and I think it's in bytes, set is something nice and large, several tens of MB

from a random web page, for 25MB 26214400

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
so, one cycle of stats

2019-08-01T19:31:30.625929-04:00 lnx21648 rsyslogd-pstats: global:
origin=dynstats

2019-08-01T19:31:30.625943-04:00 lnx21648 rsyslogd-pstats: imuxsock:
origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0

no stats from /dev/log

2019-08-01T19:31:30.625946-04:00 lnx21648 rsyslogd-pstats: action 0:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625947-04:00 lnx21648 rsyslogd-pstats: action 1:
origin=core.action processed=2444154 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625949-04:00 lnx21648 rsyslogd-pstats: action 2:
origin=core.action processed=2444154 failed=0 suspended=0 suspended.duration=0
resumed=0

action # are the outputs (with the new action() syntax you can give them a name,
otherwise you just get a numbe to show where they are in the file)

processed is the number of logs written, if syspended/resumed are not zero, you
have had problems (should never happen when writing to files, but from your
other e-mail, you do have problems sending logs out)

note that message counts default to being running totals, so this is 2444154
since you started rsyslog, not this cycle


2019-08-01T19:31:30.625957-04:00 lnx21648 rsyslogd-pstats: dynafile cache
RemoteClient: origin=omfile requests=3583721 level0=1938992 missed=61922
evicted=61912 maxused=10 closetimeouts=0

you have an output writing to files using a template for the filename, but you
have the cache size set to 10, so
every time it tries to open a new files, it has to close an existing file (this
happened 61922 times in this cycle). This is a _really_ bad thing, you need to
increase the dynafilecachesize to something well over the number of files you
are writing to. change this action to the action() syntax to make it clear
what's happening here.

2019-08-01T19:31:30.625959-04:00 lnx21648 rsyslogd-pstats: action 3:
origin=core.action processed=3584143 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625961-04:00 lnx21648 rsyslogd-pstats: dynafile cache
RemoteClient: origin=omfile requests=0 level0=0 missed=0 evicted=0 maxused=0
closetimeouts=0
2019-08-01T19:31:30.625962-04:00 lnx21648 rsyslogd-pstats: action 4:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625963-04:00 lnx21648 rsyslogd-pstats: action 5:
origin=core.action processed=7 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625964-04:00 lnx21648 rsyslogd-pstats: action 6:
origin=core.action processed=4 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625967-04:00 lnx21648 rsyslogd-pstats: action 7:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650614-04:00 lnx21648 rsyslogd-pstats: action 8:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650617-04:00 lnx21648 rsyslogd-pstats: action 9:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650620-04:00 lnx21648 rsyslogd-pstats: action 10:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650623-04:00 lnx21648 rsyslogd-pstats: action 11:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650624-04:00 lnx21648 rsyslogd-pstats: action 12:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650629-04:00 lnx21648 rsyslogd-pstats: action 13:
origin=core.action processed=7 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650633-04:00 lnx21648 rsyslogd-pstats: action 14:
origin=core.action processed=4 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650634-04:00 lnx21648 rsyslogd-pstats: action 15:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650635-04:00 lnx21648 rsyslogd-pstats: action 16:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0

more actions

2019-08-01T19:31:30.650646-04:00 lnx21648 rsyslogd-pstats: imudp(*:514):
origin=imudp submitted=3683624

you recei ved 3683624 messages via UDP since rsyslog started

2019-08-01T19:31:30.650647-04:00 lnx21648 rsyslogd-pstats: imudp(*:514):
origin=imudp submitted=0
2019-08-01T19:31:30.650648-04:00 lnx21648 rsyslogd-pstats: imtcp(514):
origin=imtcp submitted=0

2019-08-01T19:31:30.650657-04:00 lnx21648 rsyslogd-pstats: resource-usage:
origin=impstats utime=29638410 stime=66783415 maxrss=103028 minflt=237512
majflt=0 inblock=0 oublock=1783768 nvcsw=3306119 nivcsw=18216

this is how much ram you are using and how much cpu you used

2019-08-01T19:31:30.650659-04:00 lnx21648 rsyslogd-pstats: main Q:
origin=core.queue size=99758 enqueued=3683657 full=12831 discarded.full=0
discarded.nf=0 maxqsize=100000

your main queue is currently at 99758 items, you received a total of 3683657 new
mesages, you were full 12831 times, and your max q size is the default 100000

normally you would want to fix this fast, but fix the dynafilecache size above
first and then let's see what happens

2019-08-01T19:31:30.650661-04:00 lnx21648 rsyslogd-pstats: imudp(w0):
origin=imudp called.recvmmsg=3397708 called.recvmsg=0 msgs.received=3683624

more stats about udp messages that you recevied

David Lang

On Fri, 2 Aug 2019, Singh, Radesh wrote:

> Date: Fri, 2 Aug 2019 09:18:35 +0000
> From: "Singh, Radesh" <Radesh_Singh@csx.com>
> To: David Lang <david@lang.hm>
> Cc: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> I enabled stats, by adding the following:
>
> module(load="impstats"
> interval="600"
> severity="7")
>
> # to actually gather the data:
> syslog.=debug /dumps/rsyslogd/stats
>
> I've got a bunch of data, but not really sure what to look for.
> As you see I'm sampling in 10 minute intervals, and I took severity from an example provided... not sure if that was the right move, pls let me know if I should choose some other value.
>
> I've provided the full output of the stats as an attachment.
>
> Thanks again!
>
> Radesh
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Thursday, August 01, 2019 8:48 PM
> To: Singh, Radesh <Radesh_Singh@csx.com>
> Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> these buffers are small compared to the rest of your system (I think you bumped them to 8M If I remember the units correctly), which is peanuts on your system
>
> so enable impstats, look for rsyslog error messages in your log output, and watch these buffer errors to see if they really go away.
>
> Then with a little more data, we can look to tune the rsyslog.conf a bit.
>
> David Lang
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [E]Re: Missing messages [ In reply to ]
fix the dynafilecachesize, that is probably why you are loosing your inbound
logs.

this should help keep you from loosing your outbound logs

David Lang

On Fri, 2 Aug 2019, Singh, Radesh wrote:

> Date: Fri, 2 Aug 2019 23:59:21 +0000
> From: "Singh, Radesh" <Radesh_Singh@csx.com>
> To: David Lang <david@lang.hm>
> Cc: "Singh, Radesh via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> :) I went big...
> sysctl net.core.wmem_max
> net.core.wmem_max = 134217728
>
> then cycled rsyslog.
>
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: Friday, August 02, 2019 7:54 PM
> To: Singh, Radesh <Radesh_Singh@csx.com>
> Cc: David Lang <david@lang.hm>; Singh, Radesh via rsyslog <rsyslog@lists.adiscon.com>
> Subject: RE: [E]Re: [rsyslog] Missing messages
>
> On Fri, 2 Aug 2019, Singh, Radesh wrote:
>
>> Outbound buffers, would that be net.core.wmem_max?
>
> yes, and I think it's in bytes, set is something nice and large, several tens of MB
>
> from a random web page, for 25MB 26214400
>
> David Lang
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

1 2  View All