Mailing List Archive

Rsyslog freezes the box when can't send logs over TCP
Hi all!

Recently we experienced a "nice" lock up of our web nodes that do
remote TCP logging to the centralized syslog server.

Central rsyslog server was shut down due the maintenance for few
hours. Very soon first Web servers with nginx/php-fpm started to fail
with timeouts and later access to the nodes themselves over ssh
stopped working, making boxes unaccessible. With the iLO it was
possible to log to the consoles of the nodes, but no problems were
spotted - memory, CPU, TCP sockets footprints were low, everything
looked normal.

Educated guess that it may be related to the rsyslog, as it was the
only change in the servers configuration on that day and killing
rsyslogd on the nodes brought machines back to normal.

A bit of googling brought up this story:

http://blog.bitbucket.org/2012/01/12/follow-up-on-our-downtime-last-week/

as well as a last year conversation on this ML:

http://lists.adiscon.net/pipermail/rsyslog/2011-October/013944.html

as well as a bug report in RedHat Bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=519201

Switching to UDP transport instead of TCP cured the problem and
running simple test with:

# for each n in `seq 1 1000`; do echo $n; echo "Syslog test $n" | logger; done

showed that with TCP remote logging enabled and absent remote log
server after first 250-300 messages all the following messages were
sent to syslog very slowly, with the 1-2sec delay each.

All nodes are running Debian 6(squeeze) with:

rsyslogd 5.8.11, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: No
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No

And configuration:

$MaxMessageSize 8k
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support

# Store PID of the process in the log
$SystemLogUsePIDFromSystem on
# Rate limit for imuxsock
$SystemLogRateLimitInterval 1
$SystemLogRateLimitBurst 500

$WorkDirectory /var/spool/rsyslog

$ModLoad imfile
$InputFilePollInterval 5

$InputFileName /var/log/nginx/access.log
$InputFilePersistStateInterval 100
$InputFileTag nginx/access:
$InputFileStateFile nginx_access_log_state
$InputFileFacility local7
$InputFileSeverity notice
$InputRunFileMonitor

$ActionQueueType LinkedList # enable a
separate queue for this action
$ActionQueueFileName remote # set file name,
also enables disk mode
$ActionResumeRetryCount -1 # infinite
retries on insert failure
$ActionQueueSaveOnShutdown on
*.* @@10.0.0.200

I went through ChangeLog on the site for the versions of rsyslogd
following 5.8.11, but didn't notice any fix to that problem. Neither
there is an open ticket in bugzilla.

Interesting enough that RedHat claims they fixed the problem in version 3.22:

http://rhn.redhat.com/errata/RHBA-2010-0213.html

Can you point me to the version where this problem is fixed? Or, if
it's not fixed yet - can it be done?

BTW, not sure how this would behave with REPL protocol instead of TCP...

with best regards,
Timur Bakeyev.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
Long story short: the problem is with your config. You configured rsyslog to never lose any message, at some point it runs out of queue space, then the inputs begin to block, so the log socket blocks and finally the apps that write to the log socket.
There are many options to control under which circumstances you permit message loss, and you need to make sure you know what you want.
The current versions also have lower enq timeout, which can kind of solve this config issue, at least for systems that are not super-busy. Note that it is not really an issue, but may be intended behavior in some environments (e.g. banking, when you absolutely need to log).

Rainer

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
> Sent: Tuesday, November 06, 2012 3:26 PM
> To: rsyslog@lists.adiscon.com
> Cc: Michael Biebl
> Subject: [rsyslog] Rsyslog freezes the box when can't send logs over
> TCP
>
> Hi all!
>
> Recently we experienced a "nice" lock up of our web nodes that do
> remote TCP logging to the centralized syslog server.
>
> Central rsyslog server was shut down due the maintenance for few
> hours. Very soon first Web servers with nginx/php-fpm started to fail
> with timeouts and later access to the nodes themselves over ssh
> stopped working, making boxes unaccessible. With the iLO it was
> possible to log to the consoles of the nodes, but no problems were
> spotted - memory, CPU, TCP sockets footprints were low, everything
> looked normal.
>
> Educated guess that it may be related to the rsyslog, as it was the
> only change in the servers configuration on that day and killing
> rsyslogd on the nodes brought machines back to normal.
>
> A bit of googling brought up this story:
>
> http://blog.bitbucket.org/2012/01/12/follow-up-on-our-downtime-last-
> week/
>
> as well as a last year conversation on this ML:
>
> http://lists.adiscon.net/pipermail/rsyslog/2011-October/013944.html
>
> as well as a bug report in RedHat Bugzilla:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=519201
>
> Switching to UDP transport instead of TCP cured the problem and
> running simple test with:
>
> # for each n in `seq 1 1000`; do echo $n; echo "Syslog test $n" |
> logger; done
>
> showed that with TCP remote logging enabled and absent remote log
> server after first 250-300 messages all the following messages were
> sent to syslog very slowly, with the 1-2sec delay each.
>
> All nodes are running Debian 6(squeeze) with:
>
> rsyslogd 5.8.11, compiled with:
> FEATURE_REGEXP: Yes
> FEATURE_LARGEFILE: No
> GSSAPI Kerberos 5 support: Yes
> FEATURE_DEBUG (debug build, slow code): No
> 32bit Atomic operations supported: Yes
> 64bit Atomic operations supported: Yes
> Runtime Instrumentation (slow code): No
>
> And configuration:
>
> $MaxMessageSize 8k
> $ModLoad imuxsock # provides support for local system logging
> $ModLoad imklog # provides kernel logging support
>
> # Store PID of the process in the log
> $SystemLogUsePIDFromSystem on
> # Rate limit for imuxsock
> $SystemLogRateLimitInterval 1
> $SystemLogRateLimitBurst 500
>
> $WorkDirectory /var/spool/rsyslog
>
> $ModLoad imfile
> $InputFilePollInterval 5
>
> $InputFileName /var/log/nginx/access.log
> $InputFilePersistStateInterval 100
> $InputFileTag nginx/access:
> $InputFileStateFile nginx_access_log_state
> $InputFileFacility local7
> $InputFileSeverity notice
> $InputRunFileMonitor
>
> $ActionQueueType LinkedList # enable a
> separate queue for this action
> $ActionQueueFileName remote # set file name,
> also enables disk mode
> $ActionResumeRetryCount -1 # infinite
> retries on insert failure
> $ActionQueueSaveOnShutdown on
> *.* @@10.0.0.200
>
> I went through ChangeLog on the site for the versions of rsyslogd
> following 5.8.11, but didn't notice any fix to that problem. Neither
> there is an open ticket in bugzilla.
>
> Interesting enough that RedHat claims they fixed the problem in version
> 3.22:
>
> http://rhn.redhat.com/errata/RHBA-2010-0213.html
>
> Can you point me to the version where this problem is fixed? Or, if
> it's not fixed yet - can it be done?
>
> BTW, not sure how this would behave with REPL protocol instead of
> TCP...
>
> with best regards,
> Timur Bakeyev.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
Hi, Rainer!

Of course you know internals of rsyslog better, so can't argue with you
here. But as for the end user such behavior for me looks undesired. What
I'd expect:

1. All local(imuxsock) go to the local log files.
2. All remote messages would be spooled in an $ActionQueueFileName file.
3. Nothing would block

I also found this old discussion:

http://www.mail-archive.com/rsyslog@lists.adiscon.com/msg01330.html

So you tried to resolve it in 3.x series, but, apparently the problem is
still there.

DO I understand correctly that the reason is in the fact that Output queue
for local files and remote TCP is shared, so until confirmation from both
backends is received the message isn't discarded?

Is there a way to spit those queues?

Or, can you suggest the configuration that does, what I've described? Any
change if TCP would be replaced with RELP?

With regards,
Timur Bakeyev.

On Tue, Nov 6, 2012 at 3:37 PM, Rainer Gerhards <rgerhards@hq.adiscon.com>wrote:

> Long story short: the problem is with your config. You configured rsyslog
> to never lose any message, at some point it runs out of queue space, then
> the inputs begin to block, so the log socket blocks and finally the apps
> that write to the log socket.
> There are many options to control under which circumstances you permit
> message loss, and you need to make sure you know what you want.
> The current versions also have lower enq timeout, which can kind of solve
> this config issue, at least for systems that are not super-busy. Note that
> it is not really an issue, but may be intended behavior in some
> environments (e.g. banking, when you absolutely need to log).
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> > bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
> > Sent: Tuesday, November 06, 2012 3:26 PM
> > To: rsyslog@lists.adiscon.com
> > Cc: Michael Biebl
> > Subject: [rsyslog] Rsyslog freezes the box when can't send logs over
> > TCP
> >
> > Hi all!
> >
> > Recently we experienced a "nice" lock up of our web nodes that do
> > remote TCP logging to the centralized syslog server.
> >
> > Central rsyslog server was shut down due the maintenance for few
> > hours. Very soon first Web servers with nginx/php-fpm started to fail
> > with timeouts and later access to the nodes themselves over ssh
> > stopped working, making boxes unaccessible. With the iLO it was
> > possible to log to the consoles of the nodes, but no problems were
> > spotted - memory, CPU, TCP sockets footprints were low, everything
> > looked normal.
> >
> > Educated guess that it may be related to the rsyslog, as it was the
> > only change in the servers configuration on that day and killing
> > rsyslogd on the nodes brought machines back to normal.
> >
> > A bit of googling brought up this story:
> >
> > http://blog.bitbucket.org/2012/01/12/follow-up-on-our-downtime-last-
> > week/
> >
> > as well as a last year conversation on this ML:
> >
> > http://lists.adiscon.net/pipermail/rsyslog/2011-October/013944.html
> >
> > as well as a bug report in RedHat Bugzilla:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=519201
> >
> > Switching to UDP transport instead of TCP cured the problem and
> > running simple test with:
> >
> > # for each n in `seq 1 1000`; do echo $n; echo "Syslog test $n" |
> > logger; done
> >
> > showed that with TCP remote logging enabled and absent remote log
> > server after first 250-300 messages all the following messages were
> > sent to syslog very slowly, with the 1-2sec delay each.
> >
> > All nodes are running Debian 6(squeeze) with:
> >
> > rsyslogd 5.8.11, compiled with:
> > FEATURE_REGEXP: Yes
> > FEATURE_LARGEFILE: No
> > GSSAPI Kerberos 5 support: Yes
> > FEATURE_DEBUG (debug build, slow code): No
> > 32bit Atomic operations supported: Yes
> > 64bit Atomic operations supported: Yes
> > Runtime Instrumentation (slow code): No
> >
> > And configuration:
> >
> > $MaxMessageSize 8k
> > $ModLoad imuxsock # provides support for local system logging
> > $ModLoad imklog # provides kernel logging support
> >
> > # Store PID of the process in the log
> > $SystemLogUsePIDFromSystem on
> > # Rate limit for imuxsock
> > $SystemLogRateLimitInterval 1
> > $SystemLogRateLimitBurst 500
> >
> > $WorkDirectory /var/spool/rsyslog
> >
> > $ModLoad imfile
> > $InputFilePollInterval 5
> >
> > $InputFileName /var/log/nginx/access.log
> > $InputFilePersistStateInterval 100
> > $InputFileTag nginx/access:
> > $InputFileStateFile nginx_access_log_state
> > $InputFileFacility local7
> > $InputFileSeverity notice
> > $InputRunFileMonitor
> >
> > $ActionQueueType LinkedList # enable a
> > separate queue for this action
> > $ActionQueueFileName remote # set file name,
> > also enables disk mode
> > $ActionResumeRetryCount -1 # infinite
> > retries on insert failure
> > $ActionQueueSaveOnShutdown on
> > *.* @@10.0.0.200
> >
> > I went through ChangeLog on the site for the versions of rsyslogd
> > following 5.8.11, but didn't notice any fix to that problem. Neither
> > there is an open ticket in bugzilla.
> >
> > Interesting enough that RedHat claims they fixed the problem in version
> > 3.22:
> >
> > http://rhn.redhat.com/errata/RHBA-2010-0213.html
> >
> > Can you point me to the version where this problem is fixed? Or, if
> > it's not fixed yet - can it be done?
> >
> > BTW, not sure how this would behave with REPL protocol instead of
> > TCP...
> >
> > with best regards,
> > Timur Bakeyev.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
> Sent: Wednesday, November 07, 2012 2:27 PM
> To: rsyslog-users
> Cc: Michael Biebl
> Subject: Re: [rsyslog] Rsyslog freezes the box when can't send logs
> over TCP
>
> Hi, Rainer!
>
> Of course you know internals of rsyslog better, so can't argue with you
> here. But as for the end user such behavior for me looks undesired.
> What
> I'd expect:
>
> 1. All local(imuxsock) go to the local log files.
> 2. All remote messages would be spooled in an $ActionQueueFileName
> file.
> 3. Nothing would block

Yup - so why not configure it that way?

Actually, very reliable logging is a tough consulting job. It's nothing you do in the lunch break.
>
> I also found this old discussion:
>
> http://www.mail-archive.com/rsyslog@lists.adiscon.com/msg01330.html
>
> So you tried to resolve it in 3.x series, but, apparently the problem
> is
> still there.
There is NO Problem. There may have been one in v3, I don't remember - too long ago.
>
> DO I understand correctly that the reason is in the fact that Output
> queue
> for local files and remote TCP is shared, so until confirmation from
> both
> backends is received the message isn't discarded?
>
> Is there a way to spit those queues?
>
> Or, can you suggest the configuration that does, what I've described?
> Any
> change if TCP would be replaced with RELP?
>
You need to read up the in-depth description on how queues work. You need to know when it is OK to lose messages. Then, you need to configure the system to do that. The root cause is not the protocol, but your desired to have "nothing lost" but not thought what should happen if you run out of resources. Again, this is a though question and doing it right is a real consulting job.

Rainer

Rainer
> With regards,
> Timur Bakeyev.
>
> On Tue, Nov 6, 2012 at 3:37 PM, Rainer Gerhards
> <rgerhards@hq.adiscon.com>wrote:
>
> > Long story short: the problem is with your config. You configured
> rsyslog
> > to never lose any message, at some point it runs out of queue space,
> then
> > the inputs begin to block, so the log socket blocks and finally the
> apps
> > that write to the log socket.
> > There are many options to control under which circumstances you
> permit
> > message loss, and you need to make sure you know what you want.
> > The current versions also have lower enq timeout, which can kind of
> solve
> > this config issue, at least for systems that are not super-busy. Note
> that
> > it is not really an issue, but may be intended behavior in some
> > environments (e.g. banking, when you absolutely need to log).
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> > > bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
> > > Sent: Tuesday, November 06, 2012 3:26 PM
> > > To: rsyslog@lists.adiscon.com
> > > Cc: Michael Biebl
> > > Subject: [rsyslog] Rsyslog freezes the box when can't send logs
> over
> > > TCP
> > >
> > > Hi all!
> > >
> > > Recently we experienced a "nice" lock up of our web nodes that do
> > > remote TCP logging to the centralized syslog server.
> > >
> > > Central rsyslog server was shut down due the maintenance for few
> > > hours. Very soon first Web servers with nginx/php-fpm started to
> fail
> > > with timeouts and later access to the nodes themselves over ssh
> > > stopped working, making boxes unaccessible. With the iLO it was
> > > possible to log to the consoles of the nodes, but no problems were
> > > spotted - memory, CPU, TCP sockets footprints were low, everything
> > > looked normal.
> > >
> > > Educated guess that it may be related to the rsyslog, as it was the
> > > only change in the servers configuration on that day and killing
> > > rsyslogd on the nodes brought machines back to normal.
> > >
> > > A bit of googling brought up this story:
> > >
> > > http://blog.bitbucket.org/2012/01/12/follow-up-on-our-downtime-
> last-
> > > week/
> > >
> > > as well as a last year conversation on this ML:
> > >
> > > http://lists.adiscon.net/pipermail/rsyslog/2011-October/013944.html
> > >
> > > as well as a bug report in RedHat Bugzilla:
> > >
> > > https://bugzilla.redhat.com/show_bug.cgi?id=519201
> > >
> > > Switching to UDP transport instead of TCP cured the problem and
> > > running simple test with:
> > >
> > > # for each n in `seq 1 1000`; do echo $n; echo "Syslog test $n" |
> > > logger; done
> > >
> > > showed that with TCP remote logging enabled and absent remote log
> > > server after first 250-300 messages all the following messages were
> > > sent to syslog very slowly, with the 1-2sec delay each.
> > >
> > > All nodes are running Debian 6(squeeze) with:
> > >
> > > rsyslogd 5.8.11, compiled with:
> > > FEATURE_REGEXP: Yes
> > > FEATURE_LARGEFILE: No
> > > GSSAPI Kerberos 5 support: Yes
> > > FEATURE_DEBUG (debug build, slow code): No
> > > 32bit Atomic operations supported: Yes
> > > 64bit Atomic operations supported: Yes
> > > Runtime Instrumentation (slow code): No
> > >
> > > And configuration:
> > >
> > > $MaxMessageSize 8k
> > > $ModLoad imuxsock # provides support for local system logging
> > > $ModLoad imklog # provides kernel logging support
> > >
> > > # Store PID of the process in the log
> > > $SystemLogUsePIDFromSystem on
> > > # Rate limit for imuxsock
> > > $SystemLogRateLimitInterval 1
> > > $SystemLogRateLimitBurst 500
> > >
> > > $WorkDirectory /var/spool/rsyslog
> > >
> > > $ModLoad imfile
> > > $InputFilePollInterval 5
> > >
> > > $InputFileName /var/log/nginx/access.log
> > > $InputFilePersistStateInterval 100
> > > $InputFileTag nginx/access:
> > > $InputFileStateFile nginx_access_log_state
> > > $InputFileFacility local7
> > > $InputFileSeverity notice
> > > $InputRunFileMonitor
> > >
> > > $ActionQueueType LinkedList # enable a
> > > separate queue for this action
> > > $ActionQueueFileName remote # set file
> name,
> > > also enables disk mode
> > > $ActionResumeRetryCount -1 # infinite
> > > retries on insert failure
> > > $ActionQueueSaveOnShutdown on
> > > *.* @@10.0.0.200
> > >
> > > I went through ChangeLog on the site for the versions of rsyslogd
> > > following 5.8.11, but didn't notice any fix to that problem.
> Neither
> > > there is an open ticket in bugzilla.
> > >
> > > Interesting enough that RedHat claims they fixed the problem in
> version
> > > 3.22:
> > >
> > > http://rhn.redhat.com/errata/RHBA-2010-0213.html
> > >
> > > Can you point me to the version where this problem is fixed? Or, if
> > > it's not fixed yet - can it be done?
> > >
> > > BTW, not sure how this would behave with REPL protocol instead of
> > > TCP...
> > >
> > > with best regards,
> > > Timur Bakeyev.
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> POST
> > > if you DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
Sorry, I may have been a bit harsh... But I have to admit that very often folks come up and ask for real consulting advise on rsyslog (both on and off list). At some point, we must make a living out of something. So you get real great software for free. If that free package also includes unlimited free consulting, rsyslog would die, simply because I could not effort it as a hobby and so I'd need to look for a job to make my living. I try really hard to help everyone as far as I can, but there are limits. The problem you come up with tends to drive me over the edge, as it is so often discussed. The cure is to very intensely think about what one really wants. The only way to avoid the issue you have, I think, is to remove that functionality from rsyslog (and actually any other syslog product as it has nothing to do with the software). We even tried other -more lossy- default, but then again lots of folks come up and complain (because they don't understand why it is not "reliabl
e"...).

What you need is basically simple:

a) you don't want to lose messages (up to some point)
b) think how many resources you can dedicate
c) think which actions are really critical
d) think what kind/method of message loss you can accept when you run
out of resources
Then
e) configure async queues for the critical actions
f) allocate resources to these queues (and make sure they are reserved!)
g) configure queue to lose message in exceptional cases, as identified in
step d

You need solid understanding in your business needs (and legal regulations, if such apply), systems involved, protocols and rsyslog queue operations as well as config statements. That's why I say it's a real consulting job.

If step's b) to d) are done, finding a conf is often not very hard, but depending on the needs it may be.

That's the best advise I can dispense without going deep into consulting.

Again, my apologies for the harsh tone!
Rainer

> -----Original Message-----
> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> bounces@lists.adiscon.com] On Behalf Of Rainer Gerhards
> Sent: Wednesday, November 07, 2012 3:53 PM
> To: rsyslog-users
> Cc: Michael Biebl
> Subject: Re: [rsyslog] Rsyslog freezes the box when can't send logs
> over TCP
>
>
>
> > -----Original Message-----
> > From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> > bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
> > Sent: Wednesday, November 07, 2012 2:27 PM
> > To: rsyslog-users
> > Cc: Michael Biebl
> > Subject: Re: [rsyslog] Rsyslog freezes the box when can't send logs
> > over TCP
> >
> > Hi, Rainer!
> >
> > Of course you know internals of rsyslog better, so can't argue with
> you
> > here. But as for the end user such behavior for me looks undesired.
> > What
> > I'd expect:
> >
> > 1. All local(imuxsock) go to the local log files.
> > 2. All remote messages would be spooled in an $ActionQueueFileName
> > file.
> > 3. Nothing would block
>
> Yup - so why not configure it that way?
>
> Actually, very reliable logging is a tough consulting job. It's nothing
> you do in the lunch break.
> >
> > I also found this old discussion:
> >
> > http://www.mail-archive.com/rsyslog@lists.adiscon.com/msg01330.html
> >
> > So you tried to resolve it in 3.x series, but, apparently the problem
> > is
> > still there.
> There is NO Problem. There may have been one in v3, I don't remember -
> too long ago.
> >
> > DO I understand correctly that the reason is in the fact that Output
> > queue
> > for local files and remote TCP is shared, so until confirmation from
> > both
> > backends is received the message isn't discarded?
> >
> > Is there a way to spit those queues?
> >
> > Or, can you suggest the configuration that does, what I've described?
> > Any
> > change if TCP would be replaced with RELP?
> >
> You need to read up the in-depth description on how queues work. You
> need to know when it is OK to lose messages. Then, you need to
> configure the system to do that. The root cause is not the protocol,
> but your desired to have "nothing lost" but not thought what should
> happen if you run out of resources. Again, this is a though question
> and doing it right is a real consulting job.
>
> Rainer
>
> Rainer
> > With regards,
> > Timur Bakeyev.
> >
> > On Tue, Nov 6, 2012 at 3:37 PM, Rainer Gerhards
> > <rgerhards@hq.adiscon.com>wrote:
> >
> > > Long story short: the problem is with your config. You configured
> > rsyslog
> > > to never lose any message, at some point it runs out of queue
> space,
> > then
> > > the inputs begin to block, so the log socket blocks and finally the
> > apps
> > > that write to the log socket.
> > > There are many options to control under which circumstances you
> > permit
> > > message loss, and you need to make sure you know what you want.
> > > The current versions also have lower enq timeout, which can kind of
> > solve
> > > this config issue, at least for systems that are not super-busy.
> Note
> > that
> > > it is not really an issue, but may be intended behavior in some
> > > environments (e.g. banking, when you absolutely need to log).
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> > > > bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
> > > > Sent: Tuesday, November 06, 2012 3:26 PM
> > > > To: rsyslog@lists.adiscon.com
> > > > Cc: Michael Biebl
> > > > Subject: [rsyslog] Rsyslog freezes the box when can't send logs
> > over
> > > > TCP
> > > >
> > > > Hi all!
> > > >
> > > > Recently we experienced a "nice" lock up of our web nodes that do
> > > > remote TCP logging to the centralized syslog server.
> > > >
> > > > Central rsyslog server was shut down due the maintenance for few
> > > > hours. Very soon first Web servers with nginx/php-fpm started to
> > fail
> > > > with timeouts and later access to the nodes themselves over ssh
> > > > stopped working, making boxes unaccessible. With the iLO it was
> > > > possible to log to the consoles of the nodes, but no problems
> were
> > > > spotted - memory, CPU, TCP sockets footprints were low,
> everything
> > > > looked normal.
> > > >
> > > > Educated guess that it may be related to the rsyslog, as it was
> the
> > > > only change in the servers configuration on that day and killing
> > > > rsyslogd on the nodes brought machines back to normal.
> > > >
> > > > A bit of googling brought up this story:
> > > >
> > > > http://blog.bitbucket.org/2012/01/12/follow-up-on-our-downtime-
> > last-
> > > > week/
> > > >
> > > > as well as a last year conversation on this ML:
> > > >
> > > > http://lists.adiscon.net/pipermail/rsyslog/2011-
> October/013944.html
> > > >
> > > > as well as a bug report in RedHat Bugzilla:
> > > >
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=519201
> > > >
> > > > Switching to UDP transport instead of TCP cured the problem and
> > > > running simple test with:
> > > >
> > > > # for each n in `seq 1 1000`; do echo $n; echo "Syslog test $n" |
> > > > logger; done
> > > >
> > > > showed that with TCP remote logging enabled and absent remote log
> > > > server after first 250-300 messages all the following messages
> were
> > > > sent to syslog very slowly, with the 1-2sec delay each.
> > > >
> > > > All nodes are running Debian 6(squeeze) with:
> > > >
> > > > rsyslogd 5.8.11, compiled with:
> > > > FEATURE_REGEXP: Yes
> > > > FEATURE_LARGEFILE: No
> > > > GSSAPI Kerberos 5 support: Yes
> > > > FEATURE_DEBUG (debug build, slow code): No
> > > > 32bit Atomic operations supported: Yes
> > > > 64bit Atomic operations supported: Yes
> > > > Runtime Instrumentation (slow code): No
> > > >
> > > > And configuration:
> > > >
> > > > $MaxMessageSize 8k
> > > > $ModLoad imuxsock # provides support for local system logging
> > > > $ModLoad imklog # provides kernel logging support
> > > >
> > > > # Store PID of the process in the log
> > > > $SystemLogUsePIDFromSystem on
> > > > # Rate limit for imuxsock
> > > > $SystemLogRateLimitInterval 1
> > > > $SystemLogRateLimitBurst 500
> > > >
> > > > $WorkDirectory /var/spool/rsyslog
> > > >
> > > > $ModLoad imfile
> > > > $InputFilePollInterval 5
> > > >
> > > > $InputFileName /var/log/nginx/access.log
> > > > $InputFilePersistStateInterval 100
> > > > $InputFileTag nginx/access:
> > > > $InputFileStateFile nginx_access_log_state
> > > > $InputFileFacility local7
> > > > $InputFileSeverity notice
> > > > $InputRunFileMonitor
> > > >
> > > > $ActionQueueType LinkedList # enable
> a
> > > > separate queue for this action
> > > > $ActionQueueFileName remote # set file
> > name,
> > > > also enables disk mode
> > > > $ActionResumeRetryCount -1 # infinite
> > > > retries on insert failure
> > > > $ActionQueueSaveOnShutdown on
> > > > *.* @@10.0.0.200
> > > >
> > > > I went through ChangeLog on the site for the versions of rsyslogd
> > > > following 5.8.11, but didn't notice any fix to that problem.
> > Neither
> > > > there is an open ticket in bugzilla.
> > > >
> > > > Interesting enough that RedHat claims they fixed the problem in
> > version
> > > > 3.22:
> > > >
> > > > http://rhn.redhat.com/errata/RHBA-2010-0213.html
> > > >
> > > > Can you point me to the version where this problem is fixed? Or,
> if
> > > > it's not fixed yet - can it be done?
> > > >
> > > > BTW, not sure how this would behave with REPL protocol instead of
> > > > TCP...
> > > >
> > > > with best regards,
> > > > Timur Bakeyev.
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > POST
> > > > if you DON'T LIKE THAT.
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you
> > > DON'T LIKE THAT.
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> POST
> > if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
Following up on what Rainer is saying.

You should not think of this as fundamentally a rsyslog problem, you will run
into this problem in many other situations.

You tripped over this with nginx blocking when it couldn't write to syslog. I'll
point out that nginx would do the same thing if it was writing to disk and the
disk filled up. similarly, if you write to a NFS server that doesn't have a fast
disk cache, the performance of nginx will slow to a crawl because the NFS server
is required by spec to save each write to the log to disk before it considers
the write complete.

This is because nginx is written in the mode that they consider logs so critical
that they would prefer to have the system halt rather than loose a single log
message. Nginx is not alone in this, Apache does the same thing. This is why the
Apache docs tell you not to have your log files on NFS.

The fear of this sort of problem is why nginx doesn't support logging to syslog
by default and you need to patch it to do so.

Traditional syslog did a sync to disk for every message it recieved. This caused
it to have a very low peak message rate, and also meant that if the disk filled
up you would not be able to login to fix it (since login or su would try and
create a log message, which could not be written to disk, so the login or su
would never complete). This problem extended across a reboot as the disk would
still be full.

Using a reliable transport between systems (TCP or RELP) now extends this
behavior across systems.


Rsyslog relaxes this strict behavior by default because it has a queue (in
memory by default) where some number of messages can accumulate without needing
to hit the disk. These messages will be lost if the machine looses power before
they get written to disk.

But if this queue fills up, you end up back in the exact same problem again.
There is nothing that you can do with the log message, and you are still in the
mode that stopping everything instead of loosing a log message is the 'right'
thing to do.

To some extent you can address this by creating a disk queue, but disk queues
are very slow compared to memory queues, so you may not be able to handle your
log volume when doing so.

Another thing that you can do is to configure rsyslog to discard messages when
the queue gets full instead of blocking. Rsyslog has fairly extensive options
for specifying what to discard and when.

And then you can always opt to use UDP as your transport. As an 'unreliable'
trnasport protocol (i.e. no acknowledgement of if the message has arrived or
not) a down reciever will not block the senders. I put unreliable in quotes,
because while it can loose messages, a properly configured system can handle
messages at wire speed without loosing anything (unless a system is down, and
you can implement high availability on youre recieving servers to deal with
that).

David Lang

On Wed, 7 Nov 2012, Timur I. Bakeyev wrote:

> Hi, Rainer!
>
> Of course you know internals of rsyslog better, so can't argue with you
> here. But as for the end user such behavior for me looks undesired. What
> I'd expect:
>
> 1. All local(imuxsock) go to the local log files.
> 2. All remote messages would be spooled in an $ActionQueueFileName file.
> 3. Nothing would block
>
> I also found this old discussion:
>
> http://www.mail-archive.com/rsyslog@lists.adiscon.com/msg01330.html
>
> So you tried to resolve it in 3.x series, but, apparently the problem is
> still there.
>
> DO I understand correctly that the reason is in the fact that Output queue
> for local files and remote TCP is shared, so until confirmation from both
> backends is received the message isn't discarded?
>
> Is there a way to spit those queues?
>
> Or, can you suggest the configuration that does, what I've described? Any
> change if TCP would be replaced with RELP?
>
> With regards,
> Timur Bakeyev.
>
> On Tue, Nov 6, 2012 at 3:37 PM, Rainer Gerhards <rgerhards@hq.adiscon.com>wrote:
>
>> Long story short: the problem is with your config. You configured rsyslog
>> to never lose any message, at some point it runs out of queue space, then
>> the inputs begin to block, so the log socket blocks and finally the apps
>> that write to the log socket.
>> There are many options to control under which circumstances you permit
>> message loss, and you need to make sure you know what you want.
>> The current versions also have lower enq timeout, which can kind of solve
>> this config issue, at least for systems that are not super-busy. Note that
>> it is not really an issue, but may be intended behavior in some
>> environments (e.g. banking, when you absolutely need to log).
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
>>> bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
>>> Sent: Tuesday, November 06, 2012 3:26 PM
>>> To: rsyslog@lists.adiscon.com
>>> Cc: Michael Biebl
>>> Subject: [rsyslog] Rsyslog freezes the box when can't send logs over
>>> TCP
>>>
>>> Hi all!
>>>
>>> Recently we experienced a "nice" lock up of our web nodes that do
>>> remote TCP logging to the centralized syslog server.
>>>
>>> Central rsyslog server was shut down due the maintenance for few
>>> hours. Very soon first Web servers with nginx/php-fpm started to fail
>>> with timeouts and later access to the nodes themselves over ssh
>>> stopped working, making boxes unaccessible. With the iLO it was
>>> possible to log to the consoles of the nodes, but no problems were
>>> spotted - memory, CPU, TCP sockets footprints were low, everything
>>> looked normal.
>>>
>>> Educated guess that it may be related to the rsyslog, as it was the
>>> only change in the servers configuration on that day and killing
>>> rsyslogd on the nodes brought machines back to normal.
>>>
>>> A bit of googling brought up this story:
>>>
>>> http://blog.bitbucket.org/2012/01/12/follow-up-on-our-downtime-last-
>>> week/
>>>
>>> as well as a last year conversation on this ML:
>>>
>>> http://lists.adiscon.net/pipermail/rsyslog/2011-October/013944.html
>>>
>>> as well as a bug report in RedHat Bugzilla:
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=519201
>>>
>>> Switching to UDP transport instead of TCP cured the problem and
>>> running simple test with:
>>>
>>> # for each n in `seq 1 1000`; do echo $n; echo "Syslog test $n" |
>>> logger; done
>>>
>>> showed that with TCP remote logging enabled and absent remote log
>>> server after first 250-300 messages all the following messages were
>>> sent to syslog very slowly, with the 1-2sec delay each.
>>>
>>> All nodes are running Debian 6(squeeze) with:
>>>
>>> rsyslogd 5.8.11, compiled with:
>>> FEATURE_REGEXP: Yes
>>> FEATURE_LARGEFILE: No
>>> GSSAPI Kerberos 5 support: Yes
>>> FEATURE_DEBUG (debug build, slow code): No
>>> 32bit Atomic operations supported: Yes
>>> 64bit Atomic operations supported: Yes
>>> Runtime Instrumentation (slow code): No
>>>
>>> And configuration:
>>>
>>> $MaxMessageSize 8k
>>> $ModLoad imuxsock # provides support for local system logging
>>> $ModLoad imklog # provides kernel logging support
>>>
>>> # Store PID of the process in the log
>>> $SystemLogUsePIDFromSystem on
>>> # Rate limit for imuxsock
>>> $SystemLogRateLimitInterval 1
>>> $SystemLogRateLimitBurst 500
>>>
>>> $WorkDirectory /var/spool/rsyslog
>>>
>>> $ModLoad imfile
>>> $InputFilePollInterval 5
>>>
>>> $InputFileName /var/log/nginx/access.log
>>> $InputFilePersistStateInterval 100
>>> $InputFileTag nginx/access:
>>> $InputFileStateFile nginx_access_log_state
>>> $InputFileFacility local7
>>> $InputFileSeverity notice
>>> $InputRunFileMonitor
>>>
>>> $ActionQueueType LinkedList # enable a
>>> separate queue for this action
>>> $ActionQueueFileName remote # set file name,
>>> also enables disk mode
>>> $ActionResumeRetryCount -1 # infinite
>>> retries on insert failure
>>> $ActionQueueSaveOnShutdown on
>>> *.* @@10.0.0.200
>>>
>>> I went through ChangeLog on the site for the versions of rsyslogd
>>> following 5.8.11, but didn't notice any fix to that problem. Neither
>>> there is an open ticket in bugzilla.
>>>
>>> Interesting enough that RedHat claims they fixed the problem in version
>>> 3.22:
>>>
>>> http://rhn.redhat.com/errata/RHBA-2010-0213.html
>>>
>>> Can you point me to the version where this problem is fixed? Or, if
>>> it's not fixed yet - can it be done?
>>>
>>> BTW, not sure how this would behave with REPL protocol instead of
>>> TCP...
>>>
>>> with best regards,
>>> Timur Bakeyev.
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>> if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
On Wed, 7 Nov 2012, Rainer Gerhards wrote:

>> -----Original Message-----
>> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
>> bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
>> Sent: Wednesday, November 07, 2012 2:27 PM
>> Hi, Rainer!
>>
>> Of course you know internals of rsyslog better, so can't argue with you
>> here. But as for the end user such behavior for me looks undesired.
>> What
>> I'd expect:
>>
>> 1. All local(imuxsock) go to the local log files.
>> 2. All remote messages would be spooled in an $ActionQueueFileName
>> file.
>> 3. Nothing would block
>
> Yup - so why not configure it that way?

Rainer,

looking through his partial config, it looks like he does have a separate
action queue with disk assist configured for the sending to the remote host.
I'm not that familiar with the disk queues, so I may be missing what he's done
wrong here, but shouldn't this queue the logs to disk and not block unless the
disk is full?

David Lang

>>>> And configuration:
>>>>
>>>> $MaxMessageSize 8k
>>>> $ModLoad imuxsock # provides support for local system logging
>>>> $ModLoad imklog # provides kernel logging support
>>>>
>>>> # Store PID of the process in the log
>>>> $SystemLogUsePIDFromSystem on
>>>> # Rate limit for imuxsock
>>>> $SystemLogRateLimitInterval 1
>>>> $SystemLogRateLimitBurst 500
>>>>
>>>> $WorkDirectory /var/spool/rsyslog
>>>>
>>>> $ModLoad imfile
>>>> $InputFilePollInterval 5
>>>>
>>>> $InputFileName /var/log/nginx/access.log
>>>> $InputFilePersistStateInterval 100
>>>> $InputFileTag nginx/access:
>>>> $InputFileStateFile nginx_access_log_state
>>>> $InputFileFacility local7
>>>> $InputFileSeverity notice
>>>> $InputRunFileMonitor
>>>>
>>>> $ActionQueueType LinkedList # enable a
>>>> separate queue for this action
>>>> $ActionQueueFileName remote # set file name,
>>>> also enables disk mode
>>>> $ActionResumeRetryCount -1 # infinite
>>>> retries on insert failure
>>>> $ActionQueueSaveOnShutdown on
>>>> *.* @@10.0.0.200
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
> >> -----Original Message-----
> >> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
> >> bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
> >> Sent: Wednesday, November 07, 2012 2:27 PM
> >> Hi, Rainer!
> >>
> >> Of course you know internals of rsyslog better, so can't argue with
> you
> >> here. But as for the end user such behavior for me looks undesired.
> >> What
> >> I'd expect:
> >>
> >> 1. All local(imuxsock) go to the local log files.
> >> 2. All remote messages would be spooled in an $ActionQueueFileName
> >> file.
> >> 3. Nothing would block
> >
> > Yup - so why not configure it that way?
>
> Rainer,
>
> looking through his partial config, it looks like he does have a
> separate
> action queue with disk assist configured for the sending to the remote
> host.
> I'm not that familiar with the disk queues, so I may be missing what
> he's done
> wrong here, but shouldn't this queue the logs to disk and not block
> unless the
> disk is full?

Autsch - you really got me. This seemd to be the standard "just uncomment and be happy" part that is shipped by many distros. That contains a size limitation to 1 or 2 gig for the disk subsystem. But... that statement is not present! So it looks like I screwed up once more.

In that case, a debug log of a failure case would probably be useful (or at least of a startup, just to see if the size limitation is still set and just not present in the pasted config).

Rainer
>
> David Lang
>
> >>>> And configuration:
> >>>>
> >>>> $MaxMessageSize 8k
> >>>> $ModLoad imuxsock # provides support for local system logging
> >>>> $ModLoad imklog # provides kernel logging support
> >>>>
> >>>> # Store PID of the process in the log
> >>>> $SystemLogUsePIDFromSystem on
> >>>> # Rate limit for imuxsock
> >>>> $SystemLogRateLimitInterval 1
> >>>> $SystemLogRateLimitBurst 500
> >>>>
> >>>> $WorkDirectory /var/spool/rsyslog
> >>>>
> >>>> $ModLoad imfile
> >>>> $InputFilePollInterval 5
> >>>>
> >>>> $InputFileName /var/log/nginx/access.log
> >>>> $InputFilePersistStateInterval 100
> >>>> $InputFileTag nginx/access:
> >>>> $InputFileStateFile nginx_access_log_state
> >>>> $InputFileFacility local7
> >>>> $InputFileSeverity notice
> >>>> $InputRunFileMonitor
> >>>>
> >>>> $ActionQueueType LinkedList # enable a
> >>>> separate queue for this action
> >>>> $ActionQueueFileName remote # set file
> name,
> >>>> also enables disk mode
> >>>> $ActionResumeRetryCount -1 # infinite
> >>>> retries on insert failure
> >>>> $ActionQueueSaveOnShutdown on
> >>>> *.* @@10.0.0.200
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
On Thu, 8 Nov 2012, Rainer Gerhards wrote:

>>>> -----Original Message-----
>>>> From: rsyslog-bounces@lists.adiscon.com [mailto:rsyslog-
>>>> bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
>>>> Sent: Wednesday, November 07, 2012 2:27 PM
>>>> Hi, Rainer!
>>>>
>>>> Of course you know internals of rsyslog better, so can't argue with
>> you
>>>> here. But as for the end user such behavior for me looks undesired.
>>>> What
>>>> I'd expect:
>>>>
>>>> 1. All local(imuxsock) go to the local log files.
>>>> 2. All remote messages would be spooled in an $ActionQueueFileName
>>>> file.
>>>> 3. Nothing would block
>>>
>>> Yup - so why not configure it that way?
>>
>> Rainer,
>>
>> looking through his partial config, it looks like he does have a
>> separate
>> action queue with disk assist configured for the sending to the remote
>> host.
>> I'm not that familiar with the disk queues, so I may be missing what
>> he's done
>> wrong here, but shouldn't this queue the logs to disk and not block
>> unless the
>> disk is full?
>
> Autsch - you really got me. This seemd to be the standard "just uncomment and
> be happy" part that is shipped by many distros. That contains a size
> limitation to 1 or 2 gig for the disk subsystem. But... that statement is not
> present! So it looks like I screwed up once more.

it could be, I'm not that familiar with the distro default configs (I always put
everything in /var/log/messages or send it to a remote system with my own config
:-)

> In that case, a debug log of a failure case would probably be useful (or at
> least of a startup, just to see if the size limitation is still set and just
> not present in the pasted config).

this seems to be a partial config (based on the comment about local files)


another possible issue:

what happens when you hit the rate limit? does it drop messages or block them?

David Lang

> Rainer
>>
>> David Lang
>>
>>>>>> And configuration:
>>>>>>
>>>>>> $MaxMessageSize 8k
>>>>>> $ModLoad imuxsock # provides support for local system logging
>>>>>> $ModLoad imklog # provides kernel logging support
>>>>>>
>>>>>> # Store PID of the process in the log
>>>>>> $SystemLogUsePIDFromSystem on
>>>>>> # Rate limit for imuxsock
>>>>>> $SystemLogRateLimitInterval 1
>>>>>> $SystemLogRateLimitBurst 500
>>>>>>
>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>>
>>>>>> $ModLoad imfile
>>>>>> $InputFilePollInterval 5
>>>>>>
>>>>>> $InputFileName /var/log/nginx/access.log
>>>>>> $InputFilePersistStateInterval 100
>>>>>> $InputFileTag nginx/access:
>>>>>> $InputFileStateFile nginx_access_log_state
>>>>>> $InputFileFacility local7
>>>>>> $InputFileSeverity notice
>>>>>> $InputRunFileMonitor
>>>>>>
>>>>>> $ActionQueueType LinkedList # enable a
>>>>>> separate queue for this action
>>>>>> $ActionQueueFileName remote # set file
>> name,
>>>>>> also enables disk mode
>>>>>> $ActionResumeRetryCount -1 # infinite
>>>>>> retries on insert failure
>>>>>> $ActionQueueSaveOnShutdown on
>>>>>> *.* @@10.0.0.200
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
Hi, David!

Here are the full configs:

Slightly modified Debian's default rsyslog.conf:

http://pastebin.com/wUnLEQF3

Plus files from the /etc/rsyslog.d/:

http://pastebin.com/8UpKHH8C - file monitor for some non-syslog log files

http://pastebin.com/JC5xRMJj

- remote sysloging.

Please note, that @@ from the original(failing) config was replaced with @
to make system responsive.

Also, there are traces of an attempt to use a separate rule for remote
logging, but version 5.8.11 didn't want to work with them for me...


That's all.

With best regards,
Timur Bakeyev.


On Thu, Nov 8, 2012 at 9:14 AM, David Lang <david@lang.hm> wrote:

> On Thu, 8 Nov 2012, Rainer Gerhards wrote:
>
> -----Original Message-----
>>>>> From: rsyslog-bounces@lists.adiscon.**com<rsyslog-bounces@lists.adiscon.com>[mailto:
>>>>> rsyslog-
>>>>> bounces@lists.adiscon.com] On Behalf Of Timur I. Bakeyev
>>>>> Sent: Wednesday, November 07, 2012 2:27 PM
>>>>> Hi, Rainer!
>>>>>
>>>>> Of course you know internals of rsyslog better, so can't argue with
>>>>>
>>>> you
>>>
>>>> here. But as for the end user such behavior for me looks undesired.
>>>>> What
>>>>> I'd expect:
>>>>>
>>>>> 1. All local(imuxsock) go to the local log files.
>>>>> 2. All remote messages would be spooled in an $ActionQueueFileName
>>>>> file.
>>>>> 3. Nothing would block
>>>>>
>>>>
>>>> Yup - so why not configure it that way?
>>>>
>>>
>>> Rainer,
>>>
>>> looking through his partial config, it looks like he does have a
>>> separate
>>> action queue with disk assist configured for the sending to the remote
>>> host.
>>> I'm not that familiar with the disk queues, so I may be missing what
>>> he's done
>>> wrong here, but shouldn't this queue the logs to disk and not block
>>> unless the
>>> disk is full?
>>>
>>
>> Autsch - you really got me. This seemd to be the standard "just uncomment
>> and be happy" part that is shipped by many distros. That contains a size
>> limitation to 1 or 2 gig for the disk subsystem. But... that statement is
>> not present! So it looks like I screwed up once more.
>>
>
> it could be, I'm not that familiar with the distro default configs (I
> always put everything in /var/log/messages or send it to a remote system
> with my own config :-)
>
>
> In that case, a debug log of a failure case would probably be useful (or
>> at least of a startup, just to see if the size limitation is still set and
>> just not present in the pasted config).
>>
>
> this seems to be a partial config (based on the comment about local files)
>
>
> another possible issue:
>
> what happens when you hit the rate limit? does it drop messages or block
> them?
>
> David Lang
>
>
> Rainer
>>
>>>
>>> David Lang
>>>
>>> And configuration:
>>>>>>>
>>>>>>> $MaxMessageSize 8k
>>>>>>> $ModLoad imuxsock # provides support for local system logging
>>>>>>> $ModLoad imklog # provides kernel logging support
>>>>>>>
>>>>>>> # Store PID of the process in the log
>>>>>>> $SystemLogUsePIDFromSystem on
>>>>>>> # Rate limit for imuxsock
>>>>>>> $SystemLogRateLimitInterval 1
>>>>>>> $SystemLogRateLimitBurst 500
>>>>>>>
>>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>>>
>>>>>>> $ModLoad imfile
>>>>>>> $InputFilePollInterval 5
>>>>>>>
>>>>>>> $InputFileName /var/log/nginx/access.log
>>>>>>> $InputFilePersistStateInterval 100
>>>>>>> $InputFileTag nginx/access:
>>>>>>> $InputFileStateFile nginx_access_log_state
>>>>>>> $InputFileFacility local7
>>>>>>> $InputFileSeverity notice
>>>>>>> $InputRunFileMonitor
>>>>>>>
>>>>>>> $ActionQueueType LinkedList # enable a
>>>>>>> separate queue for this action
>>>>>>> $ActionQueueFileName remote # set file
>>>>>>>
>>>>>> name,
>>>
>>>> also enables disk mode
>>>>>>> $ActionResumeRetryCount -1 # infinite
>>>>>>> retries on insert failure
>>>>>>> $ActionQueueSaveOnShutdown on
>>>>>>> *.* @@10.0.0.200
>>>>>>>
>>>>>> ______________________________**_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>
>>>
>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>
>>>
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>
>>>
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>> if you DON'T LIKE THAT.
>>>
>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>
>>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
>>
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>
>>
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>
>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>
>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
>
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
> another possible issue:
>
> what happens when you hit the rate limit? does it drop messages or
> block them?

Same thing: if a limit is set (or a timeframe!), then it blocks during that limit (except for the v7.3 input rate limiters, which are discarding). If the queues is overrun, it queues back into the main queue. Of course, proper discard settings for the action queue can prevent this.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog freezes the box when can't send logs over TCP [ In reply to ]
On 06.11.2012 15:26, Timur I. Bakeyev wrote:
>
> Recently we experienced a "nice" lock up of our web nodes that do
> remote TCP logging to the centralized syslog server.
>

did you try my patch shown in
http://lists.adiscon.net/pipermail/rsyslog/2011-October/013944.html?

regards,
-ap
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.