Mailing List Archive

Fwd: installing rsyslog-8.1905.0-2.el7.x86_64 and rsyslog-gnutls-8.1905.0-2.el7.x86_64 issues on AmazonLinux
I noticed the list seems to have been stripped from CC. Thus
forwarding. I hope someone on the list might have more insight into
this. Pls keep list CCed.

Rainer

El mié., 19 jun. 2019 a las 0:05, Li, Mike (<Mike.Li@finra.org>) escribió:
>
> Rainer,
> It seems if at a rate of sustained tcpflood -m13000 -p514 -Tudp and tcpflood -m7000 -p10514 -Ttls would crash the rsyslogd after 3hours 22 minutes on rsyslog 8.40

You mean you run the same commands inside a loop? I ask because the
command line params say they emit 13000 and 7000 messages and then
stop.

> I was using the following settings:
> module(load="imudp" SchedulingPolicy="fifo" SchedulingPriority="5" threads="3" timeRequery="8" batchSize="128")
> on a 2 processor AWS r5.large server (2 vCPUs & 16GB memory)
> with following messages below. For high volumes of syslog upd and tcp messages what is the recommended version of rsyslog to use?

I think this is the wrong question. Recommended is always the latests
stable (albeit we seem to have some strange issue with 8.1905.0 which
we are trying to reproduce). However, it looks to me like Amazon Linux
is incompatible with CentOs packages. Maybe they ship different TLS
library versions and thus the CentOs binary fails. The question is if
we can reproduce the issue on CentOS as well. That would also
potentially enable me to reproduce the issue.

Rainer
> Thanks.
> Mike
> ---
> [198676.277931] traps: rs:main Q:Reg[12205] general protection ip:55d63228f781 sp:7f93a2f64a30 error:0 in rsyslogd[55d63223c000+a0000]
> [221338.057744] Process accounting resumed
> [307018.430027] Process accounting resumed
> [393238.004756] Process accounting resumed
> [427408.637878] traps: rs:main Q:Reg[24839] general protection ip:7fd083655f69 sp:7fd0733eb010 error:0 in libc-2.17.so[7fd083609000+1c2000]
> [434823.376547] traps: rs:main Q:Reg[30731] general protection ip:7f9f987b6f69 sp:7f9f954d89c0 error:0 in libc-2.17.so[7f9f9876a000+1c2000]
> [457248.303949] rs:main Q:Reg[15860]: segfault at 0 ip (null) sp 00007f30199c3a48 error 14 in rsyslogd[5623e3a31000+a0000]
> [479997.153803] Process accounting resumed
> [514947.777848] traps: rs:main Q:Reg[11829] general protection ip:7fd9551b5f69 sp:7fd942ff21c0 error:0 in libc-2.17.so[7fd955169000+1c2000]
> [516032.118182] rs:main Q:Reg[14077]: segfault at 0 ip (null) sp 00007fcc9fffea48 error 14 in rsyslogd[55af0ba47000+a0000]
>
>
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 85
> model name : Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
> stepping : 4
> microcode : 0x200005e
> cpu MHz : 3110.233
> cache size : 33792 KB
> physical id : 0
> siblings : 2
> core id : 0
> cpu cores : 1
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
> bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds
> bogomips : 5000.00
> clflush size : 64
> cache_alignment : 64
> address sizes : 46 bits physical, 48 bits virtual
> power management:
>
> -----Original Message-----
> From: Li, Mike
> Sent: Tuesday, June 18, 2019 9:26 AM
> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Subject: RE: [rsyslog] installing rsyslog-8.1905.0-2.el7.x86_64 and rsyslog-gnutls-8.1905.0-2.el7.x86_64 issues on AmazonLinux
>
> Rainer,
>
> Still got :
> rsyslogd: netstream session 0x7fcc9818ec00 from 10.162.139.182 will be closed due to error [v8.1905.0 try https://www.rsyslog.com/e/2089 ]
>
> [root@ip-10-162-65-173 app_splunk_dev_18-1]# rsyslogd: netstream session 0x7fcc982aa0e0 from 10.162.139.182 will be closed due to error [v8.1905.0 try https://www.rsyslog.com/e/2089 ] And saw:
> Segmentation fault /sbin/rsyslogd -n -i /var/run/syslogd.pid
>
> Also:
> ./tcpflood -p10514 -m10 -Ttls -x./rsyslog/cis-eng-ca.pem -Z./rsyslog/logforwarder.finra.org.pem -z./rsyslog/logforwarder.finra.org-key.pem -t10.162.65.173
> 00000connect(): Connection refused
> connect() failed
> error in trying to open connection i=0
> error opening connections
>
> Please advise.
> Thanks.
> Mike
>
> -----Original Message-----
> From: Li, Mike
> Sent: Tuesday, June 18, 2019 9:11 AM
> To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Subject: RE: [rsyslog] installing rsyslog-8.1905.0-2.el7.x86_64 and rsyslog-gnutls-8.1905.0-2.el7.x86_64 issues on AmazonLinux
>
> Hi Rainer,
> Installed rsyslog-openssl-8.1905.0-2.el6.x86_64
> With following setting in rsyslog.conf:
> global(
> defaultNetstreamDriverCAFile="/opt/splunk/etc/apps/proofpoint/certs/cis-eng-ca.pem"
> defaultNetstreamDriverCertFile="/opt/splunk/etc/apps/proofpoint/certs/logforwarder.finra.org.pem"
> defaultNetstreamDriverKeyFile="/opt/splunk/etc/apps/proofpoint/certs/logforwarder.finra.org-key.pem"
> )
> module(
> load="imtcp"
> StreamDriver.Name="ossl"
> #StreamDriver.Name="gtls"
> StreamDriver.mode="1"
> StreamDriver.AuthMode="anon"
> )
> input(type="imtcp" port="10514")
>
> But I got
> syslogd: Error: OpenSSL Version to old, SSL_CONF_cmd API is not supported. [v8.1905.0 try https://www.rsyslog.com/e/2095 ]
>
> and following errors in dmesg:
> [198676.277931] traps: rs:main Q:Reg[12205] general protection ip:55d63228f781 sp:7f93a2f64a30 error:0 in rsyslogd[55d63223c000+a0000] [221338.057744] Process accounting resumed [307018.430027] Process accounting resumed [393238.004756] Process accounting resumed [427408.637878] traps: rs:main Q:Reg[24839] general protection ip:7fd083655f69 sp:7fd0733eb010 error:0 in libc-2.17.so[7fd083609000+1c2000] [434823.376547] traps: rs:main Q:Reg[30731] general protection ip:7f9f987b6f69 sp:7f9f954d89c0 error:0 in libc-2.17.so[7f9f9876a000+1c2000]
> [457248.303949] rs:main Q:Reg[15860]: segfault at 0 ip (null) sp 00007f30199c3a48 error 14 in rsyslogd[5623e3a31000+a0000]
> [479997.153803] Process accounting resumed [514947.777848] traps: rs:main Q:Reg[11829] general protection ip:7fd9551b5f69 sp:7fd942ff21c0 error:0 in libc-2.17.so[7fd955169000+1c2000]
>
> I attached rsyslog-618.log.gz to the ticket.
> Thanks.
> Mike
> -----Original Message-----
> From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Sent: Tuesday, June 18, 2019 3:56 AM
> To: Li, Mike <Mike.Li@finra.org>
> Subject: Re: [rsyslog] installing rsyslog-8.1905.0-2.el7.x86_64 and rsyslog-gnutls-8.1905.0-2.el7.x86_64 issues on AmazonLinux
>
> EXTERNAL: Verify sender before opening attachments or links.
>
> The debug log is unfortunately not starting at rsyslog start, so I am not sure that I see the root cause of the problem. What I can see looks like there is a problem with gnutls or the remote end. Looks a bit like the remote server breaks the connection. This can indeed lead to very slow processing. I would suggest to switch to the openssl driver as a try. It is known to be more robust than GnuTLS (that's the reason we added it).
>
> Rainer
>
> El lun., 17 jun. 2019 a las 19:36, Li, Mike (<Mike.Li@finra.org>) escribió:
> >
> > Rainer,
> >
> > rsyslog crashed from the tcpflood testings. From dmesg, I saw the following:
> >
> > [198676.277931] traps: rs:main Q:Reg[12205] general protection
> > ip:55d63228f781 sp:7f93a2f64a30 error:0 in
> > rsyslogd[55d63223c000+a0000]
> >
> > and I'm also seeing lots of "rsyslogd: gnutls returned error on handshake: A TLS packet with unexpected length was received. [v8.1905.0 try https://www.rsyslog.com/e/2083 ]"
> >
> > rsyslog-8.1905.0-2.el6.x86_64
> > rsyslog-gnutls-8.1905.0-2.el6.x86_64
> >
> > Please advise if you need other information.
> >
> > Thanks.
> >
> > Mike
> >
> > -----Original Message-----
> > From: Li, Mike
> > Sent: Monday, June 17, 2019 10:32 AM
> > To: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Subject: RE: [rsyslog] installing rsyslog-8.1905.0-2.el7.x86_64 and
> > rsyslog-gnutls-8.1905.0-2.el7.x86_64 issues on AmazonLinux
> >
> > Thanks a lot Rainer.
> >
> > Please see the information of the ticket.
> > BRegards,
> > Mike
> > ---
> >
> > Dear Mike Li,
> >
> > We have received your ticket and confirmation has been sent to your email address mike.li@finra.org.
> >
> > Your ticket id is #4427. You will get email notification after we post reply in your ticket but in case email notification failed, you can check your ticket status on below link:
> >
> > https://ticket.adiscon.com/?support_page=open_ticket&ticket_id=4427&au
> > th_code=SAzWWeK9vo
> >
> > -----Original Message-----
> > From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Sent: Monday, June 17, 2019 10:22 AM
> > To: Li, Mike <Mike.Li@finra.org>
> > Subject: Re: [rsyslog] installing rsyslog-8.1905.0-2.el7.x86_64 and
> > rsyslog-gnutls-8.1905.0-2.el7.x86_64 issues on AmazonLinux
> >
> > EXTERNAL: Verify sender before opening attachments or links.
> >
> > El lun., 17 jun. 2019 a las 14:53, Li, Mike (<Mike.Li@finra.org>) escribió:
> > >
> > > Hi Rainer,
> > > I started running tcpflood -p514 -m12999 -Tudp -t10.162.65.173 from 10.162.66.74, but did not receive any events on the rsyslog server. I stopped & started with " /sbin/rsyslogd -n -i /var/run/syslogd.pid"
> > > I ran a tcpflood -p514 -m12999 -Tudp -t10.162.65.173 in debug mode
> > > which was enabled using steps in
> > > https://www.rsyslog.com/how-to-use-debug-on-demand/
> > > I received only 2048 events.
> > > The debug file is 61MB gzipped not sure if I could upload it to a sftp server on Adiscon.com? Please advise.
> >
> > Please submit the log by creating a ticket at
> > https://ticket.adiscon.com/
> >
> > IMPORTANT: This system is usually for paying support customers. Please mention in the ticket that you are working with me on that case and that I have asked you to upload the file via this system.
> >
> > Rainer
> >
> > > Thanks.
> > > Mike
> > >
> > > -----Original Message-----
> > > From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > > Sent: Friday, June 14, 2019 11:09 AM
> > > To: Li, Mike <Mike.Li@finra.org>
> > > Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> > > Subject: Re: [rsyslog] installing rsyslog-8.1905.0-2.el7.x86_64 and
> > > rsyslog-gnutls-8.1905.0-2.el7.x86_64 issues on AmazonLinux
> > >
> > > EXTERNAL: Verify sender before opening attachments or links.
> > >
> > > El vie., 14 jun. 2019 a las 16:33, Li, Mike (<Mike.Li@finra.org>) escribió:
> > > >
> > > > Hi Rainer,
> > > > I'm having issue with logging with rsyslog-8.1905.0.2.el6 which stops working after some time.
> > > > ./tcpflood -m10 -p514 -Ttcp -t10.162.65.173
> > > > 00001 open connections
> > > > starting run 1
> > > > Sending 10 messages.
> > > > 00000010 messages sent
> > > > runtime: 0.000
> > > > 00001 close connections
> > > > End of tcpflood Run
> > >
> > > Can you reproduce this? If so, can you send me a debug log?
> > >
> > > >
> > > > But I cannot find the messages in log file after a few tests.
> > > >
> > > > I see lots of :
> > > > 2019 Jun 14 13:56:06 ip-10-162-65-173 gnutls returned error on
> > > > handshake: A TLS packet with unexpected length was received. [v8.1905.0 try https://www.rsyslog.com/e/2083 ] Which caused the rsyslog to hang?
> > >
> > > These seem to be unrelated, but I don't know without the debug log.
> > >
> > > >
> > > > After I restart rsyslog with pkill -9 -f rsyslog; service rsyslog start. Logging resumes.
> > > > I saw logging issues in email thread " 8.1905.0 Logging Stops"
> > > > Will there another release coming soon and when will it be available from http://rpms.adiscon.com/v8-stable/epel-6/x86_64/RPMS/?C=M;O=D?
> > >
> > > I don't know - so far I have no idea of what might be the cause of this nor can I reproduce.
> > >
> > > Rainer
> > > >
> > > > Thanks.
> > > > Mike
> > > >
> > > > -----Original Message-----
> > > > From: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > > > Sent: Friday, June 14, 2019 10:15 AM
> > > > To: rsyslog-users <rsyslog@lists.adiscon.com>
> > > > Cc: Li, Mike <Mike.Li@finra.org>
> > > > Subject: Re: [rsyslog] installing rsyslog-8.1905.0-2.el7.x86_64
> > > > and
> > > > rsyslog-gnutls-8.1905.0-2.el7.x86_64 issues on AmazonLinux
> > > >
> > > > EXTERNAL: Verify sender before opening attachments or links.
> > > >
> > > > Just so that you get a reply: I have no idea of what is the differnce between CentOS and Amazon linux.
> > > >
> > > > Rainer
> > > >
> > > > El vie., 14 jun. 2019 a las 15:33, Li, Mike via rsyslog
> > > > (<rsyslog@lists.adiscon.com>) escribió:
> > > > >
> > > > > Hi,
> > > > > FYI, I was also to install epel6's
> > > > > rsyslog-8.1905.0-2.el6.x86_64.rpm and
> > > > > rsyslog-gnutls-8.1905.0-2.el6.x86_64.rpm
> > > > > Without any error on AmazonLinux 2018.3.
> > > > > Could anyone advise why epel7's rsyslog-8.1905.0-2.el7.x86_64.rpm and rsyslog-gnutls-8.1905.0-2.el7.x86_64.rpm gave the install errors?
> > > > > Thanks.
> > > > > Mike
> > > > >
> > > > > From: Li, Mike
> > > > > Sent: Thursday, June 13, 2019 3:19 PM
> > > > > To: rsyslog@lists.adiscon.com
> > > > > Subject: installing rsyslog-8.1905.0-2.el7.x86_64 and
> > > > > rsyslog-gnutls-8.1905.0-2.el7.x86_64 issues on AmazonLinux
> > > > >
> > > > > Hi all,
> > > > >
> > > > > cat /etc/os-release
> > > > > NAME="Amazon Linux AMI"
> > > > > VERSION="2018.03"
> > > > > ID="amzn"
> > > > > ID_LIKE="rhel fedora"
> > > > > VERSION_ID="2018.03"
> > > > > PRETTY_NAME="Amazon Linux AMI 2018.03"
> > > > > ANSI_COLOR="0;33"
> > > > > CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
> > > > > HOME_URL=http://aws.amazon.com/amazon-linux-ami/
> > > > >
> > > > > Ran yum -y install resolvedep rsyslog*rpm
> > > > >
> > > > > Examining rsyslog-8.1905.0-2.el7.x86_64.rpm:
> > > > > rsyslog-8.1905.0-2.el7.x86_64 Marking
> > > > > rsyslog-8.1905.0-2.el7.x86_64.rpm as an update to
> > > > > rsyslog-8.36.0-2.el6.x86_64 Examining
> > > > > rsyslog-gnutls-8.1905.0-2.el7.x86_64.rpm:
> > > > > rsyslog-gnutls-8.1905.0-2.el7.x86_64
> > > > > Marking rsyslog-gnutls-8.1905.0-2.el7.x86_64.rpm as an update to
> > > > > rsyslog-gnutls-8.36.0-2.el6.x86_64
> > > > > Resolving Dependencies
> > > > > --> Running transaction check
> > > > > ---> Package rsyslog.x86_64 0:8.36.0-2.el6 will be updated
> > > > > ---> Package
> > > > > ---> rsyslog.x86_64 0:8.1905.0-2.el7 will be an update
> > > > > --> Processing Dependency:
> > > > > --> libsystemd.so.0(LIBSYSTEMD_209)(64bit)
> > > > > --> for
> > > > > --> package: rsyslog-8.1905.0-2.el7.x86_64 Processing Dependency:
> > > > > --> systemd for package: rsyslog-8.1905.0-2.el7.x86_64
> > > > > --> Processing
> > > > > --> Dependency: systemd for package:
> > > > > --> rsyslog-8.1905.0-2.el7.x86_64 Processing Dependency: systemd for package:
> > > > > --> rsyslog-8.1905.0-2.el7.x86_64 Processing Dependency:
> > > > > --> libsystemd.so.0()(64bit) for package:
> > > > > --> rsyslog-8.1905.0-2.el7.x86_64
> > > > > ---> Package rsyslog-gnutls.x86_64 0:8.36.0-2.el6 will be
> > > > > ---> updated Package rsyslog-gnutls.x86_64 0:8.1905.0-2.el7 will
> > > > > ---> be an update
> > > > > --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit)
> > > > > --> for
> > > > > --> package: rsyslog-gnutls-8.1905.0-2.el7.x86_64
> > > > > --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit)
> > > > > --> for
> > > > > --> package: rsyslog-gnutls-8.1905.0-2.el7.x86_64
> > > > > --> Processing Dependency: libgnutls.so.28()(64bit) for package:
> > > > > --> rsyslog-gnutls-8.1905.0-2.el7.x86_64
> > > > > --> Finished Dependency Resolution
> > > > > Error: Package: rsyslog-gnutls-8.1905.0-2.el7.x86_64 (/rsyslog-gnutls-8.1905.0-2.el7.x86_64)
> > > > > Requires: libgnutls.so.28(GNUTLS_2_12)(64bit)
> > > > > Error: Package: rsyslog-8.1905.0-2.el7.x86_64 (/rsyslog-8.1905.0-2.el7.x86_64)
> > > > > Requires: libsystemd.so.0()(64bit)
> > > > > Error: Package: rsyslog-8.1905.0-2.el7.x86_64 (/rsyslog-8.1905.0-2.el7.x86_64)
> > > > > Requires: libsystemd.so.0(LIBSYSTEMD_209)(64bit)
> > > > > Error: Package: rsyslog-gnutls-8.1905.0-2.el7.x86_64 (/rsyslog-gnutls-8.1905.0-2.el7.x86_64)
> > > > > Requires: libgnutls.so.28(GNUTLS_1_4)(64bit)
> > > > > Error: Package: rsyslog-gnutls-8.1905.0-2.el7.x86_64 (/rsyslog-gnutls-8.1905.0-2.el7.x86_64)
> > > > > Requires: libgnutls.so.28()(64bit)
> > > > > Error: Package: rsyslog-8.1905.0-2.el7.x86_64 (/rsyslog-8.1905.0-2.el7.x86_64)
> > > > > Requires: systemd
> > > > > You could try using --skip-broken to work around the problem You
> > > > > could try running: rpm -Va --nofiles --nodigest
> > > > >
> > > > >
> > > > > Please advise.
> > > > > Thanks.
> > > > > Mike
> > > > >
> > > > > Confidentiality Notice:: This email, including attachments, may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com/professional-services/
> > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > > NOTE
> > > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.