Mailing List Archive

Exim process stack 100% CPU?
Hello,

I'm running latest Centos 6 with exim 4.90.1. Recently I have noticed that
some exim process in my system hangs on phase "handling TLS incoming
connection" using 100% CPU for hours.

exiwhat

41182 handling TLS incoming connection from s16.*** [91.*.*.*]

strace

read(7, "", 6049) = 0
alarm(0) = 180
alarm(180) = 0
read(7, "", 6049) = 0
alarm(0) = 180
alarm(180) = 0
...

ls /proc/41182/fd

lrwx------ 1 root root 64 03-05 08:03 6 -> socket:[230019376]
lrwx------ 1 root root 64 03-05 08:03 7 -> socket:[230019376]

In /var/log/exim/mainlog

2018-03-04 16:46:16 H=s16** [91.*.*.*]
X=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=no F=<abc@example.com>
rejected RCPT <www@other.com>:
(empty string after colon)
...

In my opinion it's really easy way to make successful DoS attack.

I will be grateful for any help.

Regards,
Mateusz Krawczyk
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim process stack 100% CPU? [ In reply to ]
On 05/03/18 07:32, Mateusz Krawczyk via Exim-users wrote:
> I'm running latest Centos 6 with exim 4.90.1. Recently I have noticed that
> some exim process in my system hangs on phase "handling TLS incoming
> connection" using 100% CPU for hours.

Exim binary supplied by a centos package, or compiled yourself?

Also, please run "exim -d -bV" and give the initial output, down
as far as the "WHITELIST_D_MACROS" line (we're mainly interested in
the library versions).

--
Thanks,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim process stack 100% CPU? [ In reply to ]
Mateusz Krawczyk via Exim-users <exim-users@exim.org> (Mo 05 Mär 2018 08:32:13 CET):
> Hello,
>
> I'm running latest Centos 6 with exim 4.90.1. Recently I have noticed that
> some exim process in my system hangs on phase "handling TLS incoming
> connection" using 100% CPU for hours.
>
> exiwhat
>
> 41182 handling TLS incoming connection from s16.*** [91.*.*.*]
>
> strace
>
> read(7, "", 6049) = 0
> alarm(0) = 180
> alarm(180) = 0
> read(7, "", 6049) = 0
> alarm(0) = 180
> alarm(180) = 0

>
> lrwx------ 1 root root 64 03-05 08:03 6 -> socket:[230019376]
> lrwx------ 1 root root 64 03-05 08:03 7 -> socket:[230019376]

> In /var/log/exim/mainlog
>
> 2018-03-04 16:46:16 H=s16** [91.*.*.*]
> X=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=no F=<abc@example.com>
> rejected RCPT <www@other.com>:
> (empty string after colon)
> ...

The verification message is missing. The rejection is probably due to
ACL verify recipient. Do you set a message there?

Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de ---------------------------- internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --------------- key ID: F69376CE -
! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
Re: Exim process stack 100% CPU? [ In reply to ]
Thank you for quick reply. Exim is compiled using scripts/source from
DirectAdmin.

exim -d -bV

Exim version 4.90_1 #4 built 12-Feb-2018 13:32:43
Copyright (c) University of Cambridge, 1995 - 2017
(c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007 -
2017
Berkeley DB: Berkeley DB 4.7.25: (March 22, 2017)
Support for: crypteq IPv6 Perl OpenSSL move_frozen_messages
Content_Scanning DKIM DNSSEC Event OCSP PRDR
Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm
dbmjz dbmnz dnsdb
Authenticators: cram_md5 dovecot plaintext spa
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Fixed never_users: 0
Configure owner: 0:0
Size of off_t: 8
Compiler: GCC [4.4.7 20120313 (Red Hat 4.4.7-18)]
Library version: Glibc: Compile: 2.12
Runtime: 2.12
Library version: OpenSSL: Compile: OpenSSL 1.0.1e-fips 11 Feb 2013
Runtime: OpenSSL 1.0.1e-fips 11 Feb 2013
: built on: Wed Mar 22 21:43:28 UTC 2017
Library version: PCRE: Compile: 8.20
Runtime: 8.20 2011-10-21

Mateusz

2018-03-06 13:34 GMT+01:00 Jeremy Harris via Exim-users <exim-users@exim.org
>:

> On 05/03/18 07:32, Mateusz Krawczyk via Exim-users wrote:
> > I'm running latest Centos 6 with exim 4.90.1. Recently I have noticed
> that
> > some exim process in my system hangs on phase "handling TLS incoming
> > connection" using 100% CPU for hours.
>
> Exim binary supplied by a centos package, or compiled yourself?
>
> Also, please run "exim -d -bV" and give the initial output, down
> as far as the "WHITELIST_D_MACROS" line (we're mainly interested in
> the library versions).
>
> --
> Thanks,
> Jeremy
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
>
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim process stack 100% CPU? [ In reply to ]
On 07/03/18 09:19, Mateusz Krawczyk via Exim-users wrote:
>> Also, please run "exim -d -bV" and give the initial output, down
>> as far as the "WHITELIST_D_MACROS" line (we're mainly interested in
>> the library versions).

OK, OpenSSL 1.0.1e. Given the "handling TLS incoming connection"
status, I'm guessing it's looping around tls_refill(), meaning
that there's a debug_printf() that should show. Is this a
situation you can easily repeat? If so, run with debug
enabled (either from cmdline or enabled by ACL - which means
you can target a known connection source) to verify my guess.

The loop, if present, would imply that the errorhandling is
wrong in that routine. I'd have thought that a zero syscall
read() return (as seen in your strace) would result in
some non-SSL_ERROR_NONE return from SSL_read() but...
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim process stack 100% CPU? [ In reply to ]
I'm not able to repeat this situation. I have tried different scenarios
using openssl s_client. I get the same log message, the same ciphers etc.
but everything works fine.
The problem is connected (hopefully) with one particular server (owned by
some hosting company) and as I think, compromised user account which is
sending spam messages to auto-generated e-mail addresses.

Thank you,
Mateusz

2018-03-08 12:36 GMT+01:00 Jeremy Harris via Exim-users <exim-users@exim.org
>:

> On 07/03/18 09:19, Mateusz Krawczyk via Exim-users wrote:
> >> Also, please run "exim -d -bV" and give the initial output, down
> >> as far as the "WHITELIST_D_MACROS" line (we're mainly interested in
> >> the library versions).
>
> OK, OpenSSL 1.0.1e. Given the "handling TLS incoming connection"
> status, I'm guessing it's looping around tls_refill(), meaning
> that there's a debug_printf() that should show. Is this a
> situation you can easily repeat? If so, run with debug
> enabled (either from cmdline or enabled by ACL - which means
> you can target a known connection source) to verify my guess.
>
> The loop, if present, would imply that the errorhandling is
> wrong in that routine. I'd have thought that a zero syscall
> read() return (as seen in your strace) would result in
> some non-SSL_ERROR_NONE return from SSL_read() but...
> --
> Cheers,
> Jeremy
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
>
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Exim process stack 100% CPU? [ In reply to ]
On 09/03/18 14:05, Mateusz Krawczyk via Exim-users wrote:
> The problem is connected (hopefully) with one particular server (owned by
> some hosting company) and as I think, compromised user account which is
> sending spam messages to auto-generated e-mail addresses.

OK, so enable debug for mail from just that one source.
After you get one, mail me the debug file if you don't want
to look at it yourself. You might want to delete the
part after the first few iterations, if it is indeed stuck
in a loop from which it is logging.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/