Mailing List Archive

CERTifiable nightmare
I just switched off the CERT security patch in apache-pre

It was making my disk grind horribly and slowing responses down by
~ 30%

Have we decided which 'fix' to use for this problem ?
The CERT patch is a nightmare for a busy server without
enormous amount of RAM to *waste*.

Nice apache logo at hyperreal. Shame about the cgi links
being temporarily broken.


rob
Re: CERTifiable nightmare [ In reply to ]
Have we decided which 'fix' to use for this problem ?
The CERT patch is a nightmare for a busy server without
enormous amount of RAM to *waste*.

I thought there was a fairly clear consensus --- include all the
fixes for particular stack-scribbling problems (so far, the NCSA
fix, and drtr's similar fix for translate_name), and #ifdef the
CERT fix so those who want it, and can affort it, can have it
fairly easily.

(Apache-pre probably should have been done that way, but my focus
while I was building was more on making sure that my own patches
were ready for prime time, so I just applied every security patch
I had and left it there. Hopefully, these awkwardnesses will go
away when Cliff produces the real release).

rst
Re: CERTifiable nightmare [ In reply to ]
Eeks...something wierd happened...I found a bunch of text inserted
into one of the scripts (from another file). Strange. Of course this
was the script that had a bunch of subroutines used by all :-(

} Nice apache logo at hyperreal. Shame about the cgi links
} being temporarily broken.

}-- End of excerpt from Rob Hartill



Do we have any TCL hackers out here who want to futz around
with the scripts?

Cliff
Re: CERTifiable nightmare [ In reply to ]
> Do we have any TCL hackers out here who want to futz around
> with the scripts?
>
> Cliff

I do Tcl, what's the problem?

Ay.
Re: CERTifiable nightmare [ In reply to ]
Rob wrote:
>I just switched off the CERT security patch in apache-pre
>
>It was making my disk grind horribly and slowing responses down by
>~ 30%
>
>Have we decided which 'fix' to use for this problem ?
>The CERT patch is a nightmare for a busy server without
>enormous amount of RAM to *waste*.

There may be a relatively simple solution; most of this memory is going
on static storage to hold 8Kb for every config variable. I think these can
safely be kept at the shorter length. It would probably involve changing quite
a few source files, though.

David.
Re: CERTifiable nightmare [ In reply to ]
On Tue, 14 Mar 1995, David Robinson wrote:
> Ok, I've got a patch that might help. I've rewritten httpd_alias.c so that
> it uses malloc instead of static storage for the redirect and aliases.
> This does two things:
> 1. It removes the limit of 20 redirects or 20 alias in total
> 2. It reduces the size of the data segment by 80*MAX_STRING_LEN, i.e.
> by 640kb if you've applied the cert patch.
>
> Could you try running with the CERT patch again after applying this one?
> The patch alias.patch is on hyperreal; you'll need to apply B22-security-drtr
> first.

Could someone comment on how using malloc versus static strings affects
performance?

> The web server on hyperreal is down at the moment, so I haven't got a patch
> number for it.

It's back up - the problem was in not completely separating the config
files for apache from the regular web server, and so when the machine was
rebooted this morning the regular httpd died because it didn't recognize
"Multiviews" as an acceptible option :)

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@hotwired.com brian@hyperreal.com http://www.hotwired.com/Staff/brian/
Re: CERTifiable nightmare [ In reply to ]
> I won't comment, but I will add that the patch only causes the server to
> malloc memory when reading the config file, except for an extra two mallocs
> when resolving a ~user address.

For that sake of any future fork-free apache, everyone should
make sure they 'free' malloc'ed memory after using it.

This isn't aimed at David, I haven't checked his patch yet -
will give it a whirl later.

rob
Re: CERTifiable nightmare [ In reply to ]
On Tue, 14 Mar 1995, David Robinson wrote:
> Ok, I've got a patch that might help. I've rewritten httpd_alias.c so that
> it uses malloc instead of static storage for the redirect and aliases.
> This does two things:
> 1. It removes the limit of 20 redirects or 20 alias in total
> 2. It reduces the size of the data segment by 80*MAX_STRING_LEN, i.e.
> by 640kb if you've applied the cert patch.
>
> Could you try running with the CERT patch again after applying this one?
> The patch alias.patch is on hyperreal; you'll need to apply B22-security-drtr
> first.

Okay, it's implemented - I modified the main hyperreal web server (stock
NCSA) with the CERT one-line patch and put drtr's patches into my updated
apache-pre, and here's the results on BSDI for active memory usage:

stock NCSA stock w/CERT apache w/drtr & CERT apache w/o CERT
parent 472K 1272K 664K 488K
child 484K 1740K 1124K 500K

(note - apache hadn't been stripped, and was compiled with debugging on,
while NCSA's wasn't)

So there's still a place where the CERT patchs add 468K after the fork
that maybe should be worked on... 1124 is still too much for HW's servers
and probably rst's too. But drtr's patches definitely helped.

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@hotwired.com brian@hyperreal.com http://www.hotwired.com/Staff/brian/
Re: CERTifiable nightmare [ In reply to ]
[Robert S. Thau]
| (NB some security mavens are in the habit of recommending that people
| run Web servers in inetd-type setups inside TCP wrappers. They have
| the ghost of a point...).

True security mavens won't, because inetd does not give you control
of the running group-id :)

-- N.
Re: CERTifiable nightmare [ In reply to ]
Date: Tue, 14 Mar 95 19:25 GMT
From: drtr@ast.cam.ac.uk (David Robinson)

I won't comment, but I will add that the patch only causes the server to
malloc memory when reading the config file, except for an extra two mallocs
when resolving a ~user address.

David.

This will potentially affect people running the server under inetd,
but those people are already screwed so badly anyway that I'm not at
all sure they're worth worrying about.

(NB some security mavens are in the habit of recommending that people
run Web servers in inetd-type setups inside TCP wrappers. They have
the ghost of a point...).

rst
Re: mallocs in child processes [ In reply to ]
FWIW, the content-arb code preallocates space to store a certain
number of Accept: headers for clients, and goes to the well (malloc())
if a client submits more. The way things are set up now, there is a
function which is intended to be called to reset for the next
transaction. However, that function arranges to reuse the allocated
memory, rather than free it outright.

So, if the server is set up for 40 Accept:s, the current base case,
and a client subsequently submits, say 50, the server will allocate
space for 80 --- it doubles --- and use all that space for subsequent
requests. This means that multiple connections from a given piggy
client will only cause malloc() to be called once.

rst
Re: CERTifiable nightmare [ In reply to ]
>From: Rob Hartill <hartill@ooo.lanl.gov>
>Date: Mon, 13 Mar 95 14:07:06 MST
>
>I just switched off the CERT security patch in apache-pre
>
>It was making my disk grind horribly and slowing responses down by
>~ 30%
>
>Have we decided which 'fix' to use for this problem ?
>The CERT patch is a nightmare for a busy server without
>enormous amount of RAM to *waste*.

Ok, I've got a patch that might help. I've rewritten httpd_alias.c so that
it uses malloc instead of static storage for the redirect and aliases.
This does two things:
1. It removes the limit of 20 redirects or 20 alias in total
2. It reduces the size of the data segment by 80*MAX_STRING_LEN, i.e.
by 640kb if you've applied the cert patch.

Could you try running with the CERT patch again after applying this one?
The patch alias.patch is on hyperreal; you'll need to apply B22-security-drtr
first.

The web server on hyperreal is down at the moment, so I haven't got a patch
number for it.

David.
Re: CERTifiable nightmare [ In reply to ]
Brian wrote:
>Could someone comment on how using malloc versus static strings affects
>performance?
I won't comment, but I will add that the patch only causes the server to
malloc memory when reading the config file, except for an extra two mallocs
when resolving a ~user address.

David.
Re: CERTifiable nightmare [ In reply to ]
Rob wrote:
>For that sake of any future fork-free apache, everyone should
>make sure they 'free' malloc'ed memory after using it.
Absolutely.
However, httpd never frees memory in the child process, the code only ensures
that there is not a store leak in the parent. (True for my patch as well
8-( ) So it will be a lot of work to fix this; maybe some more sophisticated
malloc routines would help. e.g. a malloc which built a list of all the
per-connection data, and had a single routine to free it.
However, you'd still have a problem with increasing amounts of garbage.

A while ago I thought about this in the context of a multi-threaded server,
and the conclusion I came to was that in general you could make do with
a fixed amount of memory for each connection; i.e. no dynamic memory allocation
after connection setup. The only exception was for setting environment variables
for CGI scripts; but as the script will be run as a child, the allocation could
be done after the fork() but before the exec(). So I was quite disappointed
to find httpd mallocing lots of arrays whilst processing a request.

David.