Mailing List Archive

Threads in 4.0
Hi,

So I migrated a site dealing with a quite simple Varnish config
(basically caching images and static content, passing through dynamic
requests with no modifications etc) last week. Since then I've had to
restart varnish twice due to what looks like a thread pool leak.

Varnish is configured with a max thread count of 1000 - the other
thread options are defaults. With all defaults the problem still
occurs. This is a Debian Wheezy system using the Varnish packages (ie
not system packages).

varnishstat shows MAIN.threads as 500 and has ~7.5k, 8k for
created/destroyed (11 for limited). Varnish has served a TB of content
since restart a day ago in this state.

`ps auxH | grep varnish | wc -l` shows varnish has 7508
threads/processes running.

This graph shows the rapid increase in running threads - I restarted
varnish with the lower pool limit at the end of the 13th:
https://i.imgur.com/488IT7z.png

Any ideas?

--
Cheers,
James

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Threads in 4.0 [ In reply to ]
On Thu, May 15, 2014 at 1:41 PM, James Harrison
<james@talkunafraid.co.uk> wrote:
> Hi,
>
> So I migrated a site dealing with a quite simple Varnish config
> (basically caching images and static content, passing through dynamic
> requests with no modifications etc) last week. Since then I've had to
> restart varnish twice due to what looks like a thread pool leak.
>
> Varnish is configured with a max thread count of 1000 - the other
> thread options are defaults. With all defaults the problem still
> occurs. This is a Debian Wheezy system using the Varnish packages (ie
> not system packages).
>
> varnishstat shows MAIN.threads as 500 and has ~7.5k, 8k for
> created/destroyed (11 for limited). Varnish has served a TB of content
> since restart a day ago in this state.
>
> `ps auxH | grep varnish | wc -l` shows varnish has 7508
> threads/processes running.
>
> This graph shows the rapid increase in running threads - I restarted
> varnish with the lower pool limit at the end of the 13th:
> https://i.imgur.com/488IT7z.png
>
> Any ideas?

Hi,

I remember this:
https://www.varnish-cache.org/lists/pipermail/varnish-commit/2014-May/011395.html

Dridi

> --
> Cheers,
> James
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Threads in 4.0 [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 15/05/2014 14:19, Dridi Boukelmoune wrote:
>
> I remember this:
> https://www.varnish-cache.org/lists/pipermail/varnish-commit/2014-May/011395.html
>
>
Good to know. There's more than just that one bug, though - just had
an hourlong site outage due to varnish's child just up and dying and
not being restarted. Wouldn't know where to look for debug output -
restarted the parent and it worked again for about 3 minutes and then
the child didn't die but it did stop servicing requests. New requests
would be accepted, and held open, but nothing happened subsequently.

Just downgraded to 3.0 and everything's working again. Persistent
storage is a nice-to-have to avoid cold caches after restarts/crashes,
but 4.0 clearly needs some work before it goes anywhere near production.

Happy to help test things and kick the tyres/provide more info if
that'd be of use...

- --
Cheers,
James Harrison
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQIcBAEBAgAGBQJTdnPyAAoJENTyYHL8dmp9DJEP/1fWCfo6hLnDAl5WQWg3tmUr
iAYOuJGe0nHOgVquXbZqDIkv4Se5LRBRPBGvVs0ywfErKmfZ0CF7S7SiyDtGMvK5
xdLUEjqCs6qjE6wcPi8JILfTJWqAwIAsWb+bG9ABmbKyhz5jP1A3eZv6moVB8i6U
TUJA9nU8ksmqGF/wDlqcWpqeCMLEJcWKO8wQ2y+No7ZhIkX2ItSzyhYZ75CPLxWN
9UFY+ND8FaeLFU+TOB2iS7gubJHMzh2AnzjGNRWSMt4ZVsZHJW9U22EbmJlBMuDC
jrH7wSR4HbqjMaizh+eMIgXhCON0uB1PwMvedzrH2R1qwv+38NGjp+eSYTiFi+cm
bhqT1MPLxT43aWb32jzIHrYIxrxpX/V2ZTc7N1GoqdoKF0lLZ/JD87qfNa2m69w/
rmnPnAM51rGgAdzoRD2N6Eja/yReMtO8C7BGv0SaA5VLv0HOhlMO5uNkd2E53ulE
Osxa9ddlDIMEylDs9d0PwYw9nhtsWLp1g6AVFkC9ktjFuN20+ZApKmsf1BR7a0EQ
X1jjcqFPAcxteDn0A2sO+D7L9xaE49WYqtLo6EridnzfHDmAjPk0WY53CP1iAvaX
C8A3KNwraa9G878VJDKDMwikbPO0BWw05J6EGgKAPs5ScRdaT4eNc4/71yNBR9O7
6pooJ7BI3OsxxiMn0+jU
=wpgb
-----END PGP SIGNATURE-----

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Threads in 4.0 [ In reply to ]
On Fri, May 16, 2014 at 09:24:18PM +0100, James Harrison wrote:
> >
> Good to know. There's more than just that one bug, though - just had
> an hourlong site outage due to varnish's child just up and dying and
> not being restarted. Wouldn't know where to look for debug output -
> restarted the parent and it worked again for about 3 minutes and then
> the child didn't die but it did stop servicing requests. New requests
> would be accepted, and held open, but nothing happened subsequently.
>
> Just downgraded to 3.0 and everything's working again. Persistent
> storage is a nice-to-have to avoid cold caches after restarts/crashes,
> but 4.0 clearly needs some work before it goes anywhere near production.

It sounds like you've upgraded to get persistent? Persistent is still
experimental, just as it was in 3.0.

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Threads in 4.0 [ In reply to ]
Hi,

On Mon, May 19, 2014 at 8:54 AM, Andreas Plesner Jacobsen <apj@mutt.dk>wrote:

> On Fri, May 16, 2014 at 09:24:18PM +0100, James Harrison wrote:
> > >
> > Good to know. There's more than just that one bug, though - just had
> > an hourlong site outage due to varnish's child just up and dying and
> > not being restarted. Wouldn't know where to look for debug output -
> > restarted the parent and it worked again for about 3 minutes and then
> > the child didn't die but it did stop servicing requests. New requests
> > would be accepted, and held open, but nothing happened subsequently.
> >
> > Just downgraded to 3.0 and everything's working again. Persistent
> > storage is a nice-to-have to avoid cold caches after restarts/crashes,
> > but 4.0 clearly needs some work before it goes anywhere near production.
>
> It sounds like you've upgraded to get persistent? Persistent is still
> experimental, just as it was in 3.0.
>

Experimental is misleading, it makes it sound like the code has a future. I
think the persistence code is still there because nobody has bothered
disabling it yet.

There is no work being done on the persistent. I doubt someone will pick it
up.

--
<http://www.varnish-software.com/> *Per Buer*
CTO | Varnish Software
Phone: +47 958 39 117 | Skype: per.buer
We Make Websites Fly!

Winner of the Red Herring Top 100 Global Award 2013
Re: Threads in 4.0 [ In reply to ]
On Mon, May 19, 2014 at 09:09:58AM +0200, Per Buer wrote:
> > >
> > > Just downgraded to 3.0 and everything's working again. Persistent
> > > storage is a nice-to-have to avoid cold caches after restarts/crashes,
> > > but 4.0 clearly needs some work before it goes anywhere near production.
> >
> > It sounds like you've upgraded to get persistent? Persistent is still
> > experimental, just as it was in 3.0.
> >
>
> Experimental is misleading, it makes it sound like the code has a future. I
> think the persistence code is still there because nobody has bothered
> disabling it yet.
>
> There is no work being done on the persistent. I doubt someone will pick it
> up.

True. My point was mostly that I would expect persistent to act just as bad in
3.0 as in 4.0, so it should never be the reason for upgrading.

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Threads in 4.0 [ In reply to ]
On 19/05/14 07:54, Andreas Plesner Jacobsen wrote:
> It sounds like you've upgraded to get persistent? Persistent is
> still experimental, just as it was in 3.0.

Upgraded to be on the current latest stable maintained build, not to use
persistent. I like to do upgrades in a controlled manner while the old
version is still viable so finding bugs isn't catastrophic!

Persistent I only switched to when it was clear to work
around the thread issue I was going to be restarting the thing a lot
till a fix got issued/I had time to build from source to see if that
commit fixed it. If the persistent storage is broken/dodgy to the
extent it was causing the whole thing to lock up and die then it
shouldn't be in the release imho.

Experimental is certainly the wrong wording - deprecated, perhaps?

I'll try again when >4.0.0 is out with a fix for the thread pool issue
and see how things work using the usual transient memory store.
--
Cheers,
James

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: Threads in 4.0 [ In reply to ]
Interesting... Doe the site say the persistent store is
deprecated/unmaintained? That explains the periodic crashes I get.. ( I'll
resuscitate the ticket i had created some months anyway...)
Cheers,
On 19/05/2014 5:12 pm, "Per Buer" <perbu@varnish-software.com> wrote:

> Hi,
>
> On Mon, May 19, 2014 at 8:54 AM, Andreas Plesner Jacobsen <apj@mutt.dk>wrote:
>
>> On Fri, May 16, 2014 at 09:24:18PM +0100, James Harrison wrote:
>> > >
>> > Good to know. There's more than just that one bug, though - just had
>> > an hourlong site outage due to varnish's child just up and dying and
>> > not being restarted. Wouldn't know where to look for debug output -
>> > restarted the parent and it worked again for about 3 minutes and then
>> > the child didn't die but it did stop servicing requests. New requests
>> > would be accepted, and held open, but nothing happened subsequently.
>> >
>> > Just downgraded to 3.0 and everything's working again. Persistent
>> > storage is a nice-to-have to avoid cold caches after restarts/crashes,
>> > but 4.0 clearly needs some work before it goes anywhere near production.
>>
>> It sounds like you've upgraded to get persistent? Persistent is still
>> experimental, just as it was in 3.0.
>>
>
> Experimental is misleading, it makes it sound like the code has a future.
> I think the persistence code is still there because nobody has bothered
> disabling it yet.
>
> There is no work being done on the persistent. I doubt someone will pick
> it up.
>
> --
> <http://www.varnish-software.com/> *Per Buer*
> CTO | Varnish Software
> Phone: +47 958 39 117 | Skype: per.buer
> We Make Websites Fly!
>
> Winner of the Red Herring Top 100 Global Award 2013
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Re: Threads in 4.0 [ In reply to ]
On Tue, May 20, 2014 at 08:39:33AM +1000, Norberto Meijome wrote:

> Interesting... Doe the site say the persistent store is
> deprecated/unmaintained?

man varnishd has the following to say:

persistent (experimental)
syntax: persistent,path,size

Persistent storage. Varnish will store objects in a file in a manner
that will secure the survival of most of the objects in the event of a
planned or unplanned shutdown of Varnish. The persistent storage back‐
end has multiple issues with it and will likely be removed from a
future version of Varnish.

--
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc