Mailing List Archive

"not cacheable" even on static files
Hi folks, for the past few days I've been puzzled by this problem -
Varnish have refused to cache even static files. While it was fine
before, and will cache even dynamic HTML files (with proper cache
headers, of course)

The problem begun when my client reported that his website was
constantly going up & down. When I checked, it was under some sort of
DDoS attack. This is not news, because his website was pretty much
under 24x7 syn flood attack.
However, this attack is now able to bypass HAproxy & Varnish - and hit
Apache directly, right where it hurts most.

To cut the long story short - basically now Varnish refuses to cache
almost everything.
The X-Cacheable header that I enabled contains the dreaded "NO:Not
Cacheable" status. And I've not been able to find out why.

This is where I stuck, if I can find out the cause, the I'd be able to
rectify it.

Anyway, the DDoS attack may not be related at all to the current
problem (varnish not caching), I included the story just for the sake
of completeness.

Let's proceed to some facts that I've gathered :

Same GIF file - but different sizes (note the "Received" column) :
http://minus.com/mbawzSZUxJ#3

The URLs with port 8181 are direct requests to Varnish, bypassing HAproxy.

HTTP Headers produced by Varnish : http://minus.com/mbawzSZUxJ#4

HTTP Headers produced by HAproxy : http://minus.com/mbawzSZUxJ#2

List of cookies sent by browser & received from Varnish :
http://minus.com/mbawzSZUxJ#1
(stripped clean by varnish, basically)

Please find the result of varnishstat -1 & content of
/etc/varnish/default.vcl attached to the end of this email.

Varnish is running with the following parameters :

$ ps aux|grep varnish
myuser 3327 0.0 0.0 3324 796 pts/6 S+ 04:36 0:00 grep varnish
root 8441 0.0 1.0 86024 83532 pts/2 SL+ Jan12 0:01
/usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:8181 -f
/etc/varnish/default.vcl -T 127.0.0.1:6082 -t 180 -w 2,1000,30 -s
malloc,2G -d


Hopefully someone will be able to point / give me a hint to the right direction.


Thanks,
Harry

===================

$ telnet localhost 6082
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
200 199
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,2.6.32-25-generic-pae,i686,-smalloc,-hcritbit

Type 'help' for command list.
Type 'quit' to close CLI session.

stats
200 1978
16564302 Client connections accepted
16514647 Client requests received
2378064 Cache hits
988836 Cache misses
592750 Backend conn. success
462880 Backend conn. failures
13096234 Backend conn. reuses
63305 Backend conn. was closed
13160380 Backend conn. recycles
82 Fetch head
9937334 Fetch with Length
1856 Fetch chunked
245 Fetch wanted close
66 Fetch failed
26436 N struct sess_mem
26091 N struct sess
18773 N struct object
18862 N struct objectcore
6827 N struct objecthead
3 N struct vbe_conn
107 N worker threads
26496 N worker threads created
144188 N overflowed work requests
1 N backends
932755 N expired objects
1851315 N LRU moved objects
12437230 Objects sent with write
16564302 Total Sessions
16514647 Total Requests
870 Total pipe
13146877 Total pass
13630039 Total fetch
6464791313 Total header bytes
122615308544 Total body bytes
16564289 Session Closed
11 Session Linger
17 Session herd
1364697125 SHM records
93867958 SHM writes
2574 SHM flushes due to overflow
446883 SHM MTX contention
540 SHM cycles through buffer
10815252 SMA allocator requests
37554 SMA outstanding allocations
45497812 SMA outstanding bytes
114575871831 SMA bytes allocated
114530374019 SMA bytes free
505674 SMS allocator requests
211877406 SMS bytes allocated
211877406 SMS bytes freed
13688813 Backend requests made
1 N vcl total
1 N vcl available
1 N total active purges
1 N new purges added
3365986 HCB Lookups without lock
122890 HCB Lookups with lock
122890 HCB Inserts
99485 Client uptime
16426 Backend conn. retry
3690588 Fetch no body (304)

=================== default.vcl ===============

$ cat /etc/varnish/default.vcl

# Default backend definition. Set this to point to your content
# server.
backend default {
.host = "127.0.0.1";
.port = "81";
}


sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
}

# Below is a commented-out copy of the default VCL logic. If you
# redefine any of these subroutines, the built-in logic will be
# appended to your code.
#
sub vcl_recv {

if (req.backend.healthy) {
set req.grace = 180s;
} else {
set req.grace = 1h;
}
}



sub vcl_fetch {

set beresp.grace = 1h;

# Varnish determined the object was not cacheable
if (!beresp.cacheable) {
set beresp.http.X-Cacheable = "NO:Not Cacheable";

# You don't wish to cache content for logged in users
} elsif (req.http.Cookie ~ "(UserID|_session)") {
set beresp.http.X-Cacheable = "NO:Got Session";
return(pass);

# You are respecting the Cache-Control=private header from the backend
} elsif (beresp.http.Cache-Control ~ "private") {
set beresp.http.X-Cacheable = "NO:Cache-Control=private";
return(pass);

# You are extending the lifetime of the object artificially
} elsif (beresp.ttl < 1s) {
set beresp.ttl = 5s;
set beresp.grace = 5s;
set beresp.http.X-Cacheable = "YES:FORCED";

# Varnish determined the object was cacheable
} else {
set beresp.http.X-Cacheable = "YES";
}

if (req.url ~
"\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$")
{
unset beresp.http.set-cookie;
set beresp.ttl = 24h;
}

return(deliver);

# if (!beresp.cacheable) {
# return (pass);
# }
# if (beresp.http.Set-Cookie) {
# return (pass);
# }
# return (deliver);
}

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Re: "not cacheable" even on static files [ In reply to ]
Hi Harry,

Is HAProxy in front of your Varnish servers or between varnish and Apache?

You could use HAProxy to mitigate the attack, if you know the pattern,
I can help on this point.

cheers


On Thu, Jan 12, 2012 at 10:37 PM, Harry Sufehmi <sufehmi@gmail.com> wrote:
> Hi folks, for the past few days I've been puzzled by this problem -
> Varnish have refused to cache even static files. While it was fine
> before, and will cache even dynamic HTML files (with proper cache
> headers, of course)
>
> The problem begun when my client reported that his website was
> constantly going up & down. When I checked, it was under some sort of
> DDoS attack. This is not news, because his website was pretty much
> under 24x7 syn flood attack.
> However, this attack is now able to bypass HAproxy & Varnish - and hit
> Apache directly, right where it hurts most.
>
> To cut the long story short - basically now Varnish refuses to cache
> almost everything.
> The X-Cacheable header that I enabled contains the dreaded "NO:Not
> Cacheable" status. And I've not been able to find out why.
>
> This is where I stuck, if I can find out the cause, the I'd be able to
> rectify it.
>
> Anyway, the DDoS attack may not be related at all to the current
> problem (varnish not caching), I included the story just for the sake
> of completeness.
>
> Let's proceed to some facts that I've gathered :
>
> Same GIF file - but different sizes (note the "Received" column) :
> http://minus.com/mbawzSZUxJ#3
>
> The URLs with port 8181 are direct requests to Varnish, bypassing HAproxy.
>
> HTTP Headers produced by Varnish : http://minus.com/mbawzSZUxJ#4
>
> HTTP Headers produced by HAproxy : http://minus.com/mbawzSZUxJ#2
>
> List of cookies sent by browser & received from Varnish :
> http://minus.com/mbawzSZUxJ#1
> (stripped clean by varnish, basically)
>
> Please find the result of varnishstat -1 & content of
> /etc/varnish/default.vcl attached to the end of this email.
>
> Varnish is running with the following parameters :
>
> $ ps aux|grep varnish
> myuser   3327  0.0  0.0   3324   796 pts/6    S+   04:36   0:00 grep varnish
> root      8441  0.0  1.0  86024 83532 pts/2    SL+  Jan12   0:01
> /usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:8181 -f
> /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 180 -w 2,1000,30 -s
> malloc,2G -d
>
>
> Hopefully someone will be able to point / give me a hint to the right direction.
>
>
> Thanks,
> Harry
>
> ===================
>
> $ telnet localhost 6082
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> 200 199
> -----------------------------
> Varnish Cache CLI 1.0
> -----------------------------
> Linux,2.6.32-25-generic-pae,i686,-smalloc,-hcritbit
>
> Type 'help' for command list.
> Type 'quit' to close CLI session.
>
> stats
> 200 1978
>    16564302  Client connections accepted
>    16514647  Client requests received
>     2378064  Cache hits
>      988836  Cache misses
>      592750  Backend conn. success
>      462880  Backend conn. failures
>    13096234  Backend conn. reuses
>       63305  Backend conn. was closed
>    13160380  Backend conn. recycles
>          82  Fetch head
>     9937334  Fetch with Length
>        1856  Fetch chunked
>         245  Fetch wanted close
>          66  Fetch failed
>       26436  N struct sess_mem
>       26091  N struct sess
>       18773  N struct object
>       18862  N struct objectcore
>        6827  N struct objecthead
>           3  N struct vbe_conn
>         107  N worker threads
>       26496  N worker threads created
>      144188  N overflowed work requests
>           1  N backends
>      932755  N expired objects
>     1851315  N LRU moved objects
>    12437230  Objects sent with write
>    16564302  Total Sessions
>    16514647  Total Requests
>         870  Total pipe
>    13146877  Total pass
>    13630039  Total fetch
>  6464791313  Total header bytes
> 122615308544  Total body bytes
>    16564289  Session Closed
>          11  Session Linger
>          17  Session herd
>  1364697125  SHM records
>    93867958  SHM writes
>        2574  SHM flushes due to overflow
>      446883  SHM MTX contention
>         540  SHM cycles through buffer
>    10815252  SMA allocator requests
>       37554  SMA outstanding allocations
>    45497812  SMA outstanding bytes
> 114575871831  SMA bytes allocated
> 114530374019  SMA bytes free
>      505674  SMS allocator requests
>   211877406  SMS bytes allocated
>   211877406  SMS bytes freed
>    13688813  Backend requests made
>           1  N vcl total
>           1  N vcl available
>           1  N total active purges
>           1  N new purges added
>     3365986  HCB Lookups without lock
>      122890  HCB Lookups with lock
>      122890  HCB Inserts
>       99485  Client uptime
>       16426  Backend conn. retry
>     3690588  Fetch no body (304)
>
> =================== default.vcl ===============
>
> $ cat /etc/varnish/default.vcl
>
> # Default backend definition.  Set this to point to your content
> # server.
>  backend default {
>     .host = "127.0.0.1";
>     .port = "81";
>  }
>
>
> sub vcl_deliver {
>        if (obj.hits > 0) {
>                set resp.http.X-Cache = "HIT";
>        } else {
>                set resp.http.X-Cache = "MISS";
>        }
> }
>
> # Below is a commented-out copy of the default VCL logic.  If you
> # redefine any of these subroutines, the built-in logic will be
> # appended to your code.
> #
>  sub vcl_recv {
>
>  if (req.backend.healthy) {
>    set req.grace = 180s;
>  } else {
>    set req.grace = 1h;
>  }
>  }
>
>
>
>  sub vcl_fetch {
>
>   set beresp.grace = 1h;
>
>    # Varnish determined the object was not cacheable
>    if (!beresp.cacheable) {
>        set beresp.http.X-Cacheable = "NO:Not Cacheable";
>
>    # You don't wish to cache content for logged in users
>    } elsif (req.http.Cookie ~ "(UserID|_session)") {
>        set beresp.http.X-Cacheable = "NO:Got Session";
>        return(pass);
>
>    # You are respecting the Cache-Control=private header from the backend
>    } elsif (beresp.http.Cache-Control ~ "private") {
>        set beresp.http.X-Cacheable = "NO:Cache-Control=private";
>        return(pass);
>
>    # You are extending the lifetime of the object artificially
>    } elsif (beresp.ttl < 1s) {
>        set beresp.ttl   = 5s;
>        set beresp.grace = 5s;
>        set beresp.http.X-Cacheable = "YES:FORCED";
>
>    # Varnish determined the object was cacheable
>    } else {
>        set beresp.http.X-Cacheable = "YES";
>    }
>
>    if (req.url ~
> "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$")
> {
>            unset beresp.http.set-cookie;
>            set beresp.ttl   = 24h;
>    }
>
>    return(deliver);
>
> #     if (!beresp.cacheable) {
> #         return (pass);
> #     }
> #     if (beresp.http.Set-Cookie) {
> #         return (pass);
> #     }
> #     return (deliver);
>  }
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

_______________________________________________
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc