Mailing List Archive

#1053: Persistent storage: space leakage
#1053: Persistent storage: space leakage
----------------------+-----------------------------------------------------
Reporter: dumbbell | Type: defect
Status: new | Priority: normal
Milestone: | Component: varnishd
Version: trunk | Severity: major
Keywords: |
----------------------+-----------------------------------------------------
With persistent storage, at some point in time, the silo will have the
following layout:
{{{
|xxxxxxxESxxxx_|
}}}

where:
* `S`: start of the segments list
* `E`: end of the segments list
* `x`: segments in between
* `_`: unused space

With such a situation, the `smp_open_segs` function
(source:bin/varnishd/storage/storage_persistent.c) is responsible for
finding free space. To do this, it drops the first elements of the list
(starting from S) until it has "`free_reserve`" bytes of free space
between `E` and the new `S`:
{{{
|xxxxxxxE___Sx_|
^^^ free_reserve
}}}

When the segments at the tail of the silo are all cleared and there's
still not enough space, the function starts to reclaim space at the front
of the silo, until it reaches the `free_reserve`:
{{{
|__SxxxxE______|
^^ free_reserve
}}}

It doesn't take into account the space freed at the tail.

Unfortunately, when working on the tail of the silo, this function only
considers the distance between `E` and the segment closest to the end of
the silo. Therefore, it may found there's not enough space to satisfy
`free_reserve` between those two points but there is between `E` and the
end of the silo:
{{{
|xxxxxxxE____S_|
^^^^^^ free_reserve (but not between E and S)
}}}

In this special case, it wraps the list too early. And later, when the
same situation occurs, it won't try to reclaim the space between the
segment where the list wraps and the end of the silo:
{{{
|xxxxESxx______|
^^^^^^ leaked space
}}}

The bugfix (patch attached) consists of checking this situation before
reclaiming space at the front of the silo.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1053>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1053: Persistent storage: space leakage [ In reply to ]
#1053: Persistent storage: space leakage
----------------------+-----------------------------------------------------
Reporter: dumbbell | Type: defect
Status: new | Priority: normal
Milestone: | Component: varnishd
Version: trunk | Severity: major
Keywords: |
----------------------+-----------------------------------------------------

Comment(by dumbbell):

I forgot to mention how I tested the patch...

In smp_open_segs(), I added the following printf, to know where new
segments will be stored after the silo is reloaded:
{{{
printf("Free offset: %lu\n", sc->free_offset);
}}}
(after line 180 in source:bin/varnishd/storage/storage_persistent.c).

I then started with a freshly created silo of 5Mb and used curl(1) to fill
the cache.

* With an unpatched Varnish, the maximum free offset decreases after each
complete "rewrite" of the silo. This continues until this maximum free
offset is too low and the process pan_ic() (caused by "`assert (l >=
sc->free_reserve);`"): there's not `free_reserve` bytes between "`se`" and
the element closest to the end of the silo AND not `free_reserve` bytes
between the first offset (sc->ident->stuff[SMP_SPC_STUFF]) and "se".
* With a patched Varnish, the maximum free offset stays around
`sc->mediasize`.

I confirmed this with a bigger silo (1Gb) and more clients/requests.

--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1053#comment:1>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs
Re: #1053: Persistent storage: space leakage [ In reply to ]
#1053: Persistent storage: space leakage
----------------------+-----------------------------------------------------
Reporter: dumbbell | Owner: martin
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: trunk
Severity: major | Keywords:
----------------------+-----------------------------------------------------
Changes (by martin):

* owner: => martin


--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1053#comment:2>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator

_______________________________________________
varnish-bugs mailing list
varnish-bugs@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-bugs