Mailing List Archive

Mythtv pes_malloc and pes_free
The libmythtv implementation of mpeg2 uses its own simple heap management
overlayed on top of the standard heap allocator (malloc/free). This is
implemented by two functions in pespacket.cpp, pes_alloc and pes_free. The
reason for doing this is given in the code as "avoiding the global malloc
lock". The global malloc lock has not existed in glibc malloc for a number
of years now. Glibc malloc is based on pthreads malloc a.k.a. ptmalloc2.
This is a multithreading friendly allocator using a sophisticated multi
arena allocation algorithm.

The simple algorithm in pes_alloc does not play well in a mixed short/long
term allocation environment causing large amounts of virtual memory space
to used unnecessarily.

I would like to propose that we consider reverting to the default heap
allocator. There are also a number of alternatives to the ptmalloc
allocator that could be consider, for example tcmalloc from Google.

The topic is open to debate!

Roger


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Mythtv pes_malloc and pes_free [ In reply to ]
On Sun, May 7, 2017 at 10:53 AM, Roger James
<roger@beardandsandals.co.uk> wrote:
....
> The topic is open to debate!

Are there platform implications/idiosyncrasies?

While many platforms have reviewed and
updated their various malloc implementations
over the years [certainly glibc has] (and if
anyone asked me, I would suggest the project
should use them where they are good, or at
least good enough), I would not be at all
surprised if Windows (for example) was special
(but I have near zero knowledge of the
Windows implementation of malloc). You may
need platform SMEs to chime in.

I am not sure Daniel K (who wrote the allocator
for performance reasons) is still on the dev list,
so it might be worth trying to send email directly
to ask his opinion explicitly.


btw, while LD_PRELOAD is often considered
evil, if one moves to using malloc, using it to pull
in tcmalloc on specific under-performant systems
may be viable without requiring recompilation,
or requiring distros to build that way (and avoids
the problems with trying to use valgrind when
needed (and avoiding the tcmalloc heapchecker)).


See, no debate, just additional suggestions
for you to do more research. What could possibly
go wrong? And that is why no one asks me.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Mythtv pes_malloc and pes_free [ In reply to ]
On 07.05.2017 12:53, Roger James wrote:
> I would like to propose that we consider reverting to the default heap
> allocator. There are also a number of alternatives to the ptmalloc
> allocator that could be consider, for example tcmalloc from Google.

I like the idea. Do you know if it helps? I just read the other day that
using the standard implementations may contain some nasty surprises...
https://www.zerotier.com/blog/2017-05-05-theleak.shtml

I hear jemalloc (as used in FreeBSD) is great http://jemalloc.net/

Regards,
Karl
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Mythtv pes_malloc and pes_free [ In reply to ]
On 8 May 2017 6:43:32 am Karl Dietz <dekarl@spaetfruehstuecken.org> wrote:

> On 07.05.2017 12:53, Roger James wrote:
>> I would like to propose that we consider reverting to the default heap
>> allocator. There are also a number of alternatives to the ptmalloc
>> allocator that could be consider, for example tcmalloc from Google.
>
> I like the idea. Do you know if it helps? I just read the other day that
> using the standard implementations may contain some nasty surprises...
> https://www.zerotier.com/blog/2017-05-05-theleak.shtml
>
> I hear jemalloc (as used in FreeBSD) is great http://jemalloc.net/
>
> Regards,
> Karl

Hi Karl, Gary,

The pes_alloc stuff is is only used for allocation of PES packet data and
is of course built on top of malloc. So disabling it does not change our
exposure to malloc bugs. In fact my new code highlighted a very similar
problem in the pes_alloc code to that described by zerotier! My SDT and EIT
changes involve long term caching of PES packet data. Because these long
term allocations are randomly distributed through pes_alloc pools and the
usage profile is rapid increase to an initial peak followed by a slow decay
down to a stable level as knowledge of table section versions is gathered,
we end up with far more allocated pools than we need. So reducing the
number of memory allocators with potentially conflicting allocation
strategies we have layered on top on each other, is always in my view, a
good thing.

Does it help? Well I have been running it successfully for a couple of days
now. Only time and greater exposure will tell. It is very easy to try it
out, just put #define USING_VALGRIND at the start of pespacket.cpp.

As far as the use of other malloc replacements go I can happily leave that
discussion to you guys. I just want to get back to working on what I wanted
to do many months ago, having a look at accurate scheduling using now next
transitions and running status data!

Roger


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Mythtv pes_malloc and pes_free [ In reply to ]
On 8 May 2017 at 06:43, Karl Dietz <dekarl@spaetfruehstuecken.org> wrote:
>
> I like the idea. Do you know if it helps? I just read the other day that
> using the standard implementations may contain some nasty surprises...
> https://www.zerotier.com/blog/2017-05-05-theleak.shtml


ZeroTier isn't using libstdc++'s default allocator.
Their blog post doesn't acknowledge this or explain why.

--
Malcolm Parsons