Mailing List Archive

Updates to mythtv/external and required system libraries
Hi everyone,

I just pushed changes to the master branch to prefer system libraries
over the code in the mythtv/external. The changes are:

1) On all systems, the following libraries will be used :

libhdhomerun
libsamlplerate
lzo (or lzo2) instead of minilzo
minizip

2) On all systems except Centos 7, Debian Jessie, and Ubuntu 14.04 the
system libbluray will be used.

3) On all systems except Suse and FreeBSD, the system libXNVCtrl will
be used.

This means that after your next git update, your development directory
may not configure/compile until you add some system libraries. The
ansible scripts have been updated, so if you installed your compilation
dependencies that way you can pull updated playbooks and re-run them.
If you installed manually, you will likely need to install the
following or their equivalents:

hdhomerun-devel
libbluray-devel
libsamplerate-devel
libXNVCtrl-devel
lzo-devel
minizip-devel

The only thing known not to work after this change is that the mythgame
plugin won't compile on Ubuntu 14.04 since that system doesn't provide
minizip.

David


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Updates to mythtv/external and required system libraries [ In reply to ]
On 08/05/2018 07:12 AM, David Hampton wrote:
> Hi everyone,
>
> I just pushed changes to the master branch to prefer system libraries
> over the code in the mythtv/external. The changes are:
>
> 1) On all systems, the following libraries will be used :
>
> libhdhomerun
> libsamlplerate
> lzo (or lzo2) instead of minilzo
> minizip
>
> 2) On all systems except Centos 7, Debian Jessie, and Ubuntu 14.04 the
> system libbluray will be used.
>
> 3) On all systems except Suse and FreeBSD, the system libXNVCtrl will
> be used.
>
> This means that after your next git update, your development directory
> may not configure/compile until you add some system libraries. The
> ansible scripts have been updated, so if you installed your compilation
> dependencies that way you can pull updated playbooks and re-run them.
> If you installed manually, you will likely need to install the
> following or their equivalents:
>
> hdhomerun-devel
> libbluray-devel
> libsamplerate-devel
> libXNVCtrl-devel
> lzo-devel
> minizip-devel
>
> The only thing known not to work after this change is that the mythgame
> plugin won't compile on Ubuntu 14.04 since that system doesn't provide
> minizip.
>
> David
David,

Does mythfilldatabase run OK for you? I'm on Ubuntu 18.04.1.

I'm compiling OK, but when I run mythfilldatabase --dd-grab-all, I get
an abort (attached.) I do have an HDHR box on this test host. ./configure
says: HDHomeRun support yes.

I did remove the old external libraries (rm -rf external/libhdhomerun/)
to make git status happy.

Also, I rip everything out of /usr/lib that's MythTV related before doing
major changes like this.

--
Bill - leaving for dinner, but will be back

--
Bill
Re: Updates to mythtv/external and required system libraries [ In reply to ]
On Sun, 2018-08-05 at 19:28 -0500, Bill Meek wrote:
> On 08/05/2018 07:12 AM, David Hampton wrote:
> > Hi everyone,
> >
> > I just pushed changes to the master branch to prefer system
> > libraries
> > over the code in the mythtv/external. The changes are:
> >
> > 1) On all systems, the following libraries will be used :
> >
> > libhdhomerun
> > libsamlplerate
> > lzo (or lzo2) instead of minilzo
> > minizip
> >
> > 2) On all systems except Centos 7, Debian Jessie, and Ubuntu 14.04
> > the
> > system libbluray will be used.
> >
> > 3) On all systems except Suse and FreeBSD, the system libXNVCtrl
> > will
> > be used.
> >
> > This means that after your next git update, your development
> > directory
> > may not configure/compile until you add some system libraries. The
> > ansible scripts have been updated, so if you installed your
> > compilation
> > dependencies that way you can pull updated playbooks and re-run
> > them.
> > If you installed manually, you will likely need to install the
> > following or their equivalents:
> >
> > hdhomerun-devel
> > libbluray-devel
> > libsamplerate-devel
> > libXNVCtrl-devel
> > lzo-devel
> > minizip-devel
> >
> > The only thing known not to work after this change is that the
> > mythgame
> > plugin won't compile on Ubuntu 14.04 since that system doesn't
> > provide
> > minizip.
> >
> > David
>
> David,
>
> Does mythfilldatabase run OK for you? I'm on Ubuntu 18.04.1.

Runs fine on my Fedora28 system. I believe that system is set up to
talk to a networked HDHR box.

> I'm compiling OK, but when I run mythfilldatabase --dd-grab-all, I
> get
> an abort (attached.) I do have an HDHR box on this test host.
> ./configure
> says: HDHomeRun support yes.
>
> I did remove the old external libraries (rm -rf
> external/libhdhomerun/)
> to make git status happy.
>
> Also, I rip everything out of /usr/lib that's MythTV related before
> doing
> major changes like this.

The crash is here in #5:

Thread 1 (Thread 0x7fbfdde28940 (LWP 5000)):
#4 0x00007fbfdab3ec92 in __stack_chk_fail () at stack_chk_fail.c:29
#5 0x00007fbfdd00cfbe in CardUtil::IsCableCardPresent(unsigned int,
QString const&) (inputid=1, inputType=...) at cardutil.cpp:228
__FUNCTION__ = "IsCableCardPresent"

Its complaining that something in CardUtil::IsCableCardPresent is
overwriting the stack, which is interesting since it really doesn't
write much to the stack. It stores an int, a pointer, a QString, and
an hdhomerun_tuner_status_t structure. That's probably the culprit,
given that it has two embedded char[32] strings. The call to
hdhomerun_device_get_oob_status() could possibly be writing garbage
into one of the string pointers and trashing the stack, but its not
noticed until the return from IsCableCardPresent.

I can try and set up an Ubuntu host with my networked HDHR, but it will
take me a couple of days to get to it.

David


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Updates to mythtv/external and required system libraries [ In reply to ]
On Mon, Aug 6, 2018 at 2:31 AM David Hampton <mythtv@love2code.net> wrote:

> Its complaining that something in CardUtil::IsCableCardPresent is
> overwriting the stack, which is interesting since it really doesn't
> write much to the stack. It stores an int, a pointer, a QString, and
> an hdhomerun_tuner_status_t structure. That's probably the culprit,
> given that it has two embedded char[32] strings.

This is just an educated guess, but.....

Be aware, that as the libhdhomerun issue
https://github.com/Silicondust/libhdhomerun/issues/4
points out (which SiliconDust had decided to not address), the
various releases of libhdhomerun are most specifically *NOT*
compatible, and can result in overflow of buffers or not properly
processing the contents of the structs because they use the
same library versions for different results (the struct changed
sizes). I considered this lack of versioning a mistake (which
was the point of the github issue which provided a method for
old calls to continue to work while providing newer apps the
capability of using the newer features), and is why I (knowingly)
explicitly decided to update the mythtv external library rather
than even considering using a system library which could at
some point be the wrong version.

Of course, if you compile and run on the exact same system
every time any libraries are updated, there should be no
problem, but since there are few ways to provide those
guarantees, some issues may result.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Updates to mythtv/external and required system libraries [ In reply to ]
On Mon, Aug 6, 2018 at 2:57 AM Gary Buhrmaster
<gary.buhrmaster@gmail.com> wrote:
> ... and is why I (knowingly)
> explicitly decided to update the mythtv external library rather
> than even considering using a system library which could at
> some point be the wrong version.

btw, looking back at that trac ticket, I think I actually sent
in some wrong patches, but the goal was correct
(don't depend on the system libraries).
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Updates to mythtv/external and required system libraries [ In reply to ]
On Mon, 2018-08-06 at 02:57 +0000, Gary Buhrmaster wrote:
>
> Be aware, that as the libhdhomerun issue
> https://github.com/Silicondust/libhdhomerun/issues/4
> points out (which SiliconDust had decided to not address), the
> various releases of libhdhomerun are most specifically *NOT*
> compatible, and can result in overflow of buffers or not properly
> processing the contents of the structs because they use the
> same library versions for different results (the struct changed
> sizes). I considered this lack of versioning a mistake

If I understand correctly and you are saying that upstream are changing
interfaces (structs, function arguments etc) without bumping the SONAME
in an appropriate way then indeed that is a mistake.

If upstream are unwilling/unable to manage the SONAME properly then, at
least in the Debian/Ubuntu ecosystem, it falls to the package
maintainer to do so and bugs should be reported if they miss something.
I'd bet (but don't know) that the same is true in the Fedora ecosystem.

> Of course, if you compile and run on the exact same system
> every time any libraries are updated, there should be no
> problem, but since there are few ways to provide those
> guarantees, some issues may result.

As far as Bill's problem goes if he is both building and running on
Ubuntu 14.04 that ought to be sufficient -- that's a self contained set
of dev and runtime packages.

Ian.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Updates to mythtv/external and required system libraries [ In reply to ]
>
>If I understand correctly and you are saying that upstream are changing
>interfaces (structs, function arguments etc) without bumping the SONAME
>in an appropriate way then indeed that is a mistake.
>
>If upstream are unwilling/unable to manage the SONAME properly then, at
>least in the Debian/Ubuntu ecosystem, it falls to the package
>maintainer to do so and bugs should be reported if they miss something.
>I'd bet (but don't know) that the same is true in the Fedora ecosystem.
>
>>Of course, if you compile and run on the exact same system
>>every time any libraries are updated, there should be no
>>problem, but since there are few ways to provide those
>>guarantees, some issues may result.
>
>As far as Bill's problem goes if he is both building and running on
>Ubuntu 14.04 that ought to be sufficient -- that's a self contained set
>of dev and runtime packages.
>
Nick here from Silicondust/HDHomeRun...

If someone with experience in maintaining libs can point me in the right
direction, I am happy to make changes to help with
versioning/compatibility.

I can also be found on freenode #hdhomerun "jafa"

Nick
>

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Updates to mythtv/external and required system libraries [ In reply to ]
On Mon, 2018-08-06 at 13:50 +0000, Silicondust/HDHomeRun wrote:
> If someone with experience in maintaining libs can point me in the right
> direction, I am happy to make changes to help with
> versioning/compatibility.

In essence you simply need to change the SONAME of the library whenever
you make a binary incompatible change to the library ABI (e.g. adding a
field to an exported struct, changing the size of a struct member,
removing or adding an argument to a function's prototype etc).

The SONAME can be seen with objdump:

$ objdump -p /usr/lib/i386-linux-gnu/libhdhomerun.so.4| grep SONAME
SONAME libhdhomerun.so.4

It effectively defines the ABI of the library.

Many libraries have version additional numbers which are not present in
the SONAME, which are bumped for various kinds of binary-compatible
changes such as adding a completely new function call, you can see
these in the actual filename, to which the SONAME is usually a link:

$ ls -l /usr/lib/i386-linux-gnu/libhdhomerun.so*
lrwxrwxrwx 1 root root 21 Apr 2 20:26 /usr/lib/i386-linux-gnu/libhdhomerun.so.4 -> libhdhomerun.so.4.0.0
-rw-r--r-- 1 root root 87348 Apr 2 20:26 /usr/lib/i386-linux-gnu/libhdhomerun.so.4.0.0

There are various schemes which library authors use to manage those
numbers. The libtool one [1] is convenient if you are already using
libtool. Otherwise another popular one is http://semver.org (remember
only the first digit, called MAJOR in semver, becomes part of the
SONAME).

Of course all this assumes you can reliably determine when you are
changing the ABI (which can happen in unexpected ways). I don't have
good advice here, but being aware of the need to think about it at all
is a good first stepo.

https://lvc.github.io/abi-compliance-checker/ looks like it might be
useful but I've not used it myself.

HTH,
Ian.

[1] https://www.gnu.org/software/libtool/manual/libtool.html#Versioning
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Updates to mythtv/external and required system libraries [ In reply to ]
On Mon, Aug 6, 2018 at 1:51 PM Silicondust/HDHomeRun
<mythtv@silicondust.com> wrote:

> If someone with experience in maintaining libs can point me in the right
> direction, I am happy to make changes to help with
> versioning/compatibility.

I'll also point out(*) that issue #4 in your github repo
provided a viable way forward such that
packagers did not have to ship and maintain two
libraries on a system (one for any old apps named
(say) libhdhomerun, and one for any new apps
named (say) libhdhomerun2)(**), at the cost of some
likely minor (at that time) code changes on your
internal uses by using the new features when
desired, and leaving the old API the same. Now
that more of your code likely depends on the functions
it is likely the work (which I am guessing would now
be a larger distraction at this point of your projects)
makes it less viable. So one know pushes the load
to every distro packager.

(*) I am pointing it out here again since there
was zero response on the issue itself. Neither
comment, nor acceptance, nor rejection. And
yes, I was a bit unhappy of the lack of response
those 2+ years ago, but I also accepted that it
was your right to ignore the future problems.

(**) And on many community distributions,
the packagers are not coders, so would
not be typically able to review future commits
to know if they even should be back-ported to
the old(er) variant for important functional or
security reasons.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Updates to mythtv/external and required system libraries [ In reply to ]
On 08/05/2018 09:25 PM, David Hampton wrote:
> On Sun, 2018-08-05 at 19:28 -0500, Bill Meek wrote:
>> On 08/05/2018 07:12 AM, David Hampton wrote:
>>> Hi everyone,
>>>
>>> I just pushed changes to the master branch to prefer system
>>> libraries
>>> over the code in the mythtv/external. The changes are:
>>>
>>> 1) On all systems, the following libraries will be used :
>>>
>>> libhdhomerun
>>> libsamlplerate
>>> lzo (or lzo2) instead of minilzo
>>> minizip
>>>
>>> 2) On all systems except Centos 7, Debian Jessie, and Ubuntu 14.04
>>> the
>>> system libbluray will be used.
>>>
>>> 3) On all systems except Suse and FreeBSD, the system libXNVCtrl
>>> will
>>> be used.
>>>
>>> This means that after your next git update, your development
>>> directory
>>> may not configure/compile until you add some system libraries. The
>>> ansible scripts have been updated, so if you installed your
>>> compilation
>>> dependencies that way you can pull updated playbooks and re-run
>>> them.
>>> If you installed manually, you will likely need to install the
>>> following or their equivalents:
>>>
>>> hdhomerun-devel
>>> libbluray-devel
>>> libsamplerate-devel
>>> libXNVCtrl-devel
>>> lzo-devel
>>> minizip-devel
>>>
>>> The only thing known not to work after this change is that the
>>> mythgame
>>> plugin won't compile on Ubuntu 14.04 since that system doesn't
>>> provide
>>> minizip.
>>>
>>> David
>>
>> David,
>>
>> Does mythfilldatabase run OK for you? I'm on Ubuntu 18.04.1.
>
> Runs fine on my Fedora28 system. I believe that system is set up to
> talk to a networked HDHR box.
>
>> I'm compiling OK, but when I run mythfilldatabase --dd-grab-all, I
>> get
>> an abort (attached.) I do have an HDHR box on this test host.
>> ./configure
>> says: HDHomeRun support yes.
>>
>> I did remove the old external libraries (rm -rf
>> external/libhdhomerun/)
>> to make git status happy.
>>
>> Also, I rip everything out of /usr/lib that's MythTV related before
>> doing
>> major changes like this.
>
> The crash is here in #5:
>
> Thread 1 (Thread 0x7fbfdde28940 (LWP 5000)):
> #4 0x00007fbfdab3ec92 in __stack_chk_fail () at stack_chk_fail.c:29
> #5 0x00007fbfdd00cfbe in CardUtil::IsCableCardPresent(unsigned int,
> QString const&) (inputid=1, inputType=...) at cardutil.cpp:228
> __FUNCTION__ = "IsCableCardPresent"
>
> Its complaining that something in CardUtil::IsCableCardPresent is
> overwriting the stack, which is interesting since it really doesn't
> write much to the stack. It stores an int, a pointer, a QString, and
> an hdhomerun_tuner_status_t structure. That's probably the culprit,
> given that it has two embedded char[32] strings. The call to
> hdhomerun_device_get_oob_status() could possibly be writing garbage
> into one of the string pointers and trashing the stack, but its not
> noticed until the return from IsCableCardPresent.
>
> I can try and set up an Ubuntu host with my networked HDHR, but it will
> take me a couple of days to get to it.
>
> David

Update:

I was about to send a response accepting Nick's offer, but decided to
add the output of ldconfig -p just to show where libhdhomerun.so was.

Then I spotted this, the 4th line below:

$ ll `locate libhdhomerun.so`
lrwxrwxrwx 1 root root 21 Apr 2 14:26 /usr/lib/x86_64-linux-gnu/libhdhomerun.so -> libhdhomerun.so.4.0.0
lrwxrwxrwx 1 root root 21 Apr 2 14:26 /usr/lib/x86_64-linux-gnu/libhdhomerun.so.4 -> libhdhomerun.so.4.0.0
-rw-r--r-- 1 root root 75688 Apr 2 14:26 /usr/lib/x86_64-linux-gnu/libhdhomerun.so.4.0.0
-rwxr-xr-x 1 root root 93240 Feb 17 2016 /usr/local/lib/libhdhomerun.so <--------------------------------

which I renamed, to shut it off, reran ldconfig, and presto, mythfilldatabase works.
So does recording from my HDHR Extend (HDTC-2US.) I did do a make distclean before
reporting the original mfdb failure.

--
Bill
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Updates to mythtv/external and required system libraries [ In reply to ]
I found that on several ubuntu machines configure missed libizo dev
which that showed up in the build.
Easy enough to add the dev package and resume the build, but probably
configure needs the dependency.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org