Mailing List Archive

ttvdb v2 support for 0.27
I have back ported Mark Spieth's work to 0.27 running on mythbuntu
14.04. I've reached the point where all the doctest's pass, but before I
go further I've got some questions and concerns:

Mark's work touched more than just the ttvdb code. It also touched other
python bindings, especially database related code. I don't know what
other tools depend on those bindings and should be tested. I'm familiar
with the mytharchive plugin's mythburn.py and can test it, but what else?

For recordings, I rely only on metadata from Schedules Direct and have
disabled metadata look ups in my default recording template and in any
recording rules that predated recording templates. I do depend on
ttvdb.py for my video library and that's where I expect to focus most of
my unit testing. Will that be sufficient, or do I need to enable
metadata look ups for some, and hopefully not all of my recording rules?

Can upgrading the python package chardet from major version 2 to 3 cause
other python tools on my system to break? See background below.

Is it safe to assume that minor version upgrades to other python
packages won't cause problems?

The first time I run doctest's, when there's no ~/.mythtv/cache/tvdb_api
directory. I get a number of warnings:
1 SNIMissingWarning and 13 InsecurePlatformWarning's
Both types of warnings point to this link:
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Should I take the time to do as it recommends, install the pyOpenSSL
package and make the recommended code changes? Or, given that 0.27 isn't
officially supported, ignore the warnings if things work as is?

Should I add the contents of this email to ticket #13084, or should I
create a separate ticket just for 0.27, and maybe add cross references
between the two tickets?

If anyone is interested in my work, so far, it's here:
https://github.com/faginbagin/mythtv/tree/0.27-ttvdb-api-v2

Background on the python packages:
I installed python-pip from the ubuntu 14.04 package archive and used
pip to install or upgrade the required python packages.

To eliminate "No module named builtins"
$ sudo pip install future
To eliminate "No module named requests_cache"
$ sudo pip install requests_cache
To eliminate "AttributeError: 'module' object has no attribute
'disable_warnings'", upgrade urllib3 from 1.7.1 to 1.22
$ sudo pip install --upgrade urllib3
To eliminate "TypeError: request() got an unexpected keyword argument
'json'" upgraded requests from 2.2.1 to 2.18.4
$ sudo pip install --upgrade requests
Side effect of this last step was to also install or upgrade these packages:
Install certifi 2017.7.27.1
Install idna 2.6
Upgrade chardet from 2.0.1 to 3.0.4

The chardet package upgrade might be a concern, since it is a major
version upgrade from 2 to 3. I assume the other packages shouldn't cause
problems because they are either at the same major version as those
provided by the ubuntu 14.04 package archive or they aren't available
from there. Am I wrong?

Thanks in advance,
Helen
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
Forgot to mention: There were two areas where Mark's work did not apply
cleanly to the 0.27 code base. One was in
mythtv/bindings/python/MythTV/mythproto.py which was due to schema
changes since 0.27 (and which I think I will cover in my unit testing),
but also due to changes in how IPv6 is handled, in
mythtv/bindings/python/MythTV/database.py. My LAN and in fact my ISP are
still on IPv4, so I don't have the means to test this on an IPv6 network
(and not sure I'd want to try). Perhaps when I've done some real unit
testing, above and beyond the doctest tests, I can find someone who has
an IPv6 network and is still on 0.27? Or maybe those folks are more
likely to be running the latest LTS OS and mythtv version?

Thoughts?
Helen
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
On 08/28/2017 01:00 PM, faginbagin wrote:
> Forgot to mention: There were two areas where Mark's work did not
> apply cleanly to the 0.27 code base. One was in
> mythtv/bindings/python/MythTV/mythproto.py which was due to schema
> changes since 0.27 (and which I think I will cover in my unit
> testing), but also due to changes in how IPv6 is handled, in
> mythtv/bindings/python/MythTV/database.py. My LAN and in fact my ISP
> are still on IPv4, so I don't have the means to test this on an IPv6
> network (and not sure I'd want to try). Perhaps when I've done some
> real unit testing, above and beyond the doctest tests, I can find
> someone who has an IPv6 network and is still on 0.27? Or maybe those
> folks are more likely to be running the latest LTS OS and mythtv version?
>
> Thoughts?
> Helen
Hi Helen

In v0.27, without those changes, the database connection did not work on
IPV6 anyway. So as long as it still works on IPV4 you will not impact
anybody.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
On 8/28/2017 3:39 PM, Peter Bennett wrote:
>
>
> On 08/28/2017 01:00 PM, faginbagin wrote:
>> Forgot to mention: There were two areas where Mark's work did not
>> apply cleanly to the 0.27 code base. One was in
>> mythtv/bindings/python/MythTV/mythproto.py which was due to schema
>> changes since 0.27 (and which I think I will cover in my unit
>> testing), but also due to changes in how IPv6 is handled, in
>> mythtv/bindings/python/MythTV/database.py. My LAN and in fact my ISP
>> are still on IPv4, so I don't have the means to test this on an IPv6
>> network (and not sure I'd want to try). Perhaps when I've done some
>> real unit testing, above and beyond the doctest tests, I can find
>> someone who has an IPv6 network and is still on 0.27? Or maybe those
>> folks are more likely to be running the latest LTS OS and mythtv version?
>>
>> Thoughts?
>> Helen
> Hi Helen
>
> In v0.27, without those changes, the database connection did not work on
> IPV6 anyway. So as long as it still works on IPV4 you will not impact
> anybody.
>
> Peter

Thanks for the reassurance. Just to be sure we're on the same page. the
0.27 code in mythtv/bindings/python/MythTV/database.py, starting around
line 1359, with these lines:

def _getpreferredaddr(self, host):
ip6 = self.settings[host].BackendServerIP6
ip4 = self.settings[host].BackendServerIP

didn't work for IPv6? If they did work, it should be OK, because I
preserved them, not the later code that replaced 8 lines with just this one:

return self.settings[host].BackendServerAddr

Regards,
Helen
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
On 08/29/2017 12:06 AM, faginbagin wrote:
> Thanks for the reassurance. Just to be sure we're on the same page.
> the 0.27 code in mythtv/bindings/python/MythTV/database.py, starting
> around line 1359, with these lines:
>
> def _getpreferredaddr(self, host):
> ip6 = self.settings[host].BackendServerIP6
> ip4 = self.settings[host].BackendServerIP
>
> didn't work for IPv6? If they did work, it should be OK, because I
> preserved them, not the later code that replaced 8 lines with just
> this one:
>
> return self.settings[host].BackendServerAddr
>
> Regards,
> Helen
That one line will only work in v29. That BackendServerAddr was only
introduced in v29. I recommend you look at the v0.28 versions of the
bindings for backporting to 0.27. Did you try just cherry-picking the
commit into fixes/0.27?

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
On 8/29/2017 8:22 AM, Peter Bennett wrote:
>
>
> On 08/29/2017 12:06 AM, faginbagin wrote:
>> Thanks for the reassurance. Just to be sure we're on the same page.
>> the 0.27 code in mythtv/bindings/python/MythTV/database.py, starting
>> around line 1359, with these lines:
>>
>> def _getpreferredaddr(self, host):
>> ip6 = self.settings[host].BackendServerIP6
>> ip4 = self.settings[host].BackendServerIP
>>
>> didn't work for IPv6? If they did work, it should be OK, because I
>> preserved them, not the later code that replaced 8 lines with just
>> this one:
>>
>> return self.settings[host].BackendServerAddr
>>
>> Regards,
>> Helen
> That one line will only work in v29. That BackendServerAddr was only
> introduced in v29. I recommend you look at the v0.28 versions of the
> bindings for backporting to 0.27. Did you try just cherry-picking the
> commit into fixes/0.27?
>
> Peter

Sorry it's taken me a couple of days to reply, life ...

I think we are having a "failure to communicate". I understand that
BackendServerAddr does not exist in 0.27, but BackendServerIP and
BackendServerIP6 does exist and that the ability to connect to the
backend via IPv6 was supposedly supported since 0.25. I was asking for
clarification about database.py because you said:

"In v0.27, without those changes, the database connection did not work
on IPV6 anyway. So as long as it still works on IPV4 you will not impact
anybody."

However, the code in question in database.py was not for connecting to
the database (e.g. mysqld), it was for finding the right IP address to
connect to the master backend. So I was, in essence, asking; Are you
sure the code in question did not support connecting to the backend over
IPv6?

As for how I brought in Mark's changes, I did it before his pull request
was accepted. So I generated a pull request from his fork to mine,
brought it into a branch off of master on my fork, created a new branch
off of 0.27 and then cherry picked his commits into that branch. Feel
free to check my work.

Branch off of master containing Mark's unadulterated commits:
https://github.com/faginbagin/mythtv/tree/mspieth-ttvdb-api-v2

Branch off of 0.27 containing those commits after cherry-picking and
resolving conflicts.
https://github.com/faginbagin/mythtv/tree/0.27-ttvdb-api-v2

Regards,
Helen
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
On 08/31/2017 03:15 PM, faginbagin wrote:
> Sorry it's taken me a couple of days to reply, life ...
>
> I think we are having a "failure to communicate". I understand that
> BackendServerAddr does not exist in 0.27, but BackendServerIP and
> BackendServerIP6 does exist and that the ability to connect to the
> backend via IPv6 was supposedly supported since 0.25. I was asking for
> clarification about database.py because you said:
>
> "In v0.27, without those changes, the database connection did not work
> on IPV6 anyway. So as long as it still works on IPV4 you will not
> impact anybody."
>
> However, the code in question in database.py was not for connecting to
> the database (e.g. mysqld), it was for finding the right IP address to
> connect to the master backend. So I was, in essence, asking; Are you
> sure the code in question did not support connecting to the backend
> over IPv6?
>
Sorry for the confusion. There was recently a bug in the database
connection logic in one of the bindings, where it was neglecting to put
square brackets around the IPV6 address when connecting to the database
with IPV6, so that would not have worked in v0.27. I am not sure now
whether it was perl or python bindings that were affected, so my comment
may be incorrect.

As far as connecting to the backend over IPV6, I am not sure about that.
Although the code was supposed to support IPV6 there were some bugs that
were fixed in 29, so some things did not work correctly over IPV6.

> As for how I brought in Mark's changes, I did it before his pull
> request was accepted. So I generated a pull request from his fork to
> mine, brought it into a branch off of master on my fork, created a new
> branch off of 0.27 and then cherry picked his commits into that
> branch. Feel free to check my work.
>
OK sounds good. However I am not experienced with python so it would be
better if Mark could validate them.

> Branch off of master containing Mark's unadulterated commits:
> https://github.com/faginbagin/mythtv/tree/mspieth-ttvdb-api-v2
>
> Branch off of 0.27 containing those commits after cherry-picking and
> resolving conflicts.
> https://github.com/faginbagin/mythtv/tree/0.27-ttvdb-api-v2
>
> Regards,
> Helen

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
On 9/1/2017 8:38 AM, Peter Bennett wrote:
> OK sounds good. However I am not experienced with python so it would
> be better if Mark could validate them.
there is only one minor commit and its a good idea.
pull request it. Reviewed and passed.

mark

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
On 9/1/2017 12:44 PM, Mark Spieth wrote:
> On 9/1/2017 8:38 AM, Peter Bennett wrote:
>> OK sounds good. However I am not experienced with python so it would
>> be better if Mark could validate them.
> there is only one minor commit and its a good idea.
> pull request it. Reviewed and passed.
>
I meant to master. should have been clearer.
mark


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
On 8/31/2017 10:46 PM, Mark Spieth wrote:
> On 9/1/2017 12:44 PM, Mark Spieth wrote:
>> On 9/1/2017 8:38 AM, Peter Bennett wrote:
>>> OK sounds good. However I am not experienced with python so it would
>>> be better if Mark could validate them.
>> there is only one minor commit and its a good idea.
>> pull request it. Reviewed and passed.
>>
> I meant to master. should have been clearer.
> mark

I'm guessing you're referring to the os.mkdir -> os.makedirs commit:
https://github.com/faginbagin/mythtv/commit/772a767581c40d1f2065e9cde61f8e002805dae1
If so, I will create a pull request.

I just created my first pull request a few days ago, from Mark's fork to
mine. Is it OK to create a pull request against master for a commit that
is branched off of fixes/0.27?

Helen
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: ttvdb v2 support for 0.27 (and IPv6) [ In reply to ]
On 9/1/2017 12:29 AM, faginbagin wrote:
> On 8/31/2017 10:46 PM, Mark Spieth wrote:
>> On 9/1/2017 12:44 PM, Mark Spieth wrote:
>>> On 9/1/2017 8:38 AM, Peter Bennett wrote:
>>>> OK sounds good. However I am not experienced with python so it would
>>>> be better if Mark could validate them.
>>> there is only one minor commit and its a good idea.
>>> pull request it. Reviewed and passed.
>>>
>> I meant to master. should have been clearer.
>> mark
>
> I'm guessing you're referring to the os.mkdir -> os.makedirs commit:
> https://github.com/faginbagin/mythtv/commit/772a767581c40d1f2065e9cde61f8e002805dae1
>
> If so, I will create a pull request.
>
> I just created my first pull request a few days ago, from Mark's fork to
> mine. Is it OK to create a pull request against master for a commit that
> is branched off of fixes/0.27?
>
> Helen

FWIW, I just pushed another commit that I think makes the --debug
command line option a tad more useful. It no longer exits after printing
a bunch of debug data, and if an exception is thrown in the process, it
prints the stack trace and still keeps going. Perhaps it will help
diagnose what became of the genre and runtime info?

Regards,
Helen
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org