Mailing List Archive

FFmpeg plans
To David and other devs

I have separated the MythTV configure from the FFmpeg configure so that
they no longer need to be merged when bringing in new FFmpeg versions.
see ticket https://code.mythtv.org/trac/ticket/13259 . I will commit
this to master shortly, once I see that no compile errors caused by this
come from the build slaves and my testing is all successful.

I would like to propose that we incorporate FFmpeg master branch into
MythTV instead of periodically merging release branches. Also I would
like to merge the FFmpeg git into ours, so that bringing in FFmpeg
updates becomes a git pull operation. I am not a git expert but I think
this should work:

1. Delete mythtv/external/FFmpeg in mythtv
From git root:
git rm -r mythtv/external/FFmpeg

2. Create a subtree of FFmpeg master
From git root:
git subtree add -P mythtv/external/FFmpeg
https://github.com/FFmpeg/FFmpeg.git master [--squash]

3. Apply our FFmpeg customizations and commit them.

4. Later, when we want to pull subsequent FFmpeg updates, run this
From git root
git subtree pull -P mythtv/external/FFmpeg
https://github.com/FFmpeg/FFmpeg.git master [--squash]

I will test this in the ffmpeg-resync branch first to make sure it all
works.

Notes: Many articles recommend the --squash flag to suppress importing
the history of the other project. Does anybody have opinions about this?

See
https://www.atlassian.com/blog/git/alternatives-to-git-submodule-git-subtree
and man git-subtree.

It is recommended but not necessary to separate commits to FFmpeg from
commits to the rest of the system. This is mainly in case you need to
submit updates back to the main FFmpeg repository, which we would not
likely be doing.

Please let me have your opinions or comments on this. Also, let me know
if I have missed something in git that may prevent this from working
correctly.

Peter


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
What about the mpeg-ts demuxer?

Get Firefox on Android
________________________________
From: Peter Bennett
Sent: Saturday, 28 April 2018 22:08
To: David Engel; Development of MythTV
Subject: [mythtv] FFmpeg plans

To David and other devs

I have separated the MythTV configure from the FFmpeg configure so that
they no longer need to be merged when bringing in new FFmpeg versions.
see ticket https://code.mythtv.org/trac/ticket/13259 . I will commit
this to master shortly, once I see that no compile errors caused by this
come from the build slaves and my testing is all successful.

I would like to propose that we incorporate FFmpeg master branch into
MythTV instead of periodically merging release branches. Also I would
like to merge the FFmpeg git into ours, so that bringing in FFmpeg
updates becomes a git pull operation. I am not a git expert but I think
this should work:

1. Delete mythtv/external/FFmpeg in mythtv
From git root:
git rm -r mythtv/external/FFmpeg

2. Create a subtree of FFmpeg master
From git root:
git subtree add -P mythtv/external/FFmpeg
https://github.com/FFmpeg/FFmpeg.git master [--squash]

3. Apply our FFmpeg customizations and commit them.

4. Later, when we want to pull subsequent FFmpeg updates, run this
From git root
git subtree pull -P mythtv/external/FFmpeg
https://github.com/FFmpeg/FFmpeg.git master [--squash]

I will test this in the ffmpeg-resync branch first to make sure it all
works.

Notes: Many articles recommend the --squash flag to suppress importing
the history of the other project. Does anybody have opinions about this?

See
https://www.atlassian.com/blog/git/alternatives-to-git-submodule-git-subtree
and man git-subtree.

It is recommended but not necessary to separate commits to FFmpeg from
commits to the rest of the system. This is mainly in case you need to
submit updates back to the main FFmpeg repository, which we would not
likely be doing.

Please let me have your opinions or comments on this. Also, let me know
if I have missed something in git that may prevent this from working
correctly.

Peter


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 04/28/2018 05:16 PM, Jean-Yves Avenard wrote:
> What about the mpeg-ts demuxer?
>
That is part of step 3. (Apply our FFmpeg customizations and commit
them). It includes adding files that were added to our copy of FFmpeg.
This is really the FFmpeg merge as I did before (without the configure
merge) but hopefully now for the last time because going forward we can
just pull from FFmpeg.

Do you see a problem with this?

Peter



> Get Firefox on Android
> ------------------------------------------------------------------------
> *From:* Peter Bennett
> *Sent:* Saturday, 28 April 2018 22:08
> *To:* David Engel; Development of MythTV
> *Subject:* [mythtv] FFmpeg plans
>
> To David and other devs
>
> I have separated the MythTV configure from the FFmpeg configure so that
> they no longer need to be merged when bringing in new FFmpeg versions.
> see ticket https://code.mythtv.org/trac/ticket/13259 . I will commit
> this to master shortly, once I see that no compile errors caused by this
> come from the build slaves and my testing is all successful.
>
> I would like to propose that we incorporate FFmpeg master branch into
> MythTV instead of periodically merging release branches. Also I would
> like to merge the FFmpeg git into ours, so that bringing in FFmpeg
> updates becomes a git pull operation. I am not a git expert but I think
> this should work:
>
> 1. Delete mythtv/external/FFmpeg in mythtv
> From git root:
> git rm -r mythtv/external/FFmpeg
>
> 2. Create a subtree of FFmpeg master
> From git root:
> git subtree add -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git master [--squash]
>
> 3. Apply our FFmpeg customizations and commit them.
>
> 4. Later, when we want to pull subsequent FFmpeg updates, run this
> From git root
> git subtree pull -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git master [--squash]
>
> I will test this in the ffmpeg-resync branch first to make sure it all
> works.
>
> Notes: Many articles recommend the --squash flag to suppress importing
> the history of the other project. Does anybody have opinions about this?
>
> See
> https://www.atlassian.com/blog/git/alternatives-to-git-submodule-git-subtree
>
> and man git-subtree.
>
> It is recommended but not necessary to separate commits to FFmpeg from
> commits to the rest of the system. This is mainly in case you need to
> submit updates back to the main FFmpeg repository, which we would not
> likely be doing.
>
> Please let me have your opinions or comments on this. Also, let me know
> if I have missed something in git that may prevent this from working
> correctly.
>
> Peter
>
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>
>
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
________________________________
From: Peter Bennett
Sent: Saturday, 28 April 2018 23:35
To: mythtv-dev@mythtv.org
Subject: Re: [mythtv] FFmpeg plans



On 04/28/2018 05:16 PM, Jean-Yves Avenard wrote:
>
> What about the mpeg-ts demuxer?
>
That is part of step 3. (Apply our FFmpeg customizations and commit them). It includes adding files that were added to our copy of FFmpeg. This is really the FFmpeg merge as I did before (without the configure merge) but hopefully now for the last time because going forward we can just pull from FFmpeg.

Do you see a problem with this?

Peter

No problem at all, just from experience every resync required manual tweaks there. Don't see how that could be automated.


BTW, 4.0 is out :)
>
> Get Firefox on Android
> ________________________________
> From: Peter Bennett
> Sent: Saturday, 28 April 2018 22:08
> To: David Engel; Development of MythTV
> Subject: [mythtv] FFmpeg plans
>
> To David and other devs
>
> I have separated the MythTV configure from the FFmpeg configure so that
> they no longer need to be merged when bringing in new FFmpeg versions.
> see ticket https://code.mythtv.org/trac/ticket/13259 . I will commit
> this to master shortly, once I see that no compile errors caused by this
> come from the build slaves and my testing is all successful.
>
> I would like to propose that we incorporate FFmpeg master branch into
> MythTV instead of periodically merging release branches. Also I would
> like to merge the FFmpeg git into ours, so that bringing in FFmpeg
> updates becomes a git pull operation. I am not a git expert but I think
> this should work:
>
> 1. Delete mythtv/external/FFmpeg in mythtv
> From git root:
> git rm -r mythtv/external/FFmpeg
>
> 2. Create a subtree of FFmpeg master
> From git root:
> git subtree add -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git master [--squash]
>
> 3. Apply our FFmpeg customizations and commit them.
>
> 4. Later, when we want to pull subsequent FFmpeg updates, run this
> From git root
> git subtree pull -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git master [--squash]
>
> I will test this in the ffmpeg-resync branch first to make sure it all
> works.
>
> Notes: Many articles recommend the --squash flag to suppress importing
> the history of the other project. Does anybody have opinions about this?
>
> See
> https://www.atlassian.com/blog/git/alternatives-to-git-submodule-git-subtree
> and man git-subtree.
>
> It is recommended but not necessary to separate commits to FFmpeg from
> commits to the rest of the system. This is mainly in case you need to
> submit updates back to the main FFmpeg repository, which we would not
> likely be doing.
>
> Please let me have your opinions or comments on this. Also, let me know
> if I have missed something in git that may prevent this from working
> correctly.
>
> Peter
>
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>
>
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 04/28/2018 06:07 PM, Jean-Yves Avenard wrote:
> *From:* Peter Bennett
> *Sent:* Saturday, 28 April 2018 23:35
> *To:* mythtv-dev@mythtv.org
> *Subject:* Re: [mythtv] FFmpeg plans
>
>
>
> On 04/28/2018 05:16 PM, Jean-Yves Avenard wrote:
>> What about the mpeg-ts demuxer?
>>
> That is part of step 3. (Apply our FFmpeg customizations and commit
> them). It includes adding files that were added to our copy of FFmpeg.
> This is really the FFmpeg merge as I did before (without the configure
> merge) but hopefully now for the last time because going forward we
> can just pull from FFmpeg.
>
> Do you see a problem with this?
>
> Peter
>
> No problem at all, just from experience every resync required manual
> tweaks there. Don't see how that could be automated.
I plan to try it out by bringing in a commit of FFmpeg from a month or
two ago and do the tweaks. Then try a pull of later commit from FFmpeg
to see how well it works. There could be conflicts, as there could with
any pull, that have to be fixed.

Peter
Re: FFmpeg plans [ In reply to ]
> On Apr 28, 2018, at 4:08 PM, Peter Bennett <pb.mythtv@gmail.com> wrote:
>
> ...
> 2. Create a subtree of FFmpeg master
> From git root:
> git subtree add -P mythtv/external/FFmpeg https://github.com/FFmpeg/FFmpeg.git master [--squash]
>
For MacPorts, I rely on GitHub producing a tarball from a commit hash. MacPorts mirrors that tarball and it ensures that our build is reproducible.[1] I could be wrong but I don’t think a subtree would be included in the tarball. If you are going to test this, please check this out. Other packagers may also be affected.

OTOH, if the MythTV project wanted to do more frequent releases and produce tarballs, that would allelivate any concern.

Craig

[1] https://trac.macports.org/wiki/ReproducibleBuilds

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 04/28/2018 08:17 PM, Craig Treleaven wrote:
>> On Apr 28, 2018, at 4:08 PM, Peter Bennett <pb.mythtv@gmail.com> wrote:
>>
>> ...
>> 2. Create a subtree of FFmpeg master
>> From git root:
>> git subtree add -P mythtv/external/FFmpeg https://github.com/FFmpeg/FFmpeg.git master [--squash]
>>
> For MacPorts, I rely on GitHub producing a tarball from a commit hash. MacPorts mirrors that tarball and it ensures that our build is reproducible.[1] I could be wrong but I don’t think a subtree would be included in the tarball. If you are going to test this, please check this out. Other packagers may also be affected.
>
> OTOH, if the MythTV project wanted to do more frequent releases and produce tarballs, that would allelivate any concern.
>
> Craig
>
> [1] https://trac.macports.org/wiki/ReproducibleBuilds
>
> _______________________________________________
>
Hi Craig

Thanks for the info. I believe it will include the subtree in the
tarball. I will make sure to test it to make sure it is included.
I see the way to get a tarball is using a URL like this
https://github.com/MythTV/mythtv/archive/devel/ffmpeg-resync.tar.gz .

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Sat, 2018-04-28 at 16:08 -0400, Peter Bennett wrote:
>
> 2. Create a subtree of FFmpeg master
> From git root:
> git subtree add -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git master [--squash]

I use https://github.com/ingydotnet/git-subrepo in another project with
good success. If you do use it, be sure to use the release/0.4.0
branch.

> 3. Apply our FFmpeg customizations and commit them.

Why are these not upstreamed? Are they not generally useful?

Cheers,
b.
Re: FFmpeg plans [ In reply to ]
On Sun, 2018-04-29 at 11:21 -0400, Peter Bennett wrote:
>
> Thanks for the info. I believe it will include the subtree in the
> tarball.

Well, to be pedantic, git sub[tree|repo] pulls a *copy* of the git
project it is subbing into the git project it is subbing into (if that
made sense). git submodule is the command that puts a "pointer" to a
sha1 of the project it is submodule'ing into the project it is subbing
into.

Think of them like "cp -r" and "ln -s" respectively.

As long as whatever makes the tarball descends the directory created by
git sub[tree|repo], the subdir should be in the tarball, and since this
is what happens currently, there should be no change there.

Cheers,
b.
Re: FFmpeg plans [ In reply to ]
On 04/29/2018 11:55 AM, Brian J. Murrell wrote:
> On Sat, 2018-04-28 at 16:08 -0400, Peter Bennett wrote:
>> 2. Create a subtree of FFmpeg master
>> From git root:
>> git subtree add -P mythtv/external/FFmpeg
>> https://github.com/FFmpeg/FFmpeg.git master [--squash]
> I use https://github.com/ingydotnet/git-subrepo in another project with
> good success. If you do use it, be sure to use the release/0.4.0
> branch.
I looked at it and I am not too clear what advantages it offers us over
subtree. One advantage I see is it keeps commits to the main code
separate from those to the subrepo, thereby making it easier to send
back upstream changes. However we do not have the ability to push
changes back to FFmpeg. I think the developers may be reluctant to use a
third-party component like this unless there are very clear advantages.
>> 3. Apply our FFmpeg customizations and commit them.
> Why are these not upstreamed? Are they not generally useful?
These have been in place for a long time and developers who originally
created them have submitted them to FFmpeg a long time ago but they were
never accepted.

Peter

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Sun, 2018-04-29 at 12:47 -0400, Peter Bennett wrote:
>
> These have been in place for a long time and developers who
> originally
> created them have submitted them to FFmpeg a long time ago but they
> were
> never accepted.

If it was a long (enough) time ago, it might be worthwhile trying to
submit them again. With enough time passing, their general usefullness
might be more clear to upstream and/or the upstream developers may have
changed enough such that they might have a more liberal view of the
changes now, etc.

Cheers,
b.
Re: FFmpeg plans [ In reply to ]
They won't be.

Our changes adds callback to detect things like resolution change, something ffmpeg moved away from a long time ago.

The biggest thing would be to update *their* mpeg-ts demuxer to support our extra features. And that's no trivial task as the two mpeg-ts demuxers have diverged significantly over the years, they almost have nothing in common these days.

Get Firefox on Android
________________________________
From: "Brian J. Murrell" <brian@interlinx.bc.ca>
Sent: Sunday, 29 April 2018 19:47
To: mythtv-dev@mythtv.org
Subject: Re: [mythtv] FFmpeg plans

On Sun, 2018-04-29 at 12:47 -0400, Peter Bennett wrote:
>
> These have been in place for a long time and developers who
> originally
> created them have submitted them to FFmpeg a long time ago but they
> were
> never accepted.

If it was a long (enough) time ago, it might be worthwhile trying to
submit them again.  With enough time passing, their general usefullness
might be more clear to upstream and/or the upstream developers may have
changed enough such that they might have a more liberal view of the
changes now, etc.

Cheers,
b.

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Sat, Apr 28, 2018 at 04:08:19PM -0400, Peter Bennett wrote:
> To David and other devs
>
> I have separated the MythTV configure from the FFmpeg configure so that they
> no longer need to be merged when bringing in new FFmpeg versions. see ticket
> https://code.mythtv.org/trac/ticket/13259 . I will commit this to master
> shortly, once I see that no compile errors caused by this come from the
> build slaves and my testing is all successful.
>
> I would like to propose that we incorporate FFmpeg master branch into MythTV
> instead of periodically merging release branches. Also I would like to merge
> the FFmpeg git into ours, so that bringing in FFmpeg updates becomes a git
> pull operation. I am not a git expert but I think this should work:
>
> 1. Delete mythtv/external/FFmpeg in mythtv
> From git root:
> git rm -r mythtv/external/FFmpeg
>
> 2. Create a subtree of FFmpeg master
> From git root:
> git subtree add -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git master [--squash]
>
> 3. Apply our FFmpeg customizations and commit them.
>
> 4. Later, when we want to pull subsequent FFmpeg updates, run this
> From git root
> git subtree pull -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git master [--squash]
>
> I will test this in the ffmpeg-resync branch first to make sure it all
> works.
>
> Notes: Many articles recommend the --squash flag to suppress importing the
> history of the other project. Does anybody have opinions about this?
>
> See
> https://www.atlassian.com/blog/git/alternatives-to-git-submodule-git-subtree
> and man git-subtree.
>
> It is recommended but not necessary to separate commits to FFmpeg from
> commits to the rest of the system. This is mainly in case you need to submit
> updates back to the main FFmpeg repository, which we would not likely be
> doing.
>
> Please let me have your opinions or comments on this. Also, let me know if I
> have missed something in git that may prevent this from working correctly.

I'm not familiar enough with the differences between using subtree and
submodule to comment about that yet.

Likewise, I'm not sure that FFmpeg's master branch is the best choice.
How do you propose to handle post-release fixes from FFmpeg? Looking
at FFmpeg's release branch for 3.4, it saw a lot of fixes spanning 4
months after the initial releases. Perhaps Aman could shed some light
on how FFmpeg handles their release and master branches and suggest
the best one to track.

David
--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 04/29/2018 04:08 PM, David Engel wrote:
> Likewise, I'm not sure that FFmpeg's master branch is the best choice.
> How do you propose to handle post-release fixes from FFmpeg? Looking
> at FFmpeg's release branch for 3.4, it saw a lot of fixes spanning 4
> months after the initial releases. Perhaps Aman could shed some light
> on how FFmpeg handles their release and master branches and suggest
> the best one to track.
Hi David

I looked at https://ffmpeg.org/download.html#get-sources
They recommend in two places there that people compiling from source
should use the development (master) branch. They also state that fixes
are first applied to master and then some of them are cherry-picked into
release branches.

If we use a release branch, I do not know if the subtree approach will
work. When we move from one release branch to the next release branch, I
don't know if pulling from a different branch would work.

Also, regarding post-release fixes, AFAIK we currently do not take them.
I have not merged in any post-release fixes for 3.4. If we use master we
can do another pull any time to get fixes.

Peter

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Sun, 2018-04-29 at 12:11 -0400, Brian J. Murrell wrote:
> On Sun, 2018-04-29 at 11:21 -0400, Peter Bennett wrote:
> >
> > Thanks for the info. I believe it will include the subtree in the
> > tarball.
>
> Well, to be pedantic, git sub[tree|repo] pulls a *copy* of the git
> project it is subbing into the git project it is subbing into (if
> that
> made sense). git submodule is the command that puts a "pointer" to a
> sha1 of the project it is submodule'ing into the project it is
> subbing
> into.

If I recall correctly, a git submodule requires write permission on the
other repository (e.g. FFmpeg) in order to commit and changes/tags/etc.
That's probably a non-starter.

I'd love to hear more about your experience with subrepos. I've been
meaning to update many of the other external components, and haven't
done so yet because I haven't had time to fully research submodules vs.
subtrees (and now vs. subrepos). Whichever we decide to use, we should
use it consistently across all the external modules.

David



> Think of them like "cp -r" and "ln -s" respectively.
>
> As long as whatever makes the tarball descends the directory created
> by
> git sub[tree|repo], the subdir should be in the tarball, and since
> this
> is what happens currently, there should be no change there.
>
> Cheers,
> b.
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 28/04/18 23:58, Peter Bennett wrote:
>
>
> On 04/28/2018 06:07 PM, Jean-Yves Avenard wrote:
>> *From:* Peter Bennett
>> *Sent:* Saturday, 28 April 2018 23:35
>> *To:* mythtv-dev@mythtv.org
>> *Subject:* Re: [mythtv] FFmpeg plans
>>
>>
>>
>> On 04/28/2018 05:16 PM, Jean-Yves Avenard wrote:
>>> What about the mpeg-ts demuxer?
>>>
>> That is part of step 3. (Apply our FFmpeg customizations and commit
>> them). It includes adding files that were added to our copy of FFmpeg.
>> This is really the FFmpeg merge as I did before (without the configure
>> merge) but hopefully now for the last time because going forward we
>> can just pull from FFmpeg.
>>
>> Do you see a problem with this?
>>
>> Peter
>>
>> No problem at all, just from experience every resync required manual
>> tweaks there. Don't see how that could be automated.
> I plan to try it out by bringing in a commit of FFmpeg from a month or
> two ago and do the tweaks. Then try a pull of later commit from FFmpeg
> to see how well it works. There could be conflicts, as there could with
> any pull, that have to be fixed.
>

Can you apply your own patches to an external tree in git?
That would be a requirement for this to work surely?


Regards
Stuart


> Peter
>
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 30/04/18 09:15, Stuart Auchterlonie wrote:

> On 28/04/18 23:58, Peter Bennett wrote:
>>
>>
>> On 04/28/2018 06:07 PM, Jean-Yves Avenard wrote:
>>> *From:* Peter Bennett
>>> *Sent:* Saturday, 28 April 2018 23:35
>>> *To:* mythtv-dev@mythtv.org
>>> *Subject:* Re: [mythtv] FFmpeg plans
>>>
>>>
>>>
>>> On 04/28/2018 05:16 PM, Jean-Yves Avenard wrote:
>>>> What about the mpeg-ts demuxer?
>>>>
>>> That is part of step 3. (Apply our FFmpeg customizations and commit
>>> them). It includes adding files that were added to our copy of
>>> FFmpeg. This is really the FFmpeg merge as I did before (without the
>>> configure merge) but hopefully now for the last time because going
>>> forward we can just pull from FFmpeg.
>>>
>>> Do you see a problem with this?
>>>
>>> Peter
>>>
>>> No problem at all, just from experience every resync required manual
>>> tweaks there. Don't see how that could be automated.
>> I plan to try it out by bringing in a commit of FFmpeg from a month
>> or two ago and do the tweaks. Then try a pull of later commit from
>> FFmpeg to see how well it works. There could be conflicts, as there
>> could with any pull, that have to be fixed.
>>
>
> Can you apply your own patches to an external tree in git?
> That would be a requirement for this to work surely?
>
>
> Regards
> Stuart
>

I think that could be a problem since you need write access to the
upstream project to commit them. What I do is fork the external project
and then add my own fork as the submodule. It's easy to keep the fork in
sync and you only have to do it when it's convenient. You don't want to
pull a big update from upstream just before you do a release for example.

I'm in no way a git expert so there may well be more elegant ways of
doing it and  usually when git is involved I suspect if you ask six
developers the best way to do it they will tell you six different answers :)

Paul H.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Mon, 2018-04-30 at 09:15 +0100, Stuart Auchterlonie wrote:
>
> Can you apply your own patches to an external tree in git?

As a sub{tree|repo} it's not external -- that's a git submodule. A git
sub{tree|repo} is a full copy of the external source in your own repo.
You are just using git sub{tree|repo} to manage (i.e. forking,
updating, etc.) it like you would any git repo you might not have
commit privileges with.

So just like you can fork any git repo, apply your own patches and push
those to your own fork, then periodically pull in changes since your
fork from upstream, commit those, rinse, repeat, you can do the same
with a git sub{tree|repo}.

> That would be a requirement for this to work surely?

Of course.

Cheers,
b.
Re: FFmpeg plans [ In reply to ]
On Mon, 2018-04-30 at 10:04 +0100, Paul Harrison wrote:
>
> I think that could be a problem since you need write access to the
> upstream project to commit them.

No. The sub{tree|repo} is a fork, not a pointer, to the upstream
module.

> What I do is fork the external project
> and then add my own fork as the submodule.

That is exactly what git sub{tree|repo} is. It's really just tooling
around that workflow, keeping track of where you last synced from so
that you can pull changes since then, etc.

> It's easy to keep the fork in
> sync and you only have to do it when it's convenient.

Right. Exactly what git sub{tree|repo} does also. You decide when to
git sub{tree|repo} pull from upstream.

Cheers,
b.
Re: FFmpeg plans [ In reply to ]
On Mon, 2018-04-30 at 00:14 -0400, David Hampton wrote:
>
> If I recall correctly, a git submodule requires write permission on
> the
> other repository (e.g. FFmpeg) in order to commit and
> changes/tags/etc.

Yes, that's because with git submodule, you only put a reference to a
sha1 of the repo you are setting up as a submodule. Think of this like
creating a symlink to a directory somewhere else.

With a sub{tree|repo} you actually create an entire copy of the source
of the module you are subbing in your own repo. Think of this like
creating a recursive copy of that directory somewhere else.

> That's probably a non-starter.

Of course. Particularly where the remote repository maintainers are
unwilling to merge your changes.

> I'd love to hear more about your experience with subrepos.

It's been fine. Using git subrepo (URL previously posted in this
thread), I did find I needed to switch to the 0.4.0 release branch
pretty quickly as the 0.3.0 branch was giving me problems. But I would
not entirely rule out that that could have just been learning curve
with the limitations of git subrepo (probably applies to git subtree
also) and rebasing commits for/from the subrepo.

> I've been
> meaning to update many of the other external components, and haven't
> done so yet because I haven't had time to fully research submodules
> vs.
> subtrees (and now vs. subrepos). Whichever we decide to use, we
> should
> use it consistently across all the external modules.

Agreed, well mostly. I do still have a like for submodules, where they
fit, particularly if I am the maintainer of both the parent and child
modules.

One use case for using git sub* is not just to fork and keep changes in
an external repo but to keep code that is common to many projects in
one git module. For this use-case, I tend to like git submodule better
as it truly does just mean maintaining the common code in one place and
then adding git submodule (references, not copies) to the projects that
want to use the module. Because changes *have* to be made in the
module directly, it instills a workflow that requires changes be made
to the module itself and doesn't allow consumers of the module to keep
local changes (i.e. forget to push them back to the original module).

Where you want to keep local changes and don't have commit rights to
the module you are consuming, git sub{tree|repo} are the better tool.
The one thing that does grate me the wrong with with git sub{tree|repo}
is the duplication. I'm efficiency-OCD and hate waste so duplicating
an entire source tree feels wasteful to me.

As for which of git sub{tree|repo} to use, they both achieve the same
goal. git subrepo is supposed to just be a better git subtree. From
the README there:


This command is an improvement from git-submodule and git-subtree;
two other git commands with similar goals, but various problems.

I'll let you decide if subrepo is better than subtree since it's pretty
subjective.

Cheers,
b.
Re: FFmpeg plans [ In reply to ]
Thank you Brian Murrell for the helpful information.

This is my take on the whole submodule/subtree/subrepo discussion:

1. submodule would incorporate a reference to the FFmpeg repository in
MythTV. This would not work because we have many customizations of
FFmpeg. It could work if we created first our own FFmpeg repository. I
do not like that idea.

2. subtree includes a copy of FFmpeg's repo in our repo. We can
customize the code as we need. We can also pull newer versions from
FFmpeg and it will merge in the changes. Sometimes you may need to
resolve conflicts if they have changed code that we have customized.

3. subrepo is a third party tool
(https://github.com/ingydotnet/git-subrepo) that seems to be almost the
same as subtree. One difference is that if you make a subsequent commit
that includes changes to MythTV as well as to FFmpeg, subrepo separates
them into two commits. It also keeps history of the subtrepo separate
from the main history. My feeling about subrepo is that I would rather
stick with built-in features of git (subtree) than rely on a third party
addon that may not be supported in future.

I believe that using subtree or subrepo requires that we use the master
branch of FFmpeg. I do not see a problem with using FFmpeg master
together with MythTV master. The FFmpeg web page recommends using master
if you are compiling FFmpeg yourself.

I will test subtree in my own repository first to make sure it behaves
as expected.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Sun, Apr 29, 2018 at 04:44:53PM -0400, Peter Bennett wrote:
>
>
> On 04/29/2018 04:08 PM, David Engel wrote:
> > Likewise, I'm not sure that FFmpeg's master branch is the best choice.
> > How do you propose to handle post-release fixes from FFmpeg? Looking
> > at FFmpeg's release branch for 3.4, it saw a lot of fixes spanning 4
> > months after the initial releases. Perhaps Aman could shed some light
> > on how FFmpeg handles their release and master branches and suggest
> > the best one to track.
> Hi David
>
> I looked at https://ffmpeg.org/download.html#get-sources
> They recommend in two places there that people compiling from source should
> use the development (master) branch. They also state that fixes are first
> applied to master and then some of them are cherry-picked into release
> branches.

That recommendation seems highly unusual to me. Why do releases at
all then? It sounds like thay handle release branches in the same,
pretty standard way we do. We try to keep our master branch stable
too, but we don't recommend others run it. I wonder why they do.

David

> If we use a release branch, I do not know if the subtree approach will work.
> When we move from one release branch to the next release branch, I don't
> know if pulling from a different branch would work.
>
> Also, regarding post-release fixes, AFAIK we currently do not take them. I
> have not merged in any post-release fixes for 3.4. If we use master we can
> do another pull any time to get fixes.
>
> Peter
>

--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
Hi

> On 30 Apr 2018, at 5:45 pm, David Engel <david@istwok.net> wrote:
>
> That recommendation seems highly unusual to me. Why do releases at
> all then? It sounds like thay handle release branches in the same,
> pretty standard way we do. We try to keep our master branch stable
> too, but we don't recommend others run it. I wonder why they do.
>
> David

Because some developers never ever introduce regressions and always produce bug free code… doh !
Re: FFmpeg plans [ In reply to ]
On Mon, 2018-04-30 at 11:05 -0400, Peter Bennett wrote:
> Thank you Brian Murrell for the helpful information.

NP.

> 1. submodule would incorporate a reference to the FFmpeg repository
> in
> MythTV.

Correct.

> This would not work because we have many customizations of
> FFmpeg.

Correct.

> It could work if we created first our own FFmpeg repository.

Right. Which is just taking the same approach as sub{tree|repo} but
doing the work tracking yourself manually.

> I
> do not like that idea.

Indeed. There is nothing about it to like. :-)

> 2. subtree includes a copy of FFmpeg's repo in our repo.

Correct. Although I don't think it's the whole "repo", where a repo is
every object ever created in a project and so includes all branches,
tags, etc. Rather I think git subtree (and subrepo) are just taking
the content of a single branch and copying it to your subdir.

> We can
> customize the code as we need.

Correct.

> We can also pull newer versions from
> FFmpeg and it will merge in the changes.

Correct.

> Sometimes you may need to
> resolve conflicts if they have changed code that we have customized.

Correct.

> 3. subrepo is a third party tool
> (https://github.com/ingydotnet/git-subrepo) that seems to be almost
> the
> same as subtree.

Correct.

> One difference is that if you make a subsequent commit
> that includes changes to MythTV as well as to FFmpeg, subrepo
> separates
> them into two commits.

TBH, I'm not entirely sure about that. I generally use subrepo in the
submodule sense, where I never commit changes to the subrepo'd dir in a
consuming project but instead commit them upstream and then pull them
back down to the consumers.

For my use-case, in fact, I would actually be using submodule except
that that requires users of my module to know there is a submodule and
do the git submodule init/blah-blah and I don't want my users to know
they have to that and for CI systems to know they have to do it.

> My feeling about subrepo is that I would
> rather
> stick with built-in features of git (subtree) than rely on a third
> party
> addon that may not be supported in future.

That's fair enough. But given that both are just keeping a copy of the
external module in a subdir locally in your module, I don't think there
is anything stopping you from switching at some point in the future.
You might just have to do a bit of gluing of one method's metadata to
the other. Or even simpler, you just make a patch of your changes vs.
upstream, blow the subdir away and then use a different method (i.e.
converting from subrepo to subtree) to recreate the subdir and then
apply your patch and you are switched.

> I believe that using subtree or subrepo requires that we use the
> master
> branch of FFmpeg.

I'm not convinced of that. At least with subrepo, you can choose which
branch you clone and track into the subdir. If/when you want to change
the upstream tracking branch, worst case scenario (assuming the tool
doesn't have a way to change) is you do as above. Create your patch
vs. upstream, blow the subdir away, recreate the subdir using
{subtree|submodule} and then apply your patch, fixing any conflicts,
etc.

Cheers
b.
Re: FFmpeg plans [ In reply to ]
On 04/30/2018 11:45 AM, David Engel wrote:
> That recommendation seems highly unusual to me. Why do releases at
> all then? It sounds like thay handle release branches in the same,
> pretty standard way we do. We try to keep our master branch stable
> too, but we don't recommend others run it. I wonder why they do.
>
> David
We can treat their master like our master. Before we tag a release
branch make sure it has been stable with a particular pull of FFmpeg. In
the pre-release branch we can do testing and cherry-pick additional
FFmpeg commits or reverse bad FFmpeg commits as necessary, as we do with
MythTV commits, if needed to fix bugs.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

1 2  View All