Mailing List Archive

1 2  View All
Re: FFmpeg plans [ In reply to ]
On Mon, 2018-04-30 at 11:52 -0400, Brian J. Murrell wrote:
> 2. subtree includes a copy of FFmpeg's repo in our repo.
>
> Correct. Although I don't think it's the whole "repo", where a repo is
> every object ever created in a project and so includes all branches,
> tags, etc. Rather I think git subtree (and subrepo) are just taking
> the content of a single branch and copying it to your subdir.

AIUI technically what it does (without --squash) is take the tree-sha
of the tree (the datastructure which is the git equivalent to a
directory) pointed to by an upstream commit (perhaps/probably one on a
branch) and grafts it in directly as an entry in a sub-tree (i.e. a
subdirectory) of the tree of the downstream repo.

Ian.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 04/30/2018 11:52 AM, Brian J. Murrell wrote:
>> I believe that using subtree or subrepo requires that we use the
>> master
>> branch of FFmpeg.
> I'm not convinced of that. At least with subrepo, you can choose which
> branch you clone and track into the subdir. If/when you want to change
> the upstream tracking branch, worst case scenario (assuming the tool
> doesn't have a way to change) is you do as above. Create your patch
> vs. upstream, blow the subdir away, recreate the subdir using
> {subtree|submodule} and then apply your patch, fixing any conflicts,
> etc.
This is essentially what we do at the moment. We delete the source code,
copy in the new source, and then work at re-applying all of our changes.
I am trying to improve on that process, which is rather time-consuming.

Peter
Re: FFmpeg plans [ In reply to ]
On Mon, 2018-04-30 at 12:37 -0400, Peter Bennett wrote:
>
> This is essentially what we do at the moment. We delete the source
> code,
> copy in the new source, and then work at re-applying all of our
> changes.
> I am trying to improve on that process, which is rather time-
> consuming.

Sure, but this is worst-case scenario. I have not tried to switch
branches with git subrepo (nor git subtree as I have not even used
that). Maybe it's easier than that. But it's also only something you
would ever have to do (again, in worst-case) when switching upstream
tracking (i.e. release) branches, so I think you are still better off.

At least you have git a workflow and sub{repo|tree} bookkeeping your
updating for you. Overall it should make keeping your ffmpeg subdir
up-to-date with upstream trivial, and the more frequently you refresh
(git subrepo pull) it the smaller the potential conflicts are.

If it were me, I'd set up a job daily or weekly just to refresh from
upstream to keep the potential for conflicts as entirely small as
possible.

Cheers,
b.
Re: FFmpeg plans [ In reply to ]
On 04/30/2018 01:52 PM, Brian J. Murrell wrote:
> Sure, but this is worst-case scenario. I have not tried to switch
> branches with git subrepo (nor git subtree as I have not even used
> that). Maybe it's easier than that. But it's also only something you
> would ever have to do (again, in worst-case) when switching upstream
> tracking (i.e. release) branches, so I think you are still better off.
Currently the only time we do any updating at all is when switching
upstream branches. So, no improvement.


> At least you have git a workflow and sub{repo|tree} bookkeeping your
> updating for you. Overall it should make keeping your ffmpeg subdir
> up-to-date with upstream trivial, and the more frequently you refresh
> (git subrepo pull) it the smaller the potential conflicts are.

> If it were me, I'd set up a job daily or weekly just to refresh from
> upstream to keep the potential for conflicts as entirely small as
> possible.
We need to decide a strategy. Currently we only refresh FFmpeg once a
year or so. I would not want to do any refresh shortly before a new
MythTV release because we would not want to possibly introduce
unexpected problems at the last minute.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
> On 30 Apr 2018, at 7:52 pm, Brian J. Murrell <brian@interlinx.bc.ca> wrote:
>
> On Mon, 2018-04-30 at 12:37 -0400, Peter Bennett wrote:
>>
>> This is essentially what we do at the moment. We delete the source
>> code,
>> copy in the new source, and then work at re-applying all of our
>> changes.
>> I am trying to improve on that process, which is rather time-
>> consuming.
>
> Sure, but this is worst-case scenario. I have not tried to switch
> branches with git subrepo (nor git subtree as I have not even used
> that). Maybe it's easier than that. But it's also only something you
> would ever have to do (again, in worst-case) when switching upstream
> tracking (i.e. release) branches, so I think you are still better off.


I’ve never seen a case where ffmpeg didn’t break API between version.

Updating automatically ffmpeg from a subrepo is just not going to work, there’s just no project that does what you’re stating. Every single project checkout a particular version of ffmpeg and stick to that version until the next stable release. It’s the one it’s tested against, and patched if necessary.

The only exception to this I can think of is mplayer, and ever attempted to checkout an older version of mplayer (which always pull the latest version of ffmpeg) ? Good luck. mplayer current checkout will work with ffmpeg of the day, never day to perform a bisect or attempt to lookup for a regression.

So not having to continually rebase our changes and particularly configure is one thing.

but all this talk about automated, git subtree vs subrepo and so forth, I’m sorry to say, clearly show you’ve really have never worked with ffmpeg before. Otherwise we wouldn’t have reached 24 messages in this thread.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Mon, 2018-04-30 at 20:08 +0200, Jean-Yves Avenard wrote:
>
> but all this talk about automated, git subtree vs subrepo and so
> forth, I’m sorry to say, clearly show you’ve really have never worked
> with ffmpeg before.

No I haven't. I never claimed to have and I find your condescending
and dismissing tone about offensive. But I work with lots of other
upstream projects in my real-life job even so I am not completely
talking out of my ass here.

I was providing input about git sub* tool experience. But if that
experience is not really appreciated and is going to be met with such
an attitude I will just keep it to myself, OK?

b.
Re: FFmpeg plans [ In reply to ]
There seems to be a little confusion concerning how git subtree works.
It doesn't do anything weird to the subdirectory it's applied to: the
only difference anyone would notice in the mythtv repo, if using git
subtree, is that the updates to the ffmpeg subdir would appear as merges
and some of the commits in the branches merged would mysteriously refer
to files within the ffmpeg subdir without the "external/FFmpeg" prefix.
Using git subtree would not make the mythtv repo rely on the ffmpeg
repo. It would just make them share some sha-1 hashes in a useful way.

Before git subtree existed, it was possible to have a local repo for one
project include remotes from an entirely different project. Without git
subtree it wasn't useful to do so because if you checked out on a branch
from one project all the files from the other project would disappear.
git subtree is just a useful way of using remotes from different
projects in the case that one project represents the files of a
subdirectory of another project.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
FFmpeg plans [ In reply to ]
In my own copy of MythTV repository, I tried using a subtree to include
FFmpeg. I hit up against a few problems -

I had wanted to try importing a particular commit from FFmpeg, but I was
unable to do that. I could only import the latest version of a branch.
Also I could see no way of cherry-picking FFmpeg commits into our copy.

git subtree add -P mythtv/external/FFmpeg
https://github.com/FFmpeg/FFmpeg.git 7e3a070 --squash
fatal: Couldn't find remote ref 7e3a070

I tried adding it as a remote, with the same result

git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git
git fetch ffmpeg
git subtree add -P mythtv/external/FFmpeg ffmpeg 7e3a070 --squash
fatal: Couldn't find remote ref 7e3a070

If you replace the commit id with a branch name it works

git subtree add -P mythtv/external/FFmpeg
https://github.com/FFmpeg/FFmpeg.git release/4.0 --squash
worked

I do not want to be forever constrained to only updating to the latest
commit of a branch in FFmpeg.

I have now created my own copy of FFmpeg repository and applied our
changes to that. Perhaps this is a better solution. We could maintain an
ffmpeg repository in MythTV and either copy from it as needed after
merging from the main FFmpeg, or else use a subtree based on our own
FFmpeg. We could create a mythtv-master branch in the private FFmpeg for
the subtree, and merge or cherry-pick to that as needed. This way we
would a better workfolw and accurate tracking of who contributed changes
in future. Currently the tracking of contributions to MythTV's copy of
FFmpeg is inaccurate, incorrectly showing most things having been
contributed by the various people who have done the FFmpeg resync.

We could also make a submodule instead of a subtree from our private
FFmpeg, but that means that people who access MythTV source would have
to run an explicit command to pull in the submodule data and to pull in
updates from it when needed.

On the current merge:
I am using FFmpeg master as of yesterday. The recent fixes to mediacodec
are not in the release branch. After applying the MythTV modifications
(5680 lines of patches) there were 12 files with conflicts. Most were
easy to fix, but FFmpeg has redesigned the code for registering demuxers
so I had to make some changes to the code for registering
ff_mythtv_mpegts_demuxer and ff_mythtv_mpegtsraw_demuxer.

It now has compile errors in the mythtv-mpegts demuxer code caused by
undeclared functions and macros, presumably deprecated things removed
from FFmpeg. I will have to fix these. The mythtv-mpegts demuxer code
also uses a lot of deprecated functions.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Fri, May 11, 2018 at 01:51:26PM -0400, Peter Bennett wrote:
> In my own copy of MythTV repository, I tried using a subtree to include
> FFmpeg. I hit up against a few problems -
>
> I had wanted to try importing a particular commit from FFmpeg, but I was
> unable to do that. I could only import the latest version of a branch. Also
> I could see no way of cherry-picking FFmpeg commits into our copy.
>
> git subtree add -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git 7e3a070 --squash
> fatal: Couldn't find remote ref 7e3a070
>
> I tried adding it as a remote, with the same result
>
> git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git
> git fetch ffmpeg
> git subtree add -P mythtv/external/FFmpeg ffmpeg 7e3a070 --squash
> fatal: Couldn't find remote ref 7e3a070
>
> If you replace the commit id with a branch name it works
>
> git subtree add -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git release/4.0 --squash
> worked
>
> I do not want to be forever constrained to only updating to the latest
> commit of a branch in FFmpeg.

I have one idea, but it's not without its own share of warts. That is
to treat the ffmpeg-master branch (and possibly others) like tracking
branches. It could go something like this.

# Remove the existing external/FFmpeg from the mythtv repo.
git rm -r external/FFmpeg

# Create an empty ffmpeg-master branch in the mythtv repo.
git co ce7a5f62 (the epoch commit with an empty repo>
git branch ffmpeg-master
git co ffmpeg-master

# Pull in ffmpeg master branch and move it to the right place.
git pull --allow-unrelated-histories https://github.com/FFmpeg/FFmpeg.git master
mkdir external/FFmpeg
git mv <list of files> external/FFmpeg

# Merge the ffmpeg-master branch into mythtv master.
git co master
git merge ffmpeg-master

One wart is that anytime ffmpeg adds new files in their root, we would
have to notice them when we pull and git mv them to external/FFmpeg.

Another wart is we would have to do the same with any other ffmpeg
branches that we might want to cherry-pick from.

> I have now created my own copy of FFmpeg repository and applied our changes
> to that. Perhaps this is a better solution. We could maintain an ffmpeg
> repository in MythTV and either copy from it as needed after merging from
> the main FFmpeg, or else use a subtree based on our own FFmpeg. We could
> create a mythtv-master branch in the private FFmpeg for the subtree, and
> merge or cherry-pick to that as needed. This way we would a better workfolw
> and accurate tracking of who contributed changes in future. Currently the
> tracking of contributions to MythTV's copy of FFmpeg is inaccurate,
> incorrectly showing most things having been contributed by the various
> people who have done the FFmpeg resync.
>
> We could also make a submodule instead of a subtree from our private FFmpeg,
> but that means that people who access MythTV source would have to run an
> explicit command to pull in the submodule data and to pull in updates from
> it when needed.
>
> On the current merge:
> I am using FFmpeg master as of yesterday. The recent fixes to mediacodec are
> not in the release branch. After applying the MythTV modifications (5680
> lines of patches) there were 12 files with conflicts. Most were easy to fix,
> but FFmpeg has redesigned the code for registering demuxers so I had to make
> some changes to the code for registering ff_mythtv_mpegts_demuxer and
> ff_mythtv_mpegtsraw_demuxer.
>
> It now has compile errors in the mythtv-mpegts demuxer code caused by
> undeclared functions and macros, presumably deprecated things removed from
> FFmpeg. I will have to fix these. The mythtv-mpegts demuxer code also uses a
> lot of deprecated functions.

FFmpeg really is a moving target, isn't it! :( Is it worth trying
again to get some of our changes accepted upstream?

David
--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Fri, 2018-05-11 at 13:51 -0400, Peter Bennett wrote:
> In my own copy of MythTV repository, I tried using a subtree to include
> FFmpeg. I hit up against a few problems -
>
> I had wanted to try importing a particular commit from FFmpeg, but I was
> unable to do that. I could only import the latest version of a branch.
> Also I could see no way of cherry-picking FFmpeg commits into our copy.
>
> git subtree add -P mythtv/external/FFmpeg
> https://github.com/FFmpeg/FFmpeg.git 7e3a070 --squash
> fatal: Couldn't find remote ref 7e3a070
>
> I tried adding it as a remote, with the same result
>
> git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git
> git fetch ffmpeg
> git subtree add -P mythtv/external/FFmpeg ffmpeg 7e3a070 --squash
> fatal: Couldn't find remote ref 7e3a070

There are two forms of `git subtree add`:
> git subtree add -P <prefix> <commit>
> git subtree add -P <prefix> <repository> <ref>

You've tried the second one but I think you need the first, in a slight
modification to your second attempt with a remote:

ijc@dagon:mythtv-subtree-test$ git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git
ijc@dagon:mythtv-subtree-test$ git fetch -n ffmpeg
warning: no common commits
[...]
ijc@dagon:mythtv-subtree-test$ git subtree add -P mythtv/external/FFmpeg-test 7e3a070 --squash
Added dir 'mythtv/external/FFmpeg-test'
ijc@dagon:mythtv-subtree-test$ ls mythtv/external/FFmpeg-test
Changelog COPYING.GPLv2 CREDITS INSTALL.md libavformat/ libswresample/ Makefile tests/
compat/ COPYING.GPLv3 doc/ libavcodec/ libavresample/ libswscale/ presets/ tools/
configure* COPYING.LGPLv2.1 ffbuild/ libavdevice/ libavutil/ LICENSE.md README.md
CONTRIBUTING.md COPYING.LGPLv3 fftools/ libavfilter/ libpostproc/ MAINTAINERS RELEASE

(I fudged the prefix because it already exists, but I guess you have
already done "git rm -r" to get rid of the old copy in your tree, I
used -n so I wouldn't get all the ffmpeg tags in my local repo). You
could avoid the remote by just doing:

git fetch https://github.com/FFmpeg/FFmpeg.git <branch|commit>

instead of the `git remote add foo` + `git fetch -n foo`.

I've not looked at how this differs from the subtree merge command:

> git subtree merge -P <prefix> <commit>

but I guess this is the one you want going forward after the initial
add.

For cherry-picking there is a process given in https://stackoverflow.co
m/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree
(second half of first answer) which looks pretty plausible to me, but
it does rely on not using `--squash` for the regular merges (although
possibly it could be adapted).

TBH I'd recommend not using squash anyway, it'll give a more accurate
picture of the history which is useful for people doing archaeology
which crosses into the ffmpeg tree (it shows the actual upstream
commits and authors instead of the subtree-merger) and I think git does
a better job of merging etc if it has more granular history to look at
(e.g. I think it can spot when the same changes appear in two commits
in different branches, perhaps due to cherry-picking, which helps merge
do the right thing more often).

Ian.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 05/12/2018 07:30 AM, Ian Campbell wrote:
> On Fri, 2018-05-11 at 13:51 -0400, Peter Bennett wrote:
>> In my own copy of MythTV repository, I tried using a subtree to include
>> FFmpeg. I hit up against a few problems -
>>
>> I had wanted to try importing a particular commit from FFmpeg, but I was
>> unable to do that. I could only import the latest version of a branch.
>> Also I could see no way of cherry-picking FFmpeg commits into our copy.
>>
>> git subtree add -P mythtv/external/FFmpeg
>> https://github.com/FFmpeg/FFmpeg.git 7e3a070 --squash
>> fatal: Couldn't find remote ref 7e3a070
>>
>> I tried adding it as a remote, with the same result
>>
>> git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git
>> git fetch ffmpeg
>> git subtree add -P mythtv/external/FFmpeg ffmpeg 7e3a070 --squash
>> fatal: Couldn't find remote ref 7e3a070
> There are two forms of `git subtree add`:
>> git subtree add -P <prefix> <commit>
>> git subtree add -P <prefix> <repository> <ref>
> You've tried the second one but I think you need the first, in a slight
> modification to your second attempt with a remote:
>
> ijc@dagon:mythtv-subtree-test$ git remote add ffmpeg https://github.com/FFmpeg/FFmpeg.git
> ijc@dagon:mythtv-subtree-test$ git fetch -n ffmpeg
> warning: no common commits
> [...]
> ijc@dagon:mythtv-subtree-test$ git subtree add -P mythtv/external/FFmpeg-test 7e3a070 --squash
> Added dir 'mythtv/external/FFmpeg-test'
> ijc@dagon:mythtv-subtree-test$ ls mythtv/external/FFmpeg-test
> Changelog COPYING.GPLv2 CREDITS INSTALL.md libavformat/ libswresample/ Makefile tests/
> compat/ COPYING.GPLv3 doc/ libavcodec/ libavresample/ libswscale/ presets/ tools/
> configure* COPYING.LGPLv2.1 ffbuild/ libavdevice/ libavutil/ LICENSE.md README.md
> CONTRIBUTING.md COPYING.LGPLv3 fftools/ libavfilter/ libpostproc/ MAINTAINERS RELEASE
>
> (I fudged the prefix because it already exists, but I guess you have
> already done "git rm -r" to get rid of the old copy in your tree, I
> used -n so I wouldn't get all the ffmpeg tags in my local repo). You
> could avoid the remote by just doing:
>
> git fetch https://github.com/FFmpeg/FFmpeg.git <branch|commit>
>
> instead of the `git remote add foo` + `git fetch -n foo`.
>
> I've not looked at how this differs from the subtree merge command:
>
>> git subtree merge -P <prefix> <commit>
> but I guess this is the one you want going forward after the initial
> add.
Thank you for the insight. This gives a better way of adding a subtree.
>
> For cherry-picking there is a process given in https://stackoverflow.co
> m/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree
> (second half of first answer) which looks pretty plausible to me, but
> it does rely on not using `--squash` for the regular merges (although
> possibly it could be adapted).
It is a bit messy - 5 step process.
> TBH I'd recommend not using squash anyway, it'll give a more accurate
> picture of the history which is useful for people doing archaeology
> which crosses into the ffmpeg tree (it shows the actual upstream
> commits and authors instead of the subtree-merger) and I think git does
> a better job of merging etc if it has more granular history to look at
> (e.g. I think it can spot when the same changes appear in two commits
> in different branches, perhaps due to cherry-picking, which helps merge
> do the right thing more often).
The thing I was concerned about with that is that there are so many
commits in FFmpeg compared with MythTV, that they would overwhelm the
history. There are 91000 commits in the master branch of FFmpeg. I
feared that the day I pulled in the subtree without squash, it may
become very difficult to find anything useful pertaining to MythTV in
the log. Perhaps that would only affect certain views of the log.

I am currently trying an approach of having a private copy of the FFmpeg
repository. Then I can use the simpler option at the bottom of the
stackoverflow  article, that says "Another more simple option, if you
have access to the original subtree repository, is to make the cherry
pick there in a branch and then just git subtree pull that specific
branch.".

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 05/11/2018 05:13 PM, David Engel wrote:
> I have one idea, but it's not without its own share of warts. That is
> to treat the ffmpeg-master branch (and possibly others) like tracking
> branches. It could go something like this.
>
> # Remove the existing external/FFmpeg from the mythtv repo.
> git rm -r external/FFmpeg
>
> # Create an empty ffmpeg-master branch in the mythtv repo.
> git co ce7a5f62 (the epoch commit with an empty repo>
> git branch ffmpeg-master
> git co ffmpeg-master
>
> # Pull in ffmpeg master branch and move it to the right place.
> git pull --allow-unrelated-historieshttps://github.com/FFmpeg/FFmpeg.git master
> mkdir external/FFmpeg
> git mv <list of files> external/FFmpeg
>
> # Merge the ffmpeg-master branch into mythtv master.
> git co master
> git merge ffmpeg-master
>
> One wart is that anytime ffmpeg adds new files in their root, we would
> have to notice them when we pull and git mv them to external/FFmpeg.
>
> Another wart is we would have to do the same with any other ffmpeg
> branches that we might want to cherry-pick from.
I originally thought of this option before I found subtree. It is still
rather complicated. I don't know how it would handle files that are
moved from one ffmpeg directory to another (that was the case with a
bunch of files between 3.2 and 3.4).

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Sat, 2018-05-12 at 12:29 -0400, Peter Bennett wrote:
> > For cherry-picking there is a process given in https://stackoverflow.co
> > m/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree
> > (second half of first answer) which looks pretty plausible to me, but
> > it does rely on not using `--squash` for the regular merges (although
> > possibly it could be adapted).
> It is a bit messy - 5 step process.

Yeah, it might be possible to script a bunch of it in a nice way.
Perhaps it is worth dropping a line to the git devs with a wishlist
request?

> > TBH I'd recommend not using squash anyway, it'll give a more accurate
> > picture of the history which is useful for people doing archaeology
> > which crosses into the ffmpeg tree (it shows the actual upstream
> > commits and authors instead of the subtree-merger) and I think git does
> > a better job of merging etc if it has more granular history to look at
> > (e.g. I think it can spot when the same changes appear in two commits
> > in different branches, perhaps due to cherry-picking, which helps merge
> > do the right thing more often).
> The thing I was concerned about with that is that there are so many
> commits in FFmpeg compared with MythTV, that they would overwhelm the
> history. There are 91000 commits in the master branch of FFmpeg. I
> feared that the day I pulled in the subtree without squash, it may
> become very difficult to find anything useful pertaining to MythTV in
> the log. Perhaps that would only affect certain views of the log.

`git log` orders things by commit date, so if the 91,000 ffmpeg commits
were largely historical then it wouldn't be so bad. But I suspect that
the ongoing flow of patches into ffmpeg probably outstrips mythtv by a
fair bit, in which case your concerns would be true by default.

You can use `git log --not ?branch|commit|etc?` to say "don't mention
things which are in here (i.e. `git log --not ffmpeg/master`), but
that's a bit tedious to remember to do.

> I am currently trying an approach of having a private copy of the FFmpeg
> repository. Then I can use the simpler option at the bottom of the
> stackoverflow article, that says "Another more simple option, if you
> have access to the original subtree repository, is to make the cherry
> pick there in a branch and then just git subtree pull that specific
> branch.".

That's effectively what the 5 step cherry-pick process above is doing,
it's just doing it all in the one repo using branches and bare
checkouts. It's certainly less things to keep straight in ones head to
do it in a separate repo though.

A separate repo presumably makes it easier to keep track of where
you've gotten up to (i.e. most recently merged) and lets you have
branches for fixes/30 etc and still keep cherry picking onto those
independently of master.

Ian.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 05/13/2018 03:32 AM, Ian Campbell wrote:
> On Sat, 2018-05-12 at 12:29 -0400, Peter Bennett wrote:
>>> For cherry-picking there is a process given in https://stackoverflow.co
>>> m/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree
>>> (second half of first answer) which looks pretty plausible to me, but
>>> it does rely on not using `--squash` for the regular merges (although
>>> possibly it could be adapted).
>> It is a bit messy - 5 step process.
> Yeah, it might be possible to script a bunch of it in a nice way.
> Perhaps it is worth dropping a line to the git devs with a wishlist
> request?
>
>>> TBH I'd recommend not using squash anyway, it'll give a more accurate
>>> picture of the history which is useful for people doing archaeology
>>> which crosses into the ffmpeg tree (it shows the actual upstream
>>> commits and authors instead of the subtree-merger) and I think git does
>>> a better job of merging etc if it has more granular history to look at
>>> (e.g. I think it can spot when the same changes appear in two commits
>>> in different branches, perhaps due to cherry-picking, which helps merge
>>> do the right thing more often).
>> The thing I was concerned about with that is that there are so many
>> commits in FFmpeg compared with MythTV, that they would overwhelm the
>> history. There are 91000 commits in the master branch of FFmpeg. I
>> feared that the day I pulled in the subtree without squash, it may
>> become very difficult to find anything useful pertaining to MythTV in
>> the log. Perhaps that would only affect certain views of the log.
> `git log` orders things by commit date, so if the 91,000 ffmpeg commits
> were largely historical then it wouldn't be so bad. But I suspect that
> the ongoing flow of patches into ffmpeg probably outstrips mythtv by a
> fair bit, in which case your concerns would be true by default.
>
> You can use `git log --not «branch|commit|etc»` to say "don't mention
> things which are in here (i.e. `git log --not ffmpeg/master`), but
> that's a bit tedious to remember to do.
>
>> I am currently trying an approach of having a private copy of the FFmpeg
>> repository. Then I can use the simpler option at the bottom of the
>> stackoverflow article, that says "Another more simple option, if you
>> have access to the original subtree repository, is to make the cherry
>> pick there in a branch and then just git subtree pull that specific
>> branch.".
> That's effectively what the 5 step cherry-pick process above is doing,
> it's just doing it all in the one repo using branches and bare
> checkouts. It's certainly less things to keep straight in ones head to
> do it in a separate repo though.
>
> A separate repo presumably makes it easier to keep track of where
> you've gotten up to (i.e. most recently merged) and lets you have
> branches for fixes/30 etc and still keep cherry picking onto those
> independently of master.
>
> Ian.
>
In my private MythTV repository I added a subtree for FFmpeg (with
squash) from my private FFmpeg repository (with patches committed) and
got everything working. Then I pulled new ffmpeg commits into the
private ffmpeg repo and tried to pull the changes into the private
MythTV. That was a disaster. Every change, even 1 line changes, was
marked as a conflict. I think that is because with the squash it cannot
find a base for the merge. So that is a bust.

Now I have deleted the FFmpeg tree from my priivate MythTV and copied
all of the files from the private FFmpeg and committed that. This is
similar to our traditional approach, with the difference that I am
copying files from my own FFmpeg repo (with MythTV patches included)
instead of copying them from the main FFmpeg and re-applying patches.

I think maybe the best is to set up a FFmpeg repository under MythTV and
do it that way - all FFmpeg changes that we make should be first applied
to the FFmpeg repository and then copy the source over to MythTV.

I have the new FFmpeg working and very lightly tested with the master
MythTV. Playback works on VAAPI, VDPAU, XVIDEO. LiveTV works and LiveTV
subtitles work. That is all I have tested.

If you want to take a look it is in github under bennettpeter - MythTV
and FFmpeg repositories, master branches.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Mon, May 14, 2018 at 10:44:19AM -0400, Peter Bennett wrote:
>
>
> On 05/13/2018 03:32 AM, Ian Campbell wrote:
> > On Sat, 2018-05-12 at 12:29 -0400, Peter Bennett wrote:
> > > > For cherry-picking there is a process given in https://stackoverflow.co
> > > > m/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree
> > > > (second half of first answer) which looks pretty plausible to me, but
> > > > it does rely on not using `--squash` for the regular merges (although
> > > > possibly it could be adapted).
> > > It is a bit messy - 5 step process.
> > Yeah, it might be possible to script a bunch of it in a nice way.
> > Perhaps it is worth dropping a line to the git devs with a wishlist
> > request?
> >
> > > > TBH I'd recommend not using squash anyway, it'll give a more accurate
> > > > picture of the history which is useful for people doing archaeology
> > > > which crosses into the ffmpeg tree (it shows the actual upstream
> > > > commits and authors instead of the subtree-merger) and I think git does
> > > > a better job of merging etc if it has more granular history to look at
> > > > (e.g. I think it can spot when the same changes appear in two commits
> > > > in different branches, perhaps due to cherry-picking, which helps merge
> > > > do the right thing more often).
> > > The thing I was concerned about with that is that there are so many
> > > commits in FFmpeg compared with MythTV, that they would overwhelm the
> > > history. There are 91000 commits in the master branch of FFmpeg. I
> > > feared that the day I pulled in the subtree without squash, it may
> > > become very difficult to find anything useful pertaining to MythTV in
> > > the log. Perhaps that would only affect certain views of the log.
> > `git log` orders things by commit date, so if the 91,000 ffmpeg commits
> > were largely historical then it wouldn't be so bad. But I suspect that
> > the ongoing flow of patches into ffmpeg probably outstrips mythtv by a
> > fair bit, in which case your concerns would be true by default.
> >
> > You can use `git log --not ?branch|commit|etc?` to say "don't mention
> > things which are in here (i.e. `git log --not ffmpeg/master`), but
> > that's a bit tedious to remember to do.
> >
> > > I am currently trying an approach of having a private copy of the FFmpeg
> > > repository. Then I can use the simpler option at the bottom of the
> > > stackoverflow article, that says "Another more simple option, if you
> > > have access to the original subtree repository, is to make the cherry
> > > pick there in a branch and then just git subtree pull that specific
> > > branch.".
> > That's effectively what the 5 step cherry-pick process above is doing,
> > it's just doing it all in the one repo using branches and bare
> > checkouts. It's certainly less things to keep straight in ones head to
> > do it in a separate repo though.
> >
> > A separate repo presumably makes it easier to keep track of where
> > you've gotten up to (i.e. most recently merged) and lets you have
> > branches for fixes/30 etc and still keep cherry picking onto those
> > independently of master.
> >
> > Ian.
> >
> In my private MythTV repository I added a subtree for FFmpeg (with squash)
> from my private FFmpeg repository (with patches committed) and got
> everything working. Then I pulled new ffmpeg commits into the private ffmpeg
> repo and tried to pull the changes into the private MythTV. That was a
> disaster. Every change, even 1 line changes, was marked as a conflict. I
> think that is because with the squash it cannot find a base for the merge.
> So that is a bust.

Did you try without squash? I know whatever we do will be tedious,
but what you describe below with our own, second repository sound more
so.

David

> Now I have deleted the FFmpeg tree from my priivate MythTV and copied all of
> the files from the private FFmpeg and committed that. This is similar to our
> traditional approach, with the difference that I am copying files from my
> own FFmpeg repo (with MythTV patches included) instead of copying them from
> the main FFmpeg and re-applying patches.
>
> I think maybe the best is to set up a FFmpeg repository under MythTV and do
> it that way - all FFmpeg changes that we make should be first applied to the
> FFmpeg repository and then copy the source over to MythTV.
>
> I have the new FFmpeg working and very lightly tested with the master
> MythTV. Playback works on VAAPI, VDPAU, XVIDEO. LiveTV works and LiveTV
> subtitles work. That is all I have tested.
>
> If you want to take a look it is in github under bennettpeter - MythTV and
> FFmpeg repositories, master branches.
>
> Peter

--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 05/14/2018 02:11 PM, David Engel wrote:
> Did you try without squash? I know whatever we do will be tedious,
> but what you describe below with our own, second repository sound more
> so.
>
> David
>
I did not. I will create another branch in my repository and try that.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 05/14/2018 04:04 PM, Peter Bennett wrote:
>
>
> On 05/14/2018 02:11 PM, David Engel wrote:
>> Did you try without squash?  I know whatever we do will be tedious,
>> but what you describe below with our own, second repository sound more
>> so.
>>
>> David
>>
> I did not. I will create another branch in my repository and try that.
>
> Peter

Tested Without squash and without private repo:

It works fine I am able to merge updates from ffmpeg wiothout conflicts
as well as cherry pick a commit without problem, using the procedure in
https://stackoverflow.com/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree

The log is full of ffmpeg entries. We will need some filtering on "git
log" to make it useful. I have not looked into that. Look at
https://github.com/bennettpeter/mythtv and select the "ffmpeg-subtree"
branch. Notice it has 125000 commits, as opposed to the master branch,
which has 34000.

I think this is a good solution, as long as other developers are not too
unhappy about the log situation.

Below are the steps I used

git remote rename ffmpeg ffmpeg-private
git remote add ffmpeg git@github.com:FFmpeg/FFmpeg.git
git checkout 2403427
git branch ffmpeg-subtree
git checkout ffmpeg-subtree
git clean -xfd
git rm -r mythtv/external/FFmpeg
rm -r mythtv/external/FFmpeg/
git fetch ffmpeg
git subtree add -P mythtv/external/FFmpeg 6dc7963
git apply --directory=mythtv/external/FFmpeg
../patch/20180510_1945_FFmpeg_MythTV_Customization_fixed.patch
git apply --directory=mythtv/external/FFmpeg
../patch/20180511_1537_FFmpeg_compile_errors.patch
git apply ../patch/20180512_1728_additional_ffmpeg_fixes.patch
git apply ../patch/20180512_1158_fix_mythtv_compile_errors.patch --reject
git apply ../patch/20180512_1534_compile_error_and_fix_playback.patch
--reject
git push --set-upstream bennettpeter ffmpeg-subtree
#sync some ffmpeg updates
git fetch ffmpeg
git subtree merge -P mythtv/external/FFmpeg 7db022e
#successful
#cherry-pick a commit from ffmpeg
git checkout 7db022e
git cherry-pick -e -x 42a03e7
git log
#copy the commit hash -> 8a57b2fced942bc1c17ff26cfe8fa36193d95fbb
git checkout ffmpeg-subtree
# you can use HEAD@{1} instead of the hash if you did not do anything in
between
git subtree merge -P mythtv/external/FFmpeg
8a57b2fced942bc1c17ff26cfe8fa36193d95fbb

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Mon, May 14, 2018 at 07:44:11PM -0400, Peter Bennett wrote:
>
>
> On 05/14/2018 04:04 PM, Peter Bennett wrote:
> >
> >
> > On 05/14/2018 02:11 PM, David Engel wrote:
> > > Did you try without squash?? I know whatever we do will be tedious,
> > > but what you describe below with our own, second repository sound more
> > > so.
> > >
> > > David
> > >
> > I did not. I will create another branch in my repository and try that.
> >
> > Peter
>
> Tested Without squash and without private repo:
>
> It works fine I am able to merge updates from ffmpeg wiothout conflicts as
> well as cherry pick a commit without problem, using the procedure in https://stackoverflow.com/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree
>
> The log is full of ffmpeg entries. We will need some filtering on "git log"
> to make it useful. I have not looked into that. Look at
> https://github.com/bennettpeter/mythtv and select the "ffmpeg-subtree"
> branch. Notice it has 125000 commits, as opposed to the master branch, which
> has 34000.
>
> I think this is a good solution, as long as other developers are not too
> unhappy about the log situation.

The log bloat is unfortunate, but I think this solution is better than
the other alternatives so far. Thanks for testing all of this.

David

> Below are the steps I used
>
> git remote rename ffmpeg ffmpeg-private
> git remote add ffmpeg git@github.com:FFmpeg/FFmpeg.git
> git checkout 2403427
> git branch ffmpeg-subtree
> git checkout ffmpeg-subtree
> git clean -xfd
> git rm -r mythtv/external/FFmpeg
> rm -r mythtv/external/FFmpeg/
> git fetch ffmpeg
> git subtree add -P mythtv/external/FFmpeg 6dc7963
> git apply --directory=mythtv/external/FFmpeg
> ../patch/20180510_1945_FFmpeg_MythTV_Customization_fixed.patch
> git apply --directory=mythtv/external/FFmpeg
> ../patch/20180511_1537_FFmpeg_compile_errors.patch
> git apply ../patch/20180512_1728_additional_ffmpeg_fixes.patch
> git apply ../patch/20180512_1158_fix_mythtv_compile_errors.patch --reject
> git apply ../patch/20180512_1534_compile_error_and_fix_playback.patch
> --reject
> git push --set-upstream bennettpeter ffmpeg-subtree
> #sync some ffmpeg updates
> git fetch ffmpeg
> git subtree merge -P mythtv/external/FFmpeg 7db022e
> #successful
> #cherry-pick a commit from ffmpeg
> git checkout 7db022e
> git cherry-pick -e -x 42a03e7
> git log
> #copy the commit hash -> 8a57b2fced942bc1c17ff26cfe8fa36193d95fbb
> git checkout ffmpeg-subtree
> # you can use HEAD@{1} instead of the hash if you did not do anything in
> between
> git subtree merge -P mythtv/external/FFmpeg
> 8a57b2fced942bc1c17ff26cfe8fa36193d95fbb
>
> Peter

--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 05/15/2018 11:51 AM, David Engel wrote:
> The log bloat is unfortunate, but I think this solution is better than
> the other alternatives so far. Thanks for testing all of this.
>
> David
I did some testing of git log and I found that git seems to have a
bug/restriction which actually helps.

If you cd to the base directory of the repo and run "git log -- mythtv"
you only see the mythtv and mythtv subdirectory changes that were made
in this repository but not any of the ffmpeg subtree changes that were
imported via the subtree.  So we see the ffmpeg customizations that we
applied here but none of those imported via the subtree. This is exactly
what I would want to see.

I cannot find a way of seeing a log of subtree imported commits using a
path. Using mythtv/external/FFmpeg finds nothing, probably because when
they were committed the path was just libavformat/ and not
mythtv/external/FFmpeg/libavformat. However searching on path
libavformat also shows nothing. If you want to see those you either have
to use the full "git log", search on a revision range, or look in the
ffmpeg repository. This seems to be a bug but works in our favor.

Peter


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On Tue, 2018-05-15 at 12:14 -0400, Peter Bennett wrote:
> If you cd to the base directory of the repo and run "git log -- mythtv"
> you only see the mythtv and mythtv subdirectory changes that were made
> in this repository but not any of the ffmpeg subtree changes that were
> imported via the subtree. So we see the ffmpeg customizations that we
> applied here but none of those imported via the subtree. This is exactly
> what I would want to see.

This is because git log isn't aware of subtrees, so as it walks over
commits which are actually from ffmpeg it isn't seeing
mythtv/external/ffmpeg/foo/bar.c (which would match your mythtv
argument) it sees just foo/bar.c (which doesn't match the argument).

So it relies a bit on ffmpeg not having a top-level directory called
"mythtv" but that seems like a safe bet.

It's not future proof against the git devs make "git log" aware of
subtrees -- but you would hope/imagine that in doing so they would add
useful options like --no-subtrees at the same time.

>
> I cannot find a way of seeing a log of subtree imported commits using a
> path. Using mythtv/external/FFmpeg finds nothing, probably because when
> they were committed the path was just libavformat/ and not
> mythtv/external/FFmpeg/libavformat.

Correct (I should have read ahead before writing the above! I'll leave
it now I've typed it)

Ian.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 15/05/18 16:51, David Engel wrote:

> On Mon, May 14, 2018 at 07:44:11PM -0400, Peter Bennett wrote:
>>
>> On 05/14/2018 04:04 PM, Peter Bennett wrote:
>>>
>>> On 05/14/2018 02:11 PM, David Engel wrote:
>>>> Did you try without squash?  I know whatever we do will be tedious,
>>>> but what you describe below with our own, second repository sound more
>>>> so.
>>>>
>>>> David
>>>>
>>> I did not. I will create another branch in my repository and try that.
>>>
>>> Peter
>> Tested Without squash and without private repo:
>>
>> It works fine I am able to merge updates from ffmpeg wiothout conflicts as
>> well as cherry pick a commit without problem, using the procedure in https://stackoverflow.com/questions/12978260/how-do-i-go-to-a-specific-commit-using-git-subtree
>>
>> The log is full of ffmpeg entries. We will need some filtering on "git log"
>> to make it useful. I have not looked into that. Look at
>> https://github.com/bennettpeter/mythtv and select the "ffmpeg-subtree"
>> branch. Notice it has 125000 commits, as opposed to the master branch, which
>> has 34000.
>>
>> I think this is a good solution, as long as other developers are not too
>> unhappy about the log situation.
> The log bloat is unfortunate, but I think this solution is better than
> the other alternatives so far. Thanks for testing all of this.
>
> David
>

Do we really want to see all the ffmpeg commits in our git logs? I hate
the idea :(

Paul H.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg plans [ In reply to ]
On 05/15/2018 01:05 PM, Paul Harrison wrote:
> Do we really want to see all the ffmpeg commits in our git logs? I
> hate the idea :(
>
> Paul H.
Paul
Did you see my later email on this? By using "git log -- mythtv" you can
exclude all the ffmpeg commits.
Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

1 2  View All