Mailing List Archive

1 2  View All
Re: Decision on FFmpeg repository [ In reply to ]
On Fri, 2018-05-18 at 08:33 +0100, Ian Campbell wrote:
> On Fri, 2018-05-18 at 08:30 +0100, Ian Campbell wrote:
> > On Thu, 2018-05-17 at 15:05 -0400, Peter Bennett wrote:
> > > 3. A commit hash may no longer identify what source code was used
> for
> > > a build, because you cannot tell if they remembered to update the
> > > submodule. I suppose we need to add a submodule hash to the
> version
> > > string.
> >
> > IIRC the commit of the submodule is committed to the main repo (in
> > .gitmodules or some such) and `git status` et al will show the repo
> as
> > dirty if the submodule is either dirty or not at the named
> revision.
>
> It turns out IRinC(orrectly), in qemu.git/.gitmodules I see:
> [submodule "roms/vgabios"]
> path = roms/vgabios
> url = git://git.qemu-project.org/vgabios.git/
> [submodule "roms/seabios"]
> path = roms/seabios
> url = git://git.qemu-project.org/seabios.git/
>
> So it seems the precise commit is not stored there. The
> gitsubmodules(7) and gitmodules(5) manpages do not mention another
> field which might be used to store the commit either.
>
> I stand corrected.

Or maybe not ;-)

https://git.qemu.org/?p=qemu.git;a=commit;h=ad30c0b0d86957a0be41dcbcd23659c18a61065c

show an update of the seabio entry, as you can see it is modifying a
magic file at the path `roms/seabios` which is then shadowed by
the submodule. This file is checked in and tracked as part of the main
repo:
diff --git a/roms/seabios b/roms/seabios
index 33fbe13a3e..01a84bea2d 160000
--- a/roms/seabios
+++ b/roms/seabios
@@ -1 +1 @@
-Subproject commit 33fbe13a3e2a01e0ba1087a8feed801a0451db21
+Subproject commit 01a84bea2d28a19d2405c1ecac4bdef17683cc0c

I can't see the sub command which was used to update this, but I guess
is must be there somewhere...

Ian.

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Decision on FFmpeg repository [ In reply to ]
On 05/17/2018 09:11 PM, David Engel wrote:
> I wonder if --no-ff would help when merging ffmpeg's changes into our
> master branch. I know there are cases, like this maybe, where git
> puts a single, "merge" commit on the main branch. When that happens,
> I believe git requires extra log options to see all of the individual
> commits that were merged.
The git subtree commands do not have a --no-ff option. However if we do
use subtree, the relatively easy "git log -- mythtv" is probably a good
solution for seeing only the mythtv commits.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Decision on FFmpeg repository [ In reply to ]
On 05/17/2018 04:33 PM, John P Poet wrote:
>
> If David and Aman are willing / able to get our patches pushed to
> FFmpeg, then this <hopefully> would become a non-issue, since we could
> just add an external dependency on FFmpeg, instead of including it. 
> Again, that is probably not practical, though.

I sent Aman files of our patches a few days ago. He noticed a couple
that he suggested I submit to FFmpeg.

1. One header file now causes c++ compiles to fail.
Response from them - the header file we have been using is not public
and should not be used by us. Patch rejected.
2. Three external names misspelled in a .v file - resulting in a link error
Response from them - those functions are not part of the public api and
they are no longer used - they should be removed.

I believe that in the past, MythTV developers have noticed useful things
in the FFmpeg code, and called functions or used structures without
regard to whether they were the official API. Now those functions are
being changed or removed we are scrambling to fix the result. It may be
better to copy useful functions into our code rather than simply call
them in the ever changing FFmpeg code.

We should only be using the external API that is exposed by the header
files they deploy as part of FFmpeg build. We are using the header files
in the FFmpeg source.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Decision on FFmpeg repository [ In reply to ]
On 05/17/2018 04:01 PM, David Engel wrote:
> My only current aim for ffmpeg is MediaCodec. Ater that is done, I
> wouldn't mind going back to stable, ffmpeg releases. The git handling
> with subtrees would probably be more complicated, but I have to
> believe it can be done. In theory, it should boil down to a rebase.
Another option we could consider is using the ffmpeg 4.0 version and
cherry-pick the commits from ffmpeg master that apply to mediacodec. We
could do this with a subtree or a private ffmpeg repository and submodule.

I examined the FFmpeg repository carefully around the 4.0 release and it
seems they use the same principles we do - the release branch is taken
from the master branch shortly before release, and all commits in the
release branch are cherry-picks from the master branch.

To be able to easily accomplish this I think the best is a private
repository and a submodule to pull in the correct commit. The submodule
could be switched to a new branch easily when FFmpeg gets a new version.
To ensure that everybody gets the correct code when building, the "git
submodule update" could be in the configure script as suggested by
Jean-Yves. It will have to allow for building from a tarball, in that
case it will have to assume everything is up to date.

I think that trying to switch release branches when using a subtree
would be excessively complicated.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Decision on FFmpeg repository [ In reply to ]
On 05/18/2018 12:05 PM, Gary Buhrmaster wrote:
> On Fri, May 18, 2018 at 3:38 PM, Peter Bennett <pb.mythtv@gmail.com> wrote:
> ....
>> To be able to easily accomplish this I think the best is a private
>> repository and a submodule to pull in the correct commit. The submodule
>> could be switched to a new branch easily when FFmpeg gets a new version. To
>> ensure that everybody gets the correct code when building, the "git
>> submodule update" could be in the configure script as suggested by
>> Jean-Yves. It will have to allow for building from a tarball, in that case
>> it will have to assume everything is up to date.
> I have not really followed this discussion (it is one for the actual
> devs, and ffmpeg has always been sufficiently problematic that
> I never had enough smarts to come up with a great alternative),
> but I do believe it is critical that when one does a download
> from github (at any particular commitish value(*)) one gets a
> fully complete source file without some later magic required to
> make it buildable by pulling in something new/different every
> time a build is performed, as it is expected that one can create
> a rebuildable source definition and use it as is to reproduce
> exactly the same binary (I am sure that is worded poorly). If
> the configure script (potentially) pulls in new upstream every
> time, that breaks reproducibility. Or maybe I am not getting
> what is being proposed (back to the I have not really been
> following this discussion).
>
>
>
>
> (*) For example:
> https://github.com/MythTV/mythtv/archive/862e510f589e3fedb8f37b61ac7967a219b7d8f4.zip
> which is (as of the moment) master head.
These git features are new to me, but as I understand it - If you add a
submodule to a git repository, it makes one directory a copy of a
specific commit in another repository. Submodule update will always
fetch that specific commit. You can change it to another commit and you
have to commit that change. If somebody has changed the submodule setup
to access a different commit, somebody else who does a pull does not
automatically get the updated source for the submodule, they have to do
a git submodule update, this is why I would put that in the config. One
specific commit from MythTV should always result in the same source code
after the submodule update.

Peter
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Decision on FFmpeg repository [ In reply to ]
On Fri, May 18, 2018 at 11:20:57AM -0400, Peter Bennett wrote:
> On 05/17/2018 04:33 PM, John P Poet wrote:
> >
> > If David and Aman are willing / able to get our patches pushed to
> > FFmpeg, then this <hopefully> would become a non-issue, since we could
> > just add an external dependency on FFmpeg, instead of including it.?
> > Again, that is probably not practical, though.
>
> I sent Aman files of our patches a few days ago. He noticed a couple that he
> suggested I submit to FFmpeg.
>
> 1. One header file now causes c++ compiles to fail.
> Response from them - the header file we have been using is not public and
> should not be used by us. Patch rejected.

Disappointing, but understandable. What do we use that header for?
Is it for any of the other things that Aman has already implemented or
is implementing differently than us?

> 2. Three external names misspelled in a .v file - resulting in a link error
> Response from them - those functions are not part of the public api and they
> are no longer used - they should be removed.

Again, understandable. Do we use any of those functions?

> I believe that in the past, MythTV developers have noticed useful things in
> the FFmpeg code, and called functions or used structures without regard to
> whether they were the official API. Now those functions are being changed or
> removed we are scrambling to fix the result. It may be better to copy useful
> functions into our code rather than simply call them in the ever changing
> FFmpeg code.

Both ways have their problems. Copying functions into our code can
give a false sense of security. There aren't any conflicts with the
latest update -- yay. Oh no, several things broke silently in suble
ways.

> We should only be using the external API that is exposed by the header files
> they deploy as part of FFmpeg build. We are using the header files in the
> FFmpeg source.

I completely agree with this. That means trying hard to get the
missing functionality we need accepted upstream and being willing to
drop some of our extra functionality if we can't get it accepted.

David
--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Decision on FFmpeg repository [ In reply to ]
On Fri, May 18, 2018 at 11:38:26AM -0400, Peter Bennett wrote:
>
>
> On 05/17/2018 04:01 PM, David Engel wrote:
> > My only current aim for ffmpeg is MediaCodec. Ater that is done, I
> > wouldn't mind going back to stable, ffmpeg releases. The git handling
> > with subtrees would probably be more complicated, but I have to
> > believe it can be done. In theory, it should boil down to a rebase.
> Another option we could consider is using the ffmpeg 4.0 version and
> cherry-pick the commits from ffmpeg master that apply to mediacodec. We
> could do this with a subtree or a private ffmpeg repository and submodule.

That becomes increasingly harder the further the development branch
deviates from the release branch. Before the 4.0 release was made, I
briefly tried porting the MediaCodec changes to the 3.4.x release. It
didn't take long before the MediaCodec changes started relying on
other changes that weren't in the relase branch.

> I examined the FFmpeg repository carefully around the 4.0 release and it
> seems they use the same principles we do - the release branch is taken from
> the master branch shortly before release, and all commits in the release
> branch are cherry-picks from the master branch.
>
> To be able to easily accomplish this I think the best is a private
> repository and a submodule to pull in the correct commit. The submodule
> could be switched to a new branch easily when FFmpeg gets a new version. To
> ensure that everybody gets the correct code when building, the "git
> submodule update" could be in the configure script as suggested by
> Jean-Yves. It will have to allow for building from a tarball, in that case
> it will have to assume everything is up to date.
>
> I think that trying to switch release branches when using a subtree would be
> excessively complicated.

Has anyone considered consulting with the git maintainers or whomever
is responsible for the subtree support. Perhaps they have some ideas
on how to do what we need. Maybe they'd even consider enhancing the
subtree support if what we need can't currently be done.

David
--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Decision on FFmpeg repository [ In reply to ]
On Fri, May 18, 2018 at 5:58 PM, Peter Bennett <pb.mythtv@gmail.com> wrote:

> These git features are new to me, but as I understand it - If you add a
> submodule to a git repository, it makes one directory a copy of a specific
> commit in another repository. Submodule update will always fetch that
> specific commit. You can change it to another commit and you have to commit
> that change. If somebody has changed the submodule setup to access a
> different commit, somebody else who does a pull does not automatically get
> the updated source for the submodule, they have to do a git submodule
> update, this is why I would put that in the config.

Just so you understand, the github download (at a commit)
does *NOT* generate a git repo (this is not a git clone).
You cannot issue git commands from inside the directory
that is (eventually) created from that download when
running configure.

If the project wishes to no longer support the ability to
download from github in that way to build, and/or to
presume build hosts will have internet connectivity
(to perform other git operations), that is up to the
project, but it will be a change in the existing practice.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Decision on FFmpeg repository [ In reply to ]
On 05/19/2018 10:25 AM, Gary Buhrmaster wrote:
> Just so you understand, the github download (at a commit)
> does*NOT* generate a git repo (this is not a git clone).
I know that.
> You cannot issue git commands from inside the directory
> that is (eventually) created from that download when
> running configure.
Configure and make already issue git commands. That is how they
currently generate the version number, which is from a "git describe
--dirty" command. They have appropriate checks in place to cater for
users that are building from a downloaded tar file. In that case the
version number is created a different way.

In the same way, I would put in place appropriate checks to ensure that
git is not called when there is no git repository.

I will perform a test to see what is downloaded from git when you
request a zip or tar file. If that download does not contain the full
source of the correct commit of the submodule, then this approach is a
non-starter.
> If the project wishes to no longer support the ability to
> download from github in that way to build, and/or to
> presume build hosts will have internet connectivity
> (to perform other git operations), that is up to the
> project, but it will be a change in the existing practice.
If is does not work as it always has, then I will be doing it another way.

Peter
Re: Decision on FFmpeg repository [ In reply to ]
On 05/19/2018 02:53 PM, Peter Bennett wrote:
>
>
>
> On 05/19/2018 10:25 AM, Gary Buhrmaster wrote:
>> Just so you understand, the github download (at a commit)
>> does*NOT* generate a git repo (this is not a git clone).
> I know that.
>> You cannot issue git commands from inside the directory
>> that is (eventually) created from that download when
>> running configure.
> Configure and make already issue git commands. That is how they
> currently generate the version number, which is from a "git describe
> --dirty" command. They have appropriate checks in place to cater for
> users that are building from a downloaded tar file. In that case the
> version number is created a different way.
>
> In the same way, I would put in place appropriate checks to ensure
> that git is not called when there is no git repository.
>
> I will perform a test to see what is downloaded from git when you
> request a zip or tar file. If that download does not contain the full
> source of the correct commit of the submodule, then this approach is a
> non-starter.
>> If the project wishes to no longer support the ability to
>> download from github in that way to build, and/or to
>> presume build hosts will have internet connectivity
>> (to perform other git operations), that is up to the
>> project, but it will be a change in the existing practice.
> If is does not work as it always has, then I will be doing it another way.
>
Looking at some issues reported on github e.g.
https://github.com/dear-github/dear-github/issues/214), it seems that
the tar or zip file download does not include the submodule. I believe
that effectively kills the idea of using a submodule for FFmpeg or any
external component.

Peter
Re: Decision on FFmpeg repository [ In reply to ]
There were objections to using FFmpeg master. FFmpeg have now
cherry-picked their mediacodec changes to the release branch for 4.0,
which was the main reason for aiming for master. So at this point I
think I should proceed with using the release branch rather than master.

The submodule approach I believe is unworkable because the submodule is
not included in github tar downloads, so that would affect everybody who
builds from those.

The subtree approach raises many concerns around bringing in 90,000
extra commits and their logs. Also I am not sure whether starting a
subtree on one FFmpeg release branch and then switching it to another
FFmpeg release branch will work.

I am now inclined to get an FFmpeg repository of our own forked into the
MythTV project, where we can merge the branch we are using with the
latest changes from FFmpeg and our customizations. Incorporating the
code into MythTV will be a copy over, as we have always done. This is 2c
in my original email. It is not ideal but a bit better than what we have
now.

We still have the option of changing course later if desired. I will
make sure to document in the README.sync file, where the source was
obtained and the relevant commit hashes that correspond in the private
FFmpeg repository as well as in the main FFmpeg repository.

Peter.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Decision on FFmpeg repository [ In reply to ]
On Sun, 2018-05-20 at 11:00 -0400, Peter Bennett wrote:
>
> The subtree approach raises many concerns around bringing in 90,000
> extra commits and their logs.

git subrepo squashes those (any update from the last updated commit to
the current one in fact) into a single commit so you don't get that
pollution.

Cheers,
b.

1 2  View All