Mailing List Archive

FFmpeg resync
Hello there.

A new branch was created earlier this week: devel/resync-stable10

It's a resync of our ffmpeg copy which is now following ffmeg stable 1.0 branch.
Hopefully this will allow better backport of bug fixes as they are
added to the ffmpeg stable branch.

This should also allow to resync with ffmpeg during point release
(e.g. 0.27.0, 0.27.1 etc) rather than only during major new release.

The branch devel/resync-stable10 is now fully functional, I would
appreciate if some of you could give it a try.

The plan is to merge this branch back into master on Monday

Regards
Jean-Yves
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On Sat, Dec 15, 2012 at 08:16:48AM +1100, Jean-Yves Avenard wrote:
>
> This should also allow to resync with ffmpeg during point release
> (e.g. 0.27.0, 0.27.1 etc) rather than only during major new release.
>

Hi

If you are making ffmpeg updates you might be interested in the following

Since 0.26 I have had to patch ffmpeg in order to handle transcoded HDPVR
recordings.

Previously I have been transcoding HDPVR recordings from mpegts to mpeg
- for some reason which I cant exactly remember now this was the only
format that worked for playing on XBMC via mythbox - although that might
have been my inexperience at the time (2 years ago)
(at the time mythtvs frontend did not work with E350 machines)

since updating to 0.26 the transcoded mpeg format recordings would not properly
run through commercial flagging (they play OK) - failing around 90%
of the way through with an ffmpeg library error - essentially EOF
- and the last 1 or 2 commercials are not flagged

the error came from the ffmpeg function ffio_limit in libavformat/utils.c
"Truncating packet of size..."

further exploration showed that the libav version of ffmpeg does not
include this ffio_limit change (This is ubuntu 12.04)

So I use a patched version of the ffmpeg mythtv sources in which
Ive removed the call to ffio_limit in av_get_packet - which seems
to allow commercial flagging to finish

David
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On 12/14/2012 03:16 PM, Jean-Yves Avenard wrote:
...
> The branch devel/resync-stable10 is now fully functional, I would
> appreciate if some of you could give it a try.
>
> The plan is to merge this branch back into master on Monday

Hi;

OK, so I did a distclean and then a git checkout devel/resync-stable10
(then built it, of course.) git branch shows I'm on: devel/resync-stable10.

I'm up and running on it, v0.27-pre2-358-ga1eac71-dirty to be specific.

I didn't expect the -dirty. The patch looks like the one from from
http://code.mythtv.org/trac/ticket/11195 which I had applied to master.

Did you have that patch applied in that branch too, or have I missed a step?

By the way, the DB Schema Version is: 1309 from commit a1f9793 which
was released on Wed Dec 5, 2012.

--
Bill
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On 15 December 2012 10:39, Bill Meek <keemllib@gmail.com> wrote:
> OK, so I did a distclean and then a git checkout devel/resync-stable10
> (then built it, of course.) git branch shows I'm on: devel/resync-stable10.
>
> I'm up and running on it, v0.27-pre2-358-ga1eac71-dirty to be specific.
>
> I didn't expect the -dirty. The patch looks like the one from from
> http://code.mythtv.org/trac/ticket/11195 which I had applied to master.

devel/resync-stable10 is other than ffmpeg related changes, master.

So look at it as if it was master.

If you had applied a patch to master, then checked out
devel/resync-stable10 ; your patch will still be applied, hence the
-dirty when you do git describe.

if you want to run a clean copy , do a git reset --hard HEAD on it.

>
> Did you have that patch applied in that branch too, or have I missed a step?

see above.

>
> By the way, the DB Schema Version is: 1309 from commit a1f9793 which
> was released on Wed Dec 5, 2012.

devel/resync-stable10 was last merged with master on December 15th (today)
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On 15 December 2012 10:00, David Osguthorpe <david.osguthorpe@gmail.com> wrote:
> the error came from the ffmpeg function ffio_limit in libavformat/utils.c
> "Truncating packet of size..."
>
> further exploration showed that the libav version of ffmpeg does not
> include this ffio_limit change (This is ubuntu 12.04)

do you have a ffmpeg changeset I can refer to ?

Thanks
JY
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On Sat, Dec 15, 2012 at 06:39:20PM +1100, Jean-Yves Avenard wrote:
> On 15 December 2012 10:00, David Osguthorpe <david.osguthorpe@gmail.com> wrote:
> > the error came from the ffmpeg function ffio_limit in libavformat/utils.c
> > "Truncating packet of size..."
> >
> > further exploration showed that the libav version of ffmpeg does not
> > include this ffio_limit change (This is ubuntu 12.04)
>
> do you have a ffmpeg changeset I can refer to ?
>

the last git sha where av_get_packet in ffmpeg and libav are equivalent
is 7f9369b743449d2aa346557d4b6ad0655ec5d56b

in git sha 47572323f2f908913b4d031af733047d481fb1f6
changes in av_get_packet to detect packet sizes was added

this was upated in sha 7c29313b387392fa305365e855fa408cd28c276e
(first time Truncated packet of size message appears - in av_get_packet)

then this code moved into the ffio_limit function
in e39eeb10d1b36b5681e42a7539d5b168fa292d61
which is the code in current ffmpeg master

David
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On Sat, Dec 15, 2012 at 06:39:20PM +1100, Jean-Yves Avenard wrote:
> On 15 December 2012 10:00, David Osguthorpe <david.osguthorpe@gmail.com> wrote:
> > the error came from the ffmpeg function ffio_limit in libavformat/utils.c
> > "Truncating packet of size..."
> >
> > further exploration showed that the libav version of ffmpeg does not
> > include this ffio_limit change (This is ubuntu 12.04)
>
> do you have a ffmpeg changeset I can refer to ?
>

from the mythtv error messages Im suspicious that the seeking around
in commerical flagging is somehow messing with the end of file detection
this ffmpeg code seems to be using - the errors suggest the commercial
detector is reading data that ffmpeg with the ffio_limit function
claims is past EOF

David
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
> > further exploration showed that the libav version of ffmpeg does not
> > include this ffio_limit change (This is ubuntu 12.04)
>
> do you have a ffmpeg changeset I can refer to ?
>

also should have said that the ffmpeg version without the ffio_limit
changes is the code used up to and including mythtv 0.25

David

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On Sunday, 16 December 2012, David Osguthorpe wrote:

> On Sat, Dec 15, 2012 at 06:39:20PM +1100, Jean-Yves Avenard wrote:
> > On 15 December 2012 10:00, David Osguthorpe <david.osguthorpe@gmail.com<javascript:;>>
> wrote:
> > > the error came from the ffmpeg function ffio_limit in
> libavformat/utils.c
> > > "Truncating packet of size..."
> > >
> > > further exploration showed that the libav version of ffmpeg does not
> > > include this ffio_limit change (This is ubuntu 12.04)
> >
> > do you have a ffmpeg changeset I can refer to ?
> >
>
> from the mythtv error messages Im suspicious that the seeking around
> in commerical flagging is somehow messing with the end of file detection
> this ffmpeg code seems to be using - the errors suggest the commercial
> detector is reading data that ffmpeg with the ffio_limit function
> claims is past EOF
>
>
If you want changes to go into ffmpeg, the best long term solution is to
get them incorporated into ffmpeg upstream, or submit code that works
around the behaviour they have introduced on our side of the code.

From most of the cases I've seen whenever we have had an incorrect
behaviour, is because we didn't use ffmpeg properly, not the other way
round.

So I suggest you raise a bug report with ffmpeg directly.
Re: FFmpeg resync [ In reply to ]
On Sun, Dec 16, 2012 at 10:13:05AM +1100, Jean-Yves Avenard wrote:
> On Sunday, 16 December 2012, David Osguthorpe wrote:
>
> On Sat, Dec 15, 2012 at 06:39:20PM +1100, Jean-Yves Avenard wrote:
> > On 15 December 2012 10:00, David Osguthorpe
> <[1]david.osguthorpe@gmail.com> wrote:
> > > the error came from the ffmpeg function ffio_limit in
> libavformat/utils.c
> > > "Truncating packet of size..."
> > >
> > > further exploration showed that the libav version of ffmpeg does
> not
> > > include this ffio_limit change (This is ubuntu 12.04)
> >
> > do you have a ffmpeg changeset I can refer to ?
> >
> from the mythtv error messages Im suspicious that the seeking around
> in commerical flagging is somehow messing with the end of file
> detection
> this ffmpeg code seems to be using - the errors suggest the
> commercial
> detector is reading data that ffmpeg with the ffio_limit function
> claims is past EOF
>
> If you want changes to go into ffmpeg, the best long term solution is
> to get them incorporated into ffmpeg upstream, or submit code that
> works around the behaviour they have introduced on our side of the
> code.
>
> From most of the cases I've seen whenever we have had an incorrect
> behaviour, is because we didn't use ffmpeg properly, not the other way
> round.
>
> So I suggest you raise a bug report with ffmpeg directly.
>

thanks

the reason I havent reported a "bug" is because Im still not sure who is
at fault - I have had a small look at both codes and nothing seems to jump out
but it is not simple code to figure out - it may just be the setup Im using
and then there is the whole libav/ffmpeg issue - being on Ubuntu Im not using
ffmpeg but libav for external transcoding - only just beginning to understand
the difference between the mpeg format and the dvd format in ffmpeg as well

David
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
Hi

On 17 December 2012 05:04, David Osguthorpe <david.osguthorpe@gmail.com> wrote:
> the reason I havent reported a "bug" is because Im still not sure who is
> at fault - I have had a small look at both codes and nothing seems to jump out
> but it is not simple code to figure out - it may just be the setup Im using
> and then there is the whole libav/ffmpeg issue - being on Ubuntu Im not using
> ffmpeg but libav for external transcoding - only just beginning to understand
> the difference between the mpeg format and the dvd format in ffmpeg as well

at worse, the bug will be closed that it's invalid...
And often, the person who review the bug will provide hint of what can
be done around it.

I didn't know ubuntu followed libav instead. as far as I can tell,
they are all pretty much the same, with everyone cherry-picking into
the other.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On Mon, Dec 17, 2012 at 11:01:42AM +1100, Jean-Yves Avenard wrote:
>
> at worse, the bug will be closed that it's invalid...
> And often, the person who review the bug will provide hint of what can
> be done around it.

ah - useful to know

>
> I didn't know ubuntu followed libav instead. as far as I can tell,
> they are all pretty much the same, with everyone cherry-picking into
> the other.

yes - but this ffio_limit addition seems to be a difference at the moment
- maybe they will pick it up later

David

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
Hi

Please note that the merge has now been done.
master is now using ffmpeg release/1.0 stable branch
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On Monday 17 Dec 2012 16:09:53 Jean-Yves Avenard wrote:
> Hi
>
> Please note that the merge has now been done.
> master is now using ffmpeg release/1.0 stable branch

This appears to have broken DVD playback, I get audio stuttering and no video.
I'm still bisecting to be absolutely certain but if I go back before the merge
playback works and as of last night it was broken. There aren't many other
candidates except the resync.
--
Stuart Morgan
MythTV
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On Wednesday, 19 December 2012, Stuart Morgan wrote:

> On Monday 17 Dec 2012 16:09:53 Jean-Yves Avenard wrote:
> > Hi
> >
> > Please note that the merge has now been done.
> > master is now using ffmpeg release/1.0 stable branch
>
> This appears to have broken DVD playback, I get audio stuttering and no
> video.
> I'm still bisecting to be absolutely certain but if I go back before the
> merge
> playback works and as of last night it was broken. There aren't many other
> candidates except the resync.
>

If you could provide more details... Logs etc.. Is this with all DVDs?
Re: FFmpeg resync [ In reply to ]
On Thursday 20 Dec 2012 01:05:44 Jean-Yves Avenard wrote:
> On Wednesday, 19 December 2012, Stuart Morgan wrote:
> > On Monday 17 Dec 2012 16:09:53 Jean-Yves Avenard wrote:
> > > Hi
> > >
> > > Please note that the merge has now been done.
> > > master is now using ffmpeg release/1.0 stable branch
> >
> > This appears to have broken DVD playback, I get audio stuttering and no
> > video.
> > I'm still bisecting to be absolutely certain but if I go back before the
> > merge
> > playback works and as of last night it was broken. There aren't many other
> > candidates except the resync.
>
> If you could provide more details... Logs etc.. Is this with all DVDs?

All DVDs. I spent most of last night looking through logs and there's nothing
to give a clue there, they appear normal, no errors - it finds the video
stream, selects it, appears to be decoding it but no frames are rendered -
just a blank screen with audio. Same applies whether using ffmpeg/opengl or
vdpau.

Recordings play just fine.

I can provide logs once I've finished recompiling.
--
Stuart Morgan
MythTV
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg resync [ In reply to ]
On Wednesday 19 Dec 2012 14:24:59 Stuart Morgan wrote:
> > If you could provide more details... Logs etc.. Is this with all DVDs?
>
> All DVDs. I spent most of last night looking through logs and there's
> nothing to give a clue there, they appear normal, no errors - it finds the
> video stream, selects it, appears to be decoding it but no frames are
> rendered - just a blank screen with audio. Same applies whether using
> ffmpeg/opengl or vdpau.
>
> Recordings play just fine.
>
> I can provide logs once I've finished recompiling.

Seems I was wrong the logs do show _something_, I just hadn't been enabling
sufficiently verbose logs last night.

See attached
--
Stuart Morgan
MythTV
Re: FFmpeg resync [ In reply to ]
On 16 December 2012 18:04, David Osguthorpe <david.osguthorpe@gmail.com> wrote:
> and then there is the whole libav/ffmpeg issue - being on Ubuntu Im not using
> ffmpeg but libav for external transcoding - only just beginning to understand
> the difference between the mpeg format and the dvd format in ffmpeg as well
>

The main differences between mpeg format and dvd format is:
1) Packet size: mpeg is variable packet size, dvd is always 2048
bytes. (The sector size on a DVD).
2) dvd contains NAV (helps with menus) and SPU (subtitles) streams.
3) dvd has to be able to handle "still" images in video and correctly
handle sequence end codes and display all completed frames instead of
holding onto them in case a B frame appears.

Kind Regards

James
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-dev
Re: FFmpeg Resync [ In reply to ]
On 6/24/20 9:01 PM, David Hampton wrote:
> Peter,
>
>> David: Note I had to add __STDC_CONSTANT_MACROS to libmythtv.pro to
>> get the compile to work.
> That flag once meant something, but it hasn't in the last decade or so
> because the items that used to be dependent on that flag are now always
> present. Requiring that flag to be present changes nothing about the
> compilation. Since you reverted 459fccb5aa you added back the check for
> a flag which is no longer supplied by MythTV, so you needed to add back
> the declaration of the flag.
>
>> David: I see a lot of warnings about "register" being obsolete. I
>> hope there is some way of avoiding making a change to FFmpeg code for
>> this.
> I have a one line workaround that can go in the MythTV sources, but
> there are still a large number of "unused parameter" warnings caused by
> including get_bits.h that can't be fixed without changing the FFmpeg
> sources. If I change get_bits.h to fix the the unused parameter
> warnings, I can fix the storage class specifier warnings at the same
> time with the same one line addition (but in get_bits.h).
>
> There's one other patch I'd like to keep that adds two comments (of
> three lines each). This will allow me to enable automated nightly
> checking for nullptr dereferences in MythTV, without producing a bunch
> of warning messages that point at FFmpeg.
>
> I would be happy to commit both these sets of change using the
> procedure you outlined above. Its a total of nine lines of comments,
> and three lines of other stuff.
>
> David
>
>
OK, sounds good.

The devel/ffpeg-resync branch compiles successfully and runs
successfully in a quick test. You should be able to follow the procedure
in that branch. If you have any questions, problems or issues please let
me know.

Peter
Re: FFmpeg Resync [ In reply to ]
On Thu, Jun 25, 2020 at 9:51 AM Peter Bennett <pb.mythtv@gmail.com> wrote:

>
> On 6/24/20 9:01 PM, David Hampton wrote:
>
> Peter,
>
>
> David: Note I had to add __STDC_CONSTANT_MACROS to libmythtv.pro to
> get the compile to work.
>
> That flag once meant something, but it hasn't in the last decade or so
> because the items that used to be dependent on that flag are now always
> present. Requiring that flag to be present changes nothing about the
> compilation. Since you reverted 459fccb5aa you added back the check for
> a flag which is no longer supplied by MythTV, so you needed to add back
> the declaration of the flag.
>
>
> David: I see a lot of warnings about "register" being obsolete. I
> hope there is some way of avoiding making a change to FFmpeg code for
> this.
>
> I have a one line workaround that can go in the MythTV sources, but
> there are still a large number of "unused parameter" warnings caused by
> including get_bits.h that can't be fixed without changing the FFmpeg
> sources. If I change get_bits.h to fix the the unused parameter
> warnings, I can fix the storage class specifier warnings at the same
> time with the same one line addition (but in get_bits.h).
>
> There's one other patch I'd like to keep that adds two comments (of
> three lines each). This will allow me to enable automated nightly
> checking for nullptr dereferences in MythTV, without producing a bunch
> of warning messages that point at FFmpeg.
>
> I would be happy to commit both these sets of change using the
> procedure you outlined above. Its a total of nine lines of comments,
> and three lines of other stuff.
>
> David
>
>
>
> OK, sounds good.
>
> The devel/ffpeg-resync branch compiles successfully and runs successfully
> in a quick test. You should be able to follow the procedure in that branch.
> If you have any questions, problems or issues please let me know.
>
> Peter
>

I take it that ffmpeg rejected those patches? I have tried to submit
patches to them in the past, and they rejected every one of them, so I am
not surprised.

John
Re: FFmpeg Resync [ In reply to ]
Den 2020-06-25 kl. 03:01, skrev David Hampton:
> On Tue, 2020-06-23 at 14:55 -0400, Peter Bennett wrote:

<snip>

>> David: I see a lot of warnings about "register" being obsolete. I
>> hope there is some way of avoiding making a change to FFmpeg code for
>> this.
>
> I have a one line workaround that can go in the MythTV sources, but
> there are still a large number of "unused parameter" warnings caused by
> including get_bits.h that can't be fixed without changing the FFmpeg
> sources. If I change get_bits.h to fix the the unused parameter
> warnings, I can fix the storage class specifier warnings at the same
> time with the same one line addition (but in get_bits.h).

I think using -isystem for external/FFmpeg would get rid of those warnings. Or is there a reason we are only using that for external/libmythdvdnav/dvdnav and external/libmythdvdnav/dvdread?

Cheers
Jonatan
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: FFmpeg Resync [ In reply to ]
On Thu, 2020-06-25 at 23:11 +0200, Jonatan Lindblad wrote:
> Den 2020-06-25 kl. 03:01, skrev David Hampton:
> > On Tue, 2020-06-23 at 14:55 -0400, Peter Bennett wrote:
>
> <snip>
>
> > > David: I see a lot of warnings about "register" being obsolete. I
> > > hope there is some way of avoiding making a change to FFmpeg code
> > > for
> > > this.
> >
> > I have a one line workaround that can go in the MythTV sources, but
> > there are still a large number of "unused parameter" warnings
> > caused by
> > including get_bits.h that can't be fixed without changing the
> > FFmpeg
> > sources. If I change get_bits.h to fix the the unused parameter
> > warnings, I can fix the storage class specifier warnings at the
> > same
> > time with the same one line addition (but in get_bits.h).
>
> I think using -isystem for external/FFmpeg would get rid of those
> warnings. Or is there a reason we are only using that for
> external/libmythdvdnav/dvdnav and external/libmythdvdnav/dvdread?

Hi Jonatan,

For some reason that doesn't work for these errors. I just cleaned my
tree and tried again from scratch. The clang compile_commands.json
database shows that external/FFmpeg is included with -isystem, but I
still see the null pointer dereference warning for a macro from the
file external/FFmpeg/libavcodec/get_bits.h .

David


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org