Mailing List Archive

Current state-of-the-art for h264 cutlist processing?
Hi: I rarely record DVB-T2 HD h264 recordings in which I want to make
internal cuts, but it would be good to be able to do make them in a way
that doesn't introduce glitches in playback.

I find that Mythfrontend does a good job in playing uncut recordings
with cutlists, but uPnP then plays all the ads and skipping is painful.

My MythDVBcut script chops at videokeyframes; there may be glitches at
cuts but there's no long-term loss of quality or a/v sync. I've found
other scripts that work similarly and are more elegantly coded.
Frontend and uPnP glitches may differ in style and severity.

I've tried going via mkvmerge but at present see a/v sync drift and
haven't found an mkv/audio format that my tv plays acceptably via uPnP

So, before I waste yet more time, has anyone found (or developed) a
toolchain that is available and does this job properly?

Thanks,

John P
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Current state-of-the-art for h264 cutlist processing? [ In reply to ]
On 02/01/2018 08:10 AM, John Pilkington wrote:
> Hi:  I rarely record DVB-T2 HD h264 recordings in which I want to make
> internal cuts, but it would be good to be able to do make them in a
> way that doesn't introduce glitches in playback.
>
> I find that Mythfrontend does a good job in playing uncut recordings
> with cutlists, but uPnP then plays all the ads and skipping is painful.
>
> My MythDVBcut script chops at videokeyframes; there may be glitches at
> cuts but there's no long-term loss of quality or a/v sync.  I've found
> other scripts that work similarly and are more elegantly coded.
> Frontend and uPnP glitches may differ in style and severity.
> I've tried going via mkvmerge but at present see a/v sync drift and
> haven't found an mkv/audio format that my tv plays acceptably via uPnP
>
> So, before I waste yet more time, has anyone found (or developed) a
> toolchain that is available and does this job properly?

I don't know if what I have is an exact match for what you are trying to
do.  All my recordings come from cable for one thing, not DVB.  But,
anyway, through experimentation I've developed a workflow that produces
cleanly cut videos from MythTV recordings and cutlists.  The key missing
link for me was discovering the "--cleancut" option on the mythtranscode
command.  This lets cuts made on non-keyframes survive the transcoding
process.

My workflow is:

Edit the recording in MythTV, fine tuning the cutlist,
Start mythtranscode feeding the recording into audio and video fifos
with --honorcutlist and --cleancut options,
Start ffmpeg simultaneously, reading from the fifos and outputing a
lossless h264 MKV,
Use Handbrake to transcode the lossless MKV down into standard mp4 format.

I have a Python script that does this plus more.  It's got stuff in it
that's specific to my needs and it isn't generalized for use by others. 
But it might provide you with some hints on command options to try.  I
run it standalone on my desktop PC where I only have mythfrontend
installed.  It talks to the backend over the network and uses NFS to
store the resulting video file on the backend. 
https://pastebin.com/P5P21zJ9

--
David King
dave@daveking.com


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Current state-of-the-art for h264 cutlist processing? [ In reply to ]
On 02/02/18 15:56, David King wrote:
> On 02/01/2018 08:10 AM, John Pilkington wrote:
>> Hi:  I rarely record DVB-T2 HD h264 recordings in which I want to make
>> internal cuts, but it would be good to be able to do make them in a
>> way that doesn't introduce glitches in playback.
>>
>> I find that Mythfrontend does a good job in playing uncut recordings
>> with cutlists, but uPnP then plays all the ads and skipping is painful.
>>
>> My MythDVBcut script chops at videokeyframes; there may be glitches at
>> cuts but there's no long-term loss of quality or a/v sync.  I've found
>> other scripts that work similarly and are more elegantly coded.
>> Frontend and uPnP glitches may differ in style and severity.
>> I've tried going via mkvmerge but at present see a/v sync drift and
>> haven't found an mkv/audio format that my tv plays acceptably via uPnP
>>
>> So, before I waste yet more time, has anyone found (or developed) a
>> toolchain that is available and does this job properly?
>
> I don't know if what I have is an exact match for what you are trying to
> do.  All my recordings come from cable for one thing, not DVB.  But,
> anyway, through experimentation I've developed a workflow that produces
> cleanly cut videos from MythTV recordings and cutlists.  The key missing
> link for me was discovering the "--cleancut" option on the mythtranscode
> command.  This lets cuts made on non-keyframes survive the transcoding
> process.
>
> My workflow is:
>
> Edit the recording in MythTV, fine tuning the cutlist,
> Start mythtranscode feeding the recording into audio and video fifos
> with --honorcutlist and --cleancut options,
> Start ffmpeg simultaneously, reading from the fifos and outputing a
> lossless h264 MKV,
> Use Handbrake to transcode the lossless MKV down into standard mp4 format.
>
> I have a Python script that does this plus more.  It's got stuff in it
> that's specific to my needs and it isn't generalized for use by others.
> But it might provide you with some hints on command options to try.  I
> run it standalone on my desktop PC where I only have mythfrontend
> installed.  It talks to the backend over the network and uses NFS to
> store the resulting video file on the backend.
> https://pastebin.com/P5P21zJ9
>
Thank you for posting this. I should have said above that I have found
cutting at keyframes quite acceptable with almost all my SD recordings
from digital sources, but I think this must depend on the way the
content is edited at source - and I'm usually just trying to remove
commercials rather than striving for a subtle artistic effect. But I
will certainly examine your code with interest. It may take some time.

Thanks again.

John

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Current state-of-the-art for h264 cutlist processing? [ In reply to ]
On 02/02/18 18:42, John Pilkington wrote:
> On 02/02/18 15:56, David King wrote:
>> On 02/01/2018 08:10 AM, John Pilkington wrote:
>>> Hi:  I rarely record DVB-T2 HD h264 recordings in which I want to make
>>> internal cuts, but it would be good to be able to do make them in a
>>> way that doesn't introduce glitches in playback.
>>>
>>> I find that Mythfrontend does a good job in playing uncut recordings
>>> with cutlists, but uPnP then plays all the ads and skipping is painful.
>>>
>>> My MythDVBcut script chops at videokeyframes; there may be glitches at
>>> cuts but there's no long-term loss of quality or a/v sync.  I've found
>>> other scripts that work similarly and are more elegantly coded.
>>> Frontend and uPnP glitches may differ in style and severity.
>>> I've tried going via mkvmerge but at present see a/v sync drift and
>>> haven't found an mkv/audio format that my tv plays acceptably via uPnP
>>>
>>> So, before I waste yet more time, has anyone found (or developed) a
>>> toolchain that is available and does this job properly?
>>
>> I don't know if what I have is an exact match for what you are trying to
>> do.  All my recordings come from cable for one thing, not DVB.  But,
>> anyway, through experimentation I've developed a workflow that produces
>> cleanly cut videos from MythTV recordings and cutlists.  The key missing
>> link for me was discovering the "--cleancut" option on the mythtranscode
>> command.  This lets cuts made on non-keyframes survive the transcoding
>> process.
>>
>> My workflow is:
>>
>> Edit the recording in MythTV, fine tuning the cutlist,
>> Start mythtranscode feeding the recording into audio and video fifos
>> with --honorcutlist and --cleancut options,
>> Start ffmpeg simultaneously, reading from the fifos and outputing a
>> lossless h264 MKV,
>> Use Handbrake to transcode the lossless MKV down into standard mp4
>> format.
>>
>> I have a Python script that does this plus more.  It's got stuff in it
>> that's specific to my needs and it isn't generalized for use by others.
>> But it might provide you with some hints on command options to try.  I
>> run it standalone on my desktop PC where I only have mythfrontend
>> installed.  It talks to the backend over the network and uses NFS to
>> store the resulting video file on the backend.
>> https://pastebin.com/P5P21zJ9
>>
> Thank you for posting this.  I should have said above that I have found
> cutting at keyframes quite acceptable with almost all my SD recordings
> from digital sources, but I think this must depend on the way the
> content is edited at source - and I'm usually just trying to remove
> commercials rather than striving for a subtle artistic effect.  But I
> will certainly examine your code with interest.  It may take some time.
>

I played with your script yesterday: interesting, and I learned some
new wrinkles. I knew of the fifo-based technique but hadn't realised
that it decodes and re-encodes the entire file, so it will be quite
slow, and I may set it aside for now.

I had a unicode-related problem when I tried it on a second box, and
google gave me lots of python-speak. It read the titles after I had
looked at (for example) mytharchive's mythburn.py and copied in the
first 3 lines.

> Thanks again.
>
> John
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Current state-of-the-art for h264 cutlist processing? [ In reply to ]
On 01/02/18 13:10, John Pilkington wrote:
> Hi:  I rarely record DVB-T2 HD h264 recordings in which I want to make
> internal cuts, but it would be good to be able to do make them in a way
> that doesn't introduce glitches in playback.
>
> I find that Mythfrontend does a good job in playing uncut recordings
> with cutlists, but uPnP then plays all the ads and skipping is painful.
>
> My MythDVBcut script chops at videokeyframes; there may be glitches at
> cuts but there's no long-term loss of quality or a/v sync.  I've found
> other scripts that work similarly and are more elegantly coded. Frontend
> and uPnP glitches may differ in style and severity.
>
> I've tried going via mkvmerge but at present see a/v sync drift and
> haven't found an mkv/audio format that my tv plays acceptably via uPnP
>
> So, before I waste yet more time, has anyone found (or developed) a
> toolchain that is available and does this job properly?
>
> Thanks,
>

I posted that a few weeks ago. There hasn't been much comeback, perhaps
because multi-terabyte disks have become affordable, or perhaps through
disillusionment, but I do now have a script that works reasonably well
with UK DVB-T2 HD h264 recordings and might be useful elsewhere. I have
appended it to the MythDVBcut entry in the MythTV wiki; I've called it
MythTScut.

It uses mythffmpeg, with mapping to give one video and one audio stream.
That could probably be changed, but I haven't tested any more. I find
that it needs two runs, the first with no cutlist, to discard
unrecognised streams and get the cuts in (about) the right position. At
present the most successful commands seem to be:
============
with no cutlist

mythTScut.sh /path/to/infile.ts R 1

mv infile.ts.old ... to somewhere else, say infile_orig.ts

establish editpoints showing wanted keyframes using the frontend editor

mythTScut.sh /path/to/infile.ts R 1
============
The output may be either a Recording (R), replacing the renamed original
file and clearing its cutlist and bookmarks, or a Video (V) which leaves
these unchanged. The third parameter (1,2) is intended to define the way
in which the input streams are synched, but I find that 2 is still
rather unpredictable or prone to failure. I have tried several ffmpeg
options without getting anything that seemed better. YMMV

ffmpeg complains during both passes, which are primarily
disk-speed-limited. It still usually works.

Like my earlier scripts, this is intended for use from the desktop,
giving feedback and creating a logfile that includes the position of the
cutpoints in the output, and stream details from ffprobe. The first
page needs editing to define working directories and database access,
and intermediate files are not all erased on exit.

I'm posting it as work-in-progress with no guarantees and no implied
support, in the hope that it may be useful.

> John P


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Current state-of-the-art for h264 cutlist processing? [ In reply to ]
On 21 March 2018 at 03:36, John Pilkington <J.Pilk@tesco.net> wrote:

> On 01/02/18 13:10, John Pilkington wrote:
>
>> Hi: I rarely record DVB-T2 HD h264 recordings in which I want to make
>> internal cuts, but it would be good to be able to do make them in a way
>> that doesn't introduce glitches in playback.
>>
>> I find that Mythfrontend does a good job in playing uncut recordings with
>> cutlists, but uPnP then plays all the ads and skipping is painful.
>>
>> My MythDVBcut script chops at videokeyframes; there may be glitches at
>> cuts but there's no long-term loss of quality or a/v sync. I've found
>> other scripts that work similarly and are more elegantly coded. Frontend
>> and uPnP glitches may differ in style and severity.
>>
>> I've tried going via mkvmerge but at present see a/v sync drift and
>> haven't found an mkv/audio format that my tv plays acceptably via uPnP
>>
>> So, before I waste yet more time, has anyone found (or developed) a
>> toolchain that is available and does this job properly?
>>
>> Thanks,
>>
>>
> I posted that a few weeks ago. There hasn't been much comeback, perhaps
> because multi-terabyte disks have become affordable, or perhaps through
> disillusionment, but I do now have a script that works reasonably well with
> UK DVB-T2 HD h264 recordings and might be useful elsewhere. I have
> appended it to the MythDVBcut entry in the MythTV wiki; I've called it
> MythTScut.
>
> It uses mythffmpeg, with mapping to give one video and one audio stream.
> That could probably be changed, but I haven't tested any more. I find that
> it needs two runs, the first with no cutlist, to discard unrecognised
> streams and get the cuts in (about) the right position. At present the
> most successful commands seem to be:
> ============
> with no cutlist
>
> mythTScut.sh /path/to/infile.ts R 1
>
> mv infile.ts.old ... to somewhere else, say infile_orig.ts
>
> establish editpoints showing wanted keyframes using the frontend editor
>
> mythTScut.sh /path/to/infile.ts R 1
> ============
> The output may be either a Recording (R), replacing the renamed original
> file and clearing its cutlist and bookmarks, or a Video (V) which leaves
> these unchanged. The third parameter (1,2) is intended to define the way in
> which the input streams are synched, but I find that 2 is still rather
> unpredictable or prone to failure. I have tried several ffmpeg options
> without getting anything that seemed better. YMMV
>
> ffmpeg complains during both passes, which are primarily
> disk-speed-limited. It still usually works.
>
> Like my earlier scripts, this is intended for use from the desktop, giving
> feedback and creating a logfile that includes the position of the cutpoints
> in the output, and stream details from ffprobe. The first page needs
> editing to define working directories and database access, and intermediate
> files are not all erased on exit.
>
> I'm posting it as work-in-progress with no guarantees and no implied
> support, in the hope that it may be useful.
>
>
> > John P
>
> Thanks John,

I'll give it a whirl, typically I just been avoiding recording h264
channels where possible.

Cheers,

Anthony
Re: Current state-of-the-art for h264 cutlist processing? [ In reply to ]
On 22/03/18 08:26, Anthony Giggins wrote:


On 21 March 2018 at 03:36, John Pilkington <J.Pilk@tesco.net<mailto:J.Pilk@tesco.net>> wrote:
On 01/02/18 13:10, John Pilkington wrote:
Hi: I rarely record DVB-T2 HD h264 recordings in which I want to make internal cuts, but it would be good to be able to do make them in a way that doesn't introduce glitches in playback.

I find that Mythfrontend does a good job in playing uncut recordings with cutlists, but uPnP then plays all the ads and skipping is painful.

My MythDVBcut script chops at videokeyframes; there may be glitches at cuts but there's no long-term loss of quality or a/v sync. I've found other scripts that work similarly and are more elegantly coded. Frontend and uPnP glitches may differ in style and severity.

I've tried going via mkvmerge but at present see a/v sync drift and haven't found an mkv/audio format that my tv plays acceptably via uPnP

So, before I waste yet more time, has anyone found (or developed) a toolchain that is available and does this job properly?

Thanks,


I posted that a few weeks ago. There hasn't been much comeback, perhaps because multi-terabyte disks have become affordable, or perhaps through disillusionment, but I do now have a script that works reasonably well with UK DVB-T2 HD h264 recordings and might be useful elsewhere. I have appended it to the MythDVBcut entry in the MythTV wiki; I've called it MythTScut.

It uses mythffmpeg, with mapping to give one video and one audio stream. That could probably be changed, but I haven't tested any more. I find that it needs two runs, the first with no cutlist, to discard unrecognised streams and get the cuts in (about) the right position. At present the most successful commands seem to be:
============
with no cutlist

mythTScut.sh /path/to/infile.ts R 1

mv infile.ts.old ... to somewhere else, say infile_orig.ts

establish editpoints showing wanted keyframes using the frontend editor

mythTScut.sh /path/to/infile.ts R 1
============
The output may be either a Recording (R), replacing the renamed original file and clearing its cutlist and bookmarks, or a Video (V) which leaves these unchanged. The third parameter (1,2) is intended to define the way in which the input streams are synched, but I find that 2 is still rather unpredictable or prone to failure. I have tried several ffmpeg options without getting anything that seemed better. YMMV

ffmpeg complains during both passes, which are primarily disk-speed-limited. It still usually works.

Like my earlier scripts, this is intended for use from the desktop, giving feedback and creating a logfile that includes the position of the cutpoints in the output, and stream details from ffprobe. The first page needs editing to define working directories and database access, and intermediate files are not all erased on exit.

I'm posting it as work-in-progress with no guarantees and no implied support, in the hope that it may be useful.


> John P

Thanks John,

I'll give it a whirl, typically I just been avoiding recording h264 channels where possible.

Cheers,

Anthony



This one is on my long term wish list to actually do properly, although it may never happen with the way online streaming is going. I use a heavily customised script that tries to be excessively clever (which perhaps is another way of saying excessively complicated with minimal benefit). But underpinning it is mythtranscode in FIFO mode piped into ffmpeg which I have found to be excellent and pretty much flawless. I must admit I haven't really looked to see whether any of my transcodes have been from h264 or not but I just tried it on one and it appeared to work without a noticeable glitch, and I think (maybe) it might be frame accurate cuts as well. What sort of glitch do you get with your script at cut points?

The biggest hold-up is checking the cut list. It is time consuming and as you say with multi TB drives it has been easier to just add more drives and use the skip keys. I have probably only bothered to transcode a dozen programs this year.
Re: Current state-of-the-art for h264 cutlist processing? [ In reply to ]
On 22/03/18 11:26, Mark Perkins wrote:
>
>
> On 22/03/18 08:26, Anthony Giggins wrote:
>>
>>
>> On 21 March 2018 at 03:36, John Pilkington <J.Pilk@tesco.net
>> <mailto:J.Pilk@tesco.net>> wrote:
>>
>> On 01/02/18 13:10, John Pilkington wrote:
>>
>> Hi:  I rarely record DVB-T2 HD h264 recordings in which I want
>> to make internal cuts, but it would be good to be able to do
>> make them in a way that doesn't introduce glitches in playback.
>>
>> I find that Mythfrontend does a good job in playing uncut
>> recordings with cutlists, but uPnP then plays all the ads and
>> skipping is painful.
>>
>> My MythDVBcut script chops at videokeyframes; there may be
>> glitches at cuts but there's no long-term loss of quality or
>> a/v sync.  I've found other scripts that work similarly and
>> are more elegantly coded. Frontend and uPnP glitches may
>> differ in style and severity.
>>
>> I've tried going via mkvmerge but at present see a/v sync
>> drift and haven't found an mkv/audio format that my tv plays
>> acceptably via uPnP
>>
>> So, before I waste yet more time, has anyone found (or
>> developed) a toolchain that is available and does this job
>> properly?
>>
>> Thanks,
>>
>>
>> I posted that a few weeks ago.  There hasn't been much comeback,
>> perhaps because multi-terabyte disks have become affordable, or
>> perhaps through disillusionment, but I do now have a script that
>> works reasonably well with UK DVB-T2 HD h264 recordings and might
>> be useful elsewhere.  I have appended it to the MythDVBcut entry
>> in the MythTV wiki; I've called it MythTScut.
>>
>> It uses mythffmpeg, with mapping to give one video and one audio
>> stream.  That could probably be changed, but I haven't tested any
>> more.  I find that it needs two runs, the first with no cutlist,
>> to discard unrecognised streams and get the cuts in (about) the
>> right position.  At present the most successful commands seem to be:
>> ============
>> with no cutlist
>>
>> mythTScut.sh /path/to/infile.ts R 1
>>
>> mv infile.ts.old ... to somewhere else, say infile_orig.ts
>>
>> establish editpoints showing wanted keyframes using the frontend
>> editor
>>
>> mythTScut.sh /path/to/infile.ts R 1
>> ============
>> The output may be either a Recording (R), replacing the renamed
>> original file and clearing its cutlist and bookmarks, or a Video
>> (V) which leaves these unchanged. The third parameter (1,2) is
>> intended to define the way in which the input streams are synched,
>> but I find that 2 is still rather unpredictable or prone to
>> failure.  I have tried several ffmpeg options without getting
>> anything that seemed better. YMMV
>>
>> ffmpeg complains during both passes, which are primarily
>> disk-speed-limited.  It still usually works.
>>
>> Like my earlier scripts, this is intended for use from the
>> desktop, giving feedback and creating a logfile that includes the
>> position of the cutpoints in the output, and stream details from
>> ffprobe.  The first page needs editing to define working
>> directories and database access, and intermediate files are not
>> all erased on exit.
>>
>> I'm posting it as work-in-progress with no guarantees and no
>> implied support, in the hope that it may be useful.
>>
>>
>> > John P
>>
>> Thanks John,
>>
>> I'll give it a whirl, typically I just been avoiding recording h264
>> channels where possible.
>>
>> Cheers,
>>
>> Anthony
>>
>>
>
> This one is on my long term wish list to actually do properly, although
> it may never happen with the way online streaming is going. I use a
> heavily customised script that tries to be excessively clever (which
> perhaps is another way of saying excessively complicated with minimal
> benefit). But underpinning it is mythtranscode in FIFO mode piped into
> ffmpeg which I have found to be excellent and pretty much flawless. I
> must admit I haven't really looked to see whether any of my transcodes
> have been from h264 or not but I just tried it on one and it appeared to
> work without a noticeable glitch, and I think (maybe) it might be frame
> accurate cuts as well. What sort of glitch do you get with your script
> at cut points?

I can't really give a 'typical' answer because I haven't done many runs
and changes have been made - but I've just played through 3 cuts in 'The
Durrells' from itv HD and they weren't conspicuous. -v playback showed
recoveries from 'Video is n frames ahead of audio', where n was around
12, 120, and 15. That was processed with two 'R 1' runs. I think the
numbers looked smaller on an earlier recording when the second run was
'R 2' but haven't rechecked that and its skew setting. It looks as if
the ffmpeg 'concat' protocol is perhaps not quite right, but I haven't
yet ventured onto the ffmpeg lists.

ISTR that when I looked at the FIFO-mode approach it looked
Ubuntu-specific or used a job-management feature that I have so far
avoided. Perhaps a mistake. But I haven't made routine HD recordings
anyway, and even few of those have needed internal cuts.


>
> The biggest hold-up is checking the cut list. It is time consuming and
> as you say with multi TB drives it has been easier to just add more
> drives and use the skip keys. I have probably only bothered to transcode
> a dozen programs this year.


>
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org