Mailing List Archive

Helpful feedback
G’day all

I’ve been struggling with the local (Australian ABC) HD H264 recordinggs. mythtranscode chokes and all my (undoubtably naive and silly) fiddling with ffmpeg trying to cut commercials fails (ie bitrate is so slow the picture is blocky) (ABC is a non-commercial channel but they sure pack in a lot of Next, Forthcomming, and what's on iView their streaming service)

I’ve found Shotcut works well (well flawlessly) just a bit slow. The deinterlacer and interpolator are really good.

James
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Helpful feedback [ In reply to ]
On Sat, 14 Apr 2018 10:37:07 +0800, you wrote:

>G?day all
>
>I?ve been struggling with the local (Australian ABC) HD H264 recordinggs. mythtranscode chokes and all my (undoubtably naive and silly) fiddling with ffmpeg trying to cut commercials fails (ie bitrate is so slow the picture is blocky) (ABC is a non-commercial channel but they sure pack in a lot of Next, Forthcomming, and what's on iView their streaming service)
>
>I?ve found Shotcut works well (well flawlessly) just a bit slow. The deinterlacer and interpolator are really good.

When you are trying to cut out bits of a video file, there are two
ways to do it. It depends on the codecs involved, but most work as I
am describing here (in a simplified way). The video data is mostly
made up of two different sorts of frames. The "key" frames, and the
other frames between the key frames. Only a key frame has all the
data necessary to display the picture for that frame. The other
frames only have data that tells the differences between that frame
and the previous one.

When you tell your cutting program that you want to cut at a key frame
and stop cutting at another key frame, there is no problem - it just
copies all the data up to the key frame at the start of the cut, then
drops all the frame data until the key frame for the end of the cut.
Similarly, if the start of the cut is at a between frames, and then
end is on a key frame, that works too. Data gets dropped for all the
frames from the start of the cut to the end, but since the end is a
key frame, it and the data after it can be copied and you still have a
valid video stream.

When you want a cut to start at a key frame and end between key
frames, there is a big problem. If the data from non-key frames does
not have all the frames prior to and including the previous key frame
in the output video stream, it will be invalid. The playing program
may be able to find another key frame later in the video stream and
recover from there, but it may also just get lost and stop playing. So
the cutting software needs to calculate all the changes in the video
output from the prior key frame to the cut point, and put into the
output stream a new calculated key frame, and then recalculate all the
non-key frames from that new key frame on until the next key frame in
the input stream.

Video cutting software has two ways it normally handles this problem.
Either it ignores the precise frame you tell it to cut at, and cuts at
the nearest key frame, or instead of just doing cutting, it completely
transcodes the entire video stream, regenerating the full frame at
each key and non-key frame, and then re-compressing to produce the
output stream. The first strategy gives you very imprecise cut
points, so you still get bits of the things you were trying to cut
out. And if the software is bad (and it often is), you can also miss
bits of the video you want, as it will choose the wrong key frame to
cut at. But the cutting is very quick - it just takes as long as it
takes to read the file and write the new one. If the second strategy
of doing a full transcode is used, you get precise cutting, but as
with any transcode using lossy encoding, the resulting video output is
of lower quality than the input was - there is another set of
compression loss. And it takes a long time as video decompression and
recompression is a very CPU intensive operation, and not well suited
to use of multiple CPU cores.

Only very few cutting programs do it correctly, using the proper
strategy of copying the data when possible, and only doing a full
transcode when necessary. This takes a little longer that cutting at
key frames only, but as only a small proportion of the data gets
transcoded, it is nearly as fast as the first strategy. You do still
get quality loss at the cut points where transcoding is necessary, but
it is generally not too noticeable as it is for only very short
periods.

My guess is that Shotcut is using the second strategy of completely
transcoding the video. That would explain why it is slow. But check
its options to see if it does have any that might make it use the
third, correct, strategy.

And I would also like to suggest that cutting out the advertising is
not really necessary. Unlike most TV software I have tried,
mythfrontend is very good at being able to manually skip forward and
backward using the remote when you hit an ad break. With the correct
settings for how skip works, it becomes second nature to just skip
forward until you see program again, then back a little to the last of
the ad break, and let it play. You see only less than 10 seconds of
ads and then the program again. In New Zealand, the automatic
commercial detection does not work on any of our channels, so if we
want to cut out the ads, we would have to manually mark where they are
before trying to transcode. That is way too much effort when it is so
easy to just manually skip when you get to the ads. This is a huge
advantage MythTV has over its competitors - it really works well. But
it relies on the correct setup of the skip settings.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Helpful feedback [ In reply to ]
On 14/04/18 03:37, James Linder wrote:
> G’day all
>
> I’ve been struggling with the local (Australian ABC) HD H264 recordinggs. mythtranscode chokes and all my (undoubtably naive and silly) fiddling with ffmpeg trying to cut commercials fails (ie bitrate is so slow the picture is blocky) (ABC is a non-commercial channel but they sure pack in a lot of Next, Forthcomming, and what's on iView their streaming service)
>
> I’ve found Shotcut works well (well flawlessly) just a bit slow. The deinterlacer and interpolator are really good.
>
> James

I've just updated mythTScut.sh at the foot of this page:

https://www.mythtv.org/wiki/MythDVBcut

You might like to try it. Cuts aren't yet perfect but the transients
are usually short-lived. Video and one audio stream only, two pass,
fast. Desktop interface.

John

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Helpful feedback [ In reply to ]
> On 14 Apr 2018, at 8:00 pm, mythtv-users-request@mythtv.org wrote:
>
>> G?day all
>>
>> I?ve been struggling with the local (Australian ABC) HD H264 recordinggs. mythtranscode chokes and all my (undoubtably naive and silly) fiddling with ffmpeg trying to cut commercials fails (ie bitrate is so slow the picture is blocky) (ABC is a non-commercial channel but they sure pack in a lot of Next, Forthcomming, and what's on iView their streaming service)
>>
>> I?ve found Shotcut works well (well flawlessly) just a bit slow. The deinterlacer and interpolator are really good.
>
> When you are trying to cut out bits of a video file, there are two
> ways to do it. It depends on the codecs involved, but most work as I
> am describing here (in a simplified way). The video data is mostly
> made up of two different sorts of frames. The "key" frames, and the
> other frames between the key frames. Only a key frame has all the
> data necessary to display the picture for that frame. The other
> frames only have data that tells the differences between that frame
> and the previous one.
>
> When you tell your cutting program that you want to cut at a key frame
> and stop cutting at another key frame, there is no problem - it just
> copies all the data up to the key frame at the start of the cut, then
> drops all the frame data until the key frame for the end of the cut.
> Similarly, if the start of the cut is at a between frames, and then
> end is on a key frame, that works too. Data gets dropped for all the
> frames from the start of the cut to the end, but since the end is a
> key frame, it and the data after it can be copied and you still have a
> valid video stream.
>
> When you want a cut to start at a key frame and end between key
> frames, there is a big problem. If the data from non-key frames does
> not have all the frames prior to and including the previous key frame
> in the output video stream, it will be invalid. The playing program
> may be able to find another key frame later in the video stream and
> recover from there, but it may also just get lost and stop playing. So
> the cutting software needs to calculate all the changes in the video
> output from the prior key frame to the cut point, and put into the
> output stream a new calculated key frame, and then recalculate all the
> non-key frames from that new key frame on until the next key frame in
> the input stream.
>
> Video cutting software has two ways it normally handles this problem.
> Either it ignores the precise frame you tell it to cut at, and cuts at
> the nearest key frame, or instead of just doing cutting, it completely
> transcodes the entire video stream, regenerating the full frame at
> each key and non-key frame, and then re-compressing to produce the
> output stream. The first strategy gives you very imprecise cut
> points, so you still get bits of the things you were trying to cut
> out. And if the software is bad (and it often is), you can also miss
> bits of the video you want, as it will choose the wrong key frame to
> cut at. But the cutting is very quick - it just takes as long as it
> takes to read the file and write the new one. If the second strategy
> of doing a full transcode is used, you get precise cutting, but as
> with any transcode using lossy encoding, the resulting video output is
> of lower quality than the input was - there is another set of
> compression loss. And it takes a long time as video decompression and
> recompression is a very CPU intensive operation, and not well suited
> to use of multiple CPU cores.
>
> Only very few cutting programs do it correctly, using the proper
> strategy of copying the data when possible, and only doing a full
> transcode when necessary. This takes a little longer that cutting at
> key frames only, but as only a small proportion of the data gets
> transcoded, it is nearly as fast as the first strategy. You do still
> get quality loss at the cut points where transcoding is necessary, but
> it is generally not too noticeable as it is for only very short
> periods.
>
> My guess is that Shotcut is using the second strategy of completely
> transcoding the video. That would explain why it is slow. But check
> its options to see if it does have any that might make it use the
> third, correct, strategy.
>
> And I would also like to suggest that cutting out the advertising is
> not really necessary. Unlike most TV software I have tried,
> mythfrontend is very good at being able to manually skip forward and
> backward using the remote when you hit an ad break. With the correct
> settings for how skip works, it becomes second nature to just skip
> forward until you see program again, then back a little to the last of
> the ad break, and let it play. You see only less than 10 seconds of
> ads and then the program again. In New Zealand, the automatic
> commercial detection does not work on any of our channels, so if we
> want to cut out the ads, we would have to manually mark where they are
> before trying to transcode. That is way too much effort when it is so
> easy to just manually skip when you get to the ads. This is a huge
> advantage MythTV has over its competitors - it really works well. But
> it relies on the correct setup of the skip settings.

Stephen thanks for all the info.

While I agree re cutting ads (and commercial detection in Oz is a bas as NZ sounds) I chop ads from movies that I want to keep because skipping on my samsung tv using uPnP is very tiresome.
I also need to recode the video and recontainer it because of foibles of samsung uPnP. ie it will procaim ‘unsupported format’
but play the same movie from a usb stick.

All my cutting is done with mythtv

Cheers
James

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Helpful feedback [ In reply to ]
> On 15 Apr 2018, at 8:00 pm, mythtv-users-request@mythtv.org wrote:
>
>> G?day all
>>
>> I?ve been struggling with the local (Australian ABC) HD H264 recordinggs. mythtranscode chokes and all my (undoubtably naive and silly) fiddling with ffmpeg trying to cut commercials fails (ie bitrate is so slow the picture is blocky) (ABC is a non-commercial channel but they sure pack in a lot of Next, Forthcomming, and what's on iView their streaming service)
>>
>> I?ve found Shotcut works well (well flawlessly) just a bit slow. The deinterlacer and interpolator are really good.
>>
>> James
>
> I've just updated mythTScut.sh at the foot of this page:
>
> https://www.mythtv.org/wiki/MythDVBcut
>
> You might like to try it. Cuts aren't yet perfect but the transients
> are usually short-lived. Video and one audio stream only, two pass,
> fast. Desktop interface.

John for a bear of little brain (AA Milne) I’m totally confused by your script:

# Suggested work sequence is:
# Preliminary test: ~/mythTScut.sh /path/to/infile T 1
# which will read the database and process a cutlist but should not touch the recording.

What are we trying to achieve


# ionice -c3 ~/mythTScut.sh /path/to/infile R 1 NC

Umm again what are we trying to achieve

# It's best to establish the cutlist here, or at least to confirm its accuracy.

What does best do and what is not best, what does establish the cutlist here mean.

# ionice -c3 ~/mythTScut.sh /path/to/infile V 1
# scan for new videos from the mythfrontend 'Watch Videos' screen
# ionice -c3 mythutil --rebuild --video /path/to/the/video/outfile

Again confused about what is happening

I’m not trying in any way to dis your work, I just dont see how to use it
Thanks so
James
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Helpful feedback [ In reply to ]
On 16/04/18 04:26, James Linder wrote:
>
>
>> On 15 Apr 2018, at 8:00 pm, mythtv-users-request@mythtv.org wrote:
>>
>>> G?day all
>>>
>>> I?ve been struggling with the local (Australian ABC) HD H264 recordinggs. mythtranscode chokes and all my (undoubtably naive and silly) fiddling with ffmpeg trying to cut commercials fails (ie bitrate is so slow the picture is blocky) (ABC is a non-commercial channel but they sure pack in a lot of Next, Forthcomming, and what's on iView their streaming service)
>>>
>>> I?ve found Shotcut works well (well flawlessly) just a bit slow. The deinterlacer and interpolator are really good.
>>>
>>> James
>>
>> I've just updated mythTScut.sh at the foot of this page:
>>
>> https://www.mythtv.org/wiki/MythDVBcut
>>
>> You might like to try it. Cuts aren't yet perfect but the transients
>> are usually short-lived. Video and one audio stream only, two pass,
>> fast. Desktop interface.
>
> John for a bear of little brain (AA Milne) I’m totally confused by your script:
>
> # Suggested work sequence is:
> # Preliminary test: ~/mythTScut.sh /path/to/infile T 1
> # which will read the database and process a cutlist but should not touch the recording.
>
> What are we trying to achieve

To establish that the fundamentals are in place: reading the cutlist
and database without changing anything, before going ahead. Terminal
output is intended to be helpful (?) and the T option should be a
one-off at setup time.
>
>
> # ionice -c3 ~/mythTScut.sh /path/to/infile R 1 NC
>
> Umm again what are we trying to achieve

This run seems to be needed to get rid of streams that ffmpeg doesn't
understand in UK HD broadcasts. And when I have gone on to the next
stage without it the files have got much bigger. I think it
concatenated more than it was intended to....
>
> # It's best to establish the cutlist here, or at least to confirm its accuracy.
>
> What does best do and what is not best, what does establish the cutlist here mean.

Use the MythTV recordings editor to set cutpoints at keyframes.
Check that you see what you want when the frontend plays over cuts.
Sometimes if a cutlist has been established for the raw recording it
might be slightly off after the NC run.

The next stage will do the cutting. I suggested the V option so that
the result will be a video and the original recording will still be
available. If you're happy with it you could mv or cp the file but it's
probably easier to do it again with R 1
>
> # ionice -c3 ~/mythTScut.sh /path/to/infile V 1
> # scan for new videos from the mythfrontend 'Watch Videos' screen
> # ionice -c3 mythutil --rebuild --video /path/to/the/video/outfile
>
> Again confused about what is happening

So, I'm afraid, am I. But it's getting clearer.

This process obviously demands some effort and you might not want to do
it often. As it is at present any user will get to see a lot of
diagnostics (and suggestions to fix your code to do it properly.)

I find the editor pretty quick, particularly since I'm only working at
the keyframe level (and for my SD recordings with mythDVBcut I have
found that entirely acceptable.) The actual processing is much faster
than real time.

At present playback by the frontend handles some cuts in
recordings-with-cutlists more smoothly than those in the files produced
by the script - but UPnP playback works for me and the commercials have
gone.
>
> I’m not trying in any way to dis your work, I just dont see how to use it
> Thanks so

I hope it will do what you want. It will be obvious that I'm not a bash
pro, and I'm still trying to get to grips with small differences between
what myth does and what ffmpeg expects,,,

John

> James
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org