Mailing List Archive

fix for IOBOUND - blocking in ThreadedFileWriter::Write
Hi,

I'm running myth 0.18 release version on a 2.6.12.1 kernel. My card is
HD3000 HDTV tuner card. I've one 180GB hard drive in my setup,
standard knoppix myth partition. /, /root, /cache and /myth , are all
ext 3.

Whenever the tuner is recording a program, and I try to copy a large
file, I get the following error message in the backend logs.

2005-07-06 16:07:00.217 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-06 16:07:05.419 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-06 16:07:09.360 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-06 16:07:17.400 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-06 16:07:26.623 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-06 16:07:31.437 IOBOUND - blocking in ThreadedFileWriter::Write()

If I stop the copy, the message will also stop.

When mythcommflag runs on a recorded file with these error messages,
it also complains about ac-tex damage, etc etc.

[mpeg2video @ 0xb7676010]ac-tex damaged at 22 53
[mpeg2video @ 0xb7676010]Warning MVs not available
[mpeg2video @ 0xb7676010]slice mismatch
[mpeg2video @ 0xb7676010]00 motion_type at 107 38
[mpeg2video @ 0xb7676010]00 motion_type at 43 39

The video has lots of stuttering and it skips badly.

I searched through the archives and found out that the
TFW_DEF_BUF_SIZE (in libs/libmythtv/RingBuffer.cpp) definition changed
to a higher number fixes the issue. The default value is 2MB. I
changed it to 8MB and recompiled the libmythtv library.

Then I restarted the mythbackend, start to record one program and also
started to copy a large file. This time, I just got one error message,
probably at the very beginning of the copy. Then I started another
copy, and just got two messages, which is not as bad compared to one
message every couple of seconds.

2005-07-06 17:38:13.106 DVB#0 Successfully tuned to channel 20_1.
2005-07-06 17:38:13.120 adding: mythtvbe as a client (events: 0)
2005-07-06 17:38:13.841 Changing from None to RecordingOnly
2005-07-06 17:38:16.303 New DB connection, total: 4
2005-07-06 17:38:16.625 DVB#0 Recorder: Card opened successfully
(using TS mode).
2005-07-06 17:38:17.075 DVB#0 Data read from DMX - This is for
debugging with transform.c
2005-07-06 17:38:19.654 MainServer::HandleAnnounce Playback
2005-07-06 17:38:20.688 adding: mythtvbe as a client (events: 0)
2005-07-06 17:40:30.398 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-06 17:45:06.337 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-06 17:47:40.387 IOBOUND - blocking in ThreadedFileWriter::Write()


I think 2MB buffer may be fine for SD stuff, but for HD stuff, we need
a bigger buffer. I'll experiment with a 12MB and 16MB buffer also. I
did check the 0.18.1 code and the SVN code and the value is still 2MB.

Is it possible to set this option in configuration or pass an option via CLI ?

Thanks

Mudit
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev
Re: fix for IOBOUND - blocking in ThreadedFileWriter::Write [ In reply to ]
On Wednesday 06 July 2005 08:55 pm, Mudit Wahal wrote:
> I searched through the archives and found out that the
> TFW_DEF_BUF_SIZE (in libs/libmythtv/RingBuffer.cpp) definition changed
> to a higher number fixes the issue. The default value is 2MB. I
> changed it to 8MB and recompiled the libmythtv library.

Instead of increasing the buffer size, a better fix would be to move the call
to TFW::Sync() to a separate thread.

Isaac
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev
Re: fix for IOBOUND - blocking in ThreadedFileWriter::Write [ In reply to ]
Mudit Wahal wrote:

>Hi,
>
>I'm running myth 0.18 release version on a 2.6.12.1 kernel. My card is
>HD3000 HDTV tuner card. I've one 180GB hard drive in my setup,
>standard knoppix myth partition. /, /root, /cache and /myth , are all
>ext 3.
>
>Whenever the tuner is recording a program, and I try to copy a large
>file, I get the following error message in the backend logs.
>
>
>
>
ext3 which isn't the best choice storing and moving around large files.
Using the write filesystem for the job would probably clear your problem
up without the need to make any changes to Myth.

>I think 2MB buffer may be fine for SD stuff, but for HD stuff, we need
>a bigger buffer. I'll experiment with a 12MB and 16MB buffer also. I
>did check the 0.18.1 code and the SVN code and the value is still 2MB.
>
>
>
I routinely copy over multi GB files while HD streams are recording
without issue.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev
Re: fix for IOBOUND - blocking in ThreadedFileWriter::Write [ In reply to ]
so which file system do you recommend ? xfs ? reiser ?

thanks
mudit


On 7/6/05, J. Donavan Stanley <jdonavan@jdonavan.net> wrote:
> Mudit Wahal wrote:
>
> >Hi,
> >
> >I'm running myth 0.18 release version on a 2.6.12.1 kernel. My card is
> >HD3000 HDTV tuner card. I've one 180GB hard drive in my setup,
> >standard knoppix myth partition. /, /root, /cache and /myth , are all
> >ext 3.
> >
> >Whenever the tuner is recording a program, and I try to copy a large
> >file, I get the following error message in the backend logs.
> >
> >
> >
> >
> ext3 which isn't the best choice storing and moving around large files.
> Using the write filesystem for the job would probably clear your problem
> up without the need to make any changes to Myth.
>
> >I think 2MB buffer may be fine for SD stuff, but for HD stuff, we need
> >a bigger buffer. I'll experiment with a 12MB and 16MB buffer also. I
> >did check the 0.18.1 code and the SVN code and the value is still 2MB.
> >
> >
> >
> I routinely copy over multi GB files while HD streams are recording
> without issue.
>
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev
Re: fix for IOBOUND - blocking in ThreadedFileWriter::Write [ In reply to ]
On Wed, 2005-07-06 at 22:22 -0400, J. Donavan Stanley wrote:
> Mudit Wahal wrote:
>
> >Hi,
> >
> >I'm running myth 0.18 release version on a 2.6.12.1 kernel. My card is
> >HD3000 HDTV tuner card. I've one 180GB hard drive in my setup,
> >standard knoppix myth partition. /, /root, /cache and /myth , are all
> >ext 3.
> >
> >Whenever the tuner is recording a program, and I try to copy a large
> >file, I get the following error message in the backend logs.
> >
> >
> >
> >
> ext3 which isn't the best choice storing and moving around large files.
> Using the write filesystem for the job would probably clear your problem
> up without the need to make any changes to Myth.
>

Are you having a bad sentence structure and spelling day? ;-)
What FS do you use? I'm using XFS and on the whole don't have many
problems.

Brian
Re: fix for IOBOUND - blocking in ThreadedFileWriter::Write [ In reply to ]
On Wed, Jul 06, 2005 at 10:27:32PM -0700, Mudit Wahal wrote:
> so which file system do you recommend ? xfs ? reiser ?

This has been covered rather thoroughly in the mythtv documentation,
with links to the mailing list archives for performance numbers. XFS,
Reiser, and JFS are all good.

> On 7/6/05, J. Donavan Stanley <jdonavan@jdonavan.net> wrote:
> > Mudit Wahal wrote:
> >
> > >Hi,
> > >
> > >I'm running myth 0.18 release version on a 2.6.12.1 kernel. My card is
> > >HD3000 HDTV tuner card. I've one 180GB hard drive in my setup,
> > >standard knoppix myth partition. /, /root, /cache and /myth , are all
> > >ext 3.
> > >
> > >Whenever the tuner is recording a program, and I try to copy a large
> > >file, I get the following error message in the backend logs.
> > >
> > >
> > >
> > >
> > ext3 which isn't the best choice storing and moving around large files.
> > Using the write filesystem for the job would probably clear your problem
> > up without the need to make any changes to Myth.
> >
> > >I think 2MB buffer may be fine for SD stuff, but for HD stuff, we need
> > >a bigger buffer. I'll experiment with a 12MB and 16MB buffer also. I
> > >did check the 0.18.1 code and the SVN code and the value is still 2MB.
> > >
> > >
> > >
> > I routinely copy over multi GB files while HD streams are recording
> > without issue.

--Rob
Re: fix for IOBOUND - blocking in ThreadedFileWriter::Write [ In reply to ]
I'll see if I can convert my file system to xfs etc. I'd like to know
if the IOBound issue can be fixed in the code. 2MB buffer is
sufficient for low bit rate mpeg, but for high bit rate mpeg such as
HD (19.2mps), the buffer has to be increased. Unless the logic changes
as suggested by Isaac. I've searched the archives for the file system
and also found that this issue IOBound does come up once in a while.

Thanks
Mudit

On 7/7/05, Robert Tsai <rtsai1111@comcast.net> wrote:
> On Wed, Jul 06, 2005 at 10:27:32PM -0700, Mudit Wahal wrote:
> > so which file system do you recommend ? xfs ? reiser ?
>
> This has been covered rather thoroughly in the mythtv documentation,
> with links to the mailing list archives for performance numbers. XFS,
> Reiser, and JFS are all good.
>
> > On 7/6/05, J. Donavan Stanley <jdonavan@jdonavan.net> wrote:
> > > Mudit Wahal wrote:
> > >
> > > >Hi,
> > > >
> > > >I'm running myth 0.18 release version on a 2.6.12.1 kernel. My card is
> > > >HD3000 HDTV tuner card. I've one 180GB hard drive in my setup,
> > > >standard knoppix myth partition. /, /root, /cache and /myth , are all
> > > >ext 3.
> > > >
> > > >Whenever the tuner is recording a program, and I try to copy a large
> > > >file, I get the following error message in the backend logs.
> > > >
> > > >
> > > >
> > > >
> > > ext3 which isn't the best choice storing and moving around large files.
> > > Using the write filesystem for the job would probably clear your problem
> > > up without the need to make any changes to Myth.
> > >
> > > >I think 2MB buffer may be fine for SD stuff, but for HD stuff, we need
> > > >a bigger buffer. I'll experiment with a 12MB and 16MB buffer also. I
> > > >did check the 0.18.1 code and the SVN code and the value is still 2MB.
> > > >
> > > >
> > > >
> > > I routinely copy over multi GB files while HD streams are recording
> > > without issue.
>
> --Rob
>
>
> BodyID:151316654.2.n.logpart (stored separately)
>
>
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev
Re: fix for IOBOUND - blocking in ThreadedFileWriter::Write [ In reply to ]
On Jul 7, 2005, at 9:38 AM, Mudit Wahal wrote:

> I'll see if I can convert my file system to xfs etc. I'd like to know
> if the IOBound issue can be fixed in the code. 2MB buffer is
> sufficient for low bit rate mpeg, but for high bit rate mpeg such as
> HD (19.2mps), the buffer has to be increased. Unless the logic changes
> as suggested by Isaac. I've searched the archives for the file system
> and also found that this issue IOBound does come up once in a while.
>
I believe I have also seen this issue in the Threaded File Reader
(when using a remote frontend over a congested network). Would it
also be possible to increase the read buffer?


Geoff Kruse
RE: fix for IOBOUND - blocking in ThreadedFileWriter::Write [ In reply to ]
mythtv-dev-bounces@mythtv.org <> wrote:
> Hi,
>
> I'm running myth 0.18 release version on a 2.6.12.1 kernel. My card is
> HD3000 HDTV tuner card. I've one 180GB hard drive in my setup,
> standard knoppix myth partition. /, /root, /cache and /myth , are all
> ext 3.
>
> Whenever the tuner is recording a program, and I try to copy a large
> file, I get the following error message in the backend logs.
>
> 2005-07-06 16:07:00.217 IOBOUND - blocking in
> ThreadedFileWriter::Write()
> 2005-07-06 16:07:05.419 IOBOUND - blocking in
> ThreadedFileWriter::Write()
> 2005-07-06 16:07:09.360 IOBOUND - blocking in
> ThreadedFileWriter::Write()
> 2005-07-06 16:07:17.400 IOBOUND - blocking in
> ThreadedFileWriter::Write()
> 2005-07-06 16:07:26.623 IOBOUND - blocking in
> ThreadedFileWriter::Write()
> 2005-07-06 16:07:31.437 IOBOUND - blocking in
> ThreadedFileWriter::Write()
>
> If I stop the copy, the message will also stop.
>

One thing you may want to try is to change the io scheduler to cfq (
elevator=cfq argument to kernel at boot )
That should allow each program to get equal access to the disk so one one is
starved

I think cfq is the default for fedora.

These 2 links may be of intrest:
http://lwn.net/Articles/22429/
http://www.redhat.com/magazine/008jun05/features/schedulers/


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev
Re: fix for IOBOUND - blocking in ThreadedFileWriter::Write [ In reply to ]
I changed the scheduler to cfq, and changed back to default
libmythtv0.18 (2MB buffer).
I'm getting
2005-07-07 17:22:13.294 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-07 17:22:15.468 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-07 17:22:19.489 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-07 17:22:23.995 IOBOUND - blocking in ThreadedFileWriter::Write()
2005-07-07 17:22:29.346 IOBOUND - blocking in ThreadedFileWriter::Write()

So, the cfq scheduler doesn't help at all. I'll revert back to 16MB
buffer library. May be if i find time in the weekend, I'll change my
/myth file system to jfs and try it out again with 2mb library.

Thanks

Mudit

On 7/7/05, Torbjörn Jansson <torbjorn.jansson@mbox200.swipnet.se> wrote:
> mythtv-dev-bounces@mythtv.org <> wrote:
> > Hi,
> >
> > I'm running myth 0.18 release version on a 2.6.12.1 kernel. My card is
> > HD3000 HDTV tuner card. I've one 180GB hard drive in my setup,
> > standard knoppix myth partition. /, /root, /cache and /myth , are all
> > ext 3.
> >
> > Whenever the tuner is recording a program, and I try to copy a large
> > file, I get the following error message in the backend logs.
> >
> > 2005-07-06 16:07:00.217 IOBOUND - blocking in
> > ThreadedFileWriter::Write()
> > 2005-07-06 16:07:05.419 IOBOUND - blocking in
> > ThreadedFileWriter::Write()
> > 2005-07-06 16:07:09.360 IOBOUND - blocking in
> > ThreadedFileWriter::Write()
> > 2005-07-06 16:07:17.400 IOBOUND - blocking in
> > ThreadedFileWriter::Write()
> > 2005-07-06 16:07:26.623 IOBOUND - blocking in
> > ThreadedFileWriter::Write()
> > 2005-07-06 16:07:31.437 IOBOUND - blocking in
> > ThreadedFileWriter::Write()
> >
> > If I stop the copy, the message will also stop.
> >
>
> One thing you may want to try is to change the io scheduler to cfq (
> elevator=cfq argument to kernel at boot )
> That should allow each program to get equal access to the disk so one one is
> starved
>
> I think cfq is the default for fedora.
>
> These 2 links may be of intrest:
> http://lwn.net/Articles/22429/
> http://www.redhat.com/magazine/008jun05/features/schedulers/
>
>
>
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev