Mailing List Archive

Commflag-related IOBOUND errors
Brandon Beattie <brandon+myth@linuxis.us> says:
> Also just so it's known, you can have the best system, raid 10, and
> still get corrupted video. There are a few things that can cause
> this. The update scheduled recordings task is nasty in Myth... and
> Mysql can be nasty too. I see (Weekly) a single recording lose data
> when it spawns this task.

Rescheduling's always seemed pretty lightweight on my setup, but I
*do* see intermittent IOBOUNDs; when they occur, they appear every
five minutes on the dot for 3-8 seconds (:10:02-05, :20:02-08,
:25:02-06, etc.), especially when I'm recording dual HD streams and
especially especially when also watching a third stream, using gigabit
Ethernet and a RAID 1 NAS. I am pretty sure this is related to my
frontend/backend checking for user/commflag jobs every five minutes;
before I changed this from the default setting of every minute, I'd
see the IOBOUNDs pop up in the exact same way (you guessed it) every
minute. (Note, also, that I see the IOBOUNDs even if I'm outside the
2:30-6am time block I've told the backend to run commflag jobs in.)
It's quite possible that this is a swap-related issue and that adding
more RAM to my 512MB system would take care of this, but as a
cheapskate I'm reluctant to do so because I haven't run into any other
memory-related issues I know of.

--
Yeechang Lee <ylee@pobox.com> | +1 650 776 7763 | San Francisco CA US
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Commflag-related IOBOUND errors [ In reply to ]
On Tue, Jan 17, 2006 at 04:52:02PM -0800, Yeechang Lee wrote:
> Rescheduling's always seemed pretty lightweight on my setup, but I
> *do* see intermittent IOBOUNDs; when they occur, they appear every
> five minutes on the dot for 3-8 seconds (:10:02-05, :20:02-08,
> :25:02-06, etc.), especially when I'm recording dual HD streams and
> especially especially when also watching a third stream, using gigabit
> Ethernet and a RAID 1 NAS. I am pretty sure this is related to my
> frontend/backend checking for user/commflag jobs every five minutes;
> before I changed this from the default setting of every minute, I'd
> see the IOBOUNDs pop up in the exact same way (you guessed it) every
> minute. (Note, also, that I see the IOBOUNDs even if I'm outside the
> 2:30-6am time block I've told the backend to run commflag jobs in.)
> It's quite possible that this is a swap-related issue and that adding
> more RAM to my 512MB system would take care of this, but as a
> cheapskate I'm reluctant to do so because I haven't run into any other
> memory-related issues I know of.

Every time a recording, transcode, commercial detection, and so on start
or finish the recordings table is updated. If you use EIT monitoring
with a HD tuner than every time this finishes it can cause the
recording scheduler to be re-run. Adding RAM will not help at all.
This is purely a system and disk IO issue. I have my mysql databases on
a seperate drive than recordings, but I still get IO bound issues once
in a while. The thread for rescheduling does take all the CPU and IO it
can get for those few seconds, and with the mysql and disk writing
threads they don't always work well together. It's a known problem but
only shows up for HD users more so because HD requires such a large
amount of bandwidth and it can't really stand a hickup. Also, the PCI
bus can be occupied and cause data from the card to be lost if the
system IO can't be freed up quick enough to handle the data. HD tuner
cards have a rather small cache on them. If anything occupies the
system IO longer than 500ms then you risk losing data from the HD tuner,
and Myth and the scheduler does this at times.

It's also not just disk access, if you use a remote frontend you'll
notice network traffic (Depending on the motherboards and ethernet
chipset) will also stutter during the time the schedular runs.

The best way to fix this? Keep the drive for the database on a
different disk than one used for recordings. Use XFS or JFS. Make sure
nothing else IO intensive is running when the schedular runs. Intel
based systems have less of a problem than AMD from what I've seen (Intel
systems with hyper threading that is). Reiser is a good choice though
for the filesystem holding the database, since it does a better job with
sub GB size files.

--Brandon
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Commflag-related IOBOUND errors [ In reply to ]
Brandon Beattie <brandon+myth@linuxis.us> says:
> Every time a recording, transcode, commercial detection, and so on start
> or finish the recordings table is updated.

The IOBOUND errors I mentioned that are occurring at five-minute
multiples (and one-minute multiples before I changed the "check every
x seconds" setting in the backend from the default) occur independent
of any recording starts or stops. As previously mentioned I have set
commflag jobs to only occur in the early morning, and I never do
transcode.

> If you use EIT monitoring with a HD tuner than every time this
> finishes it can cause the recording scheduler to be re-run.

Sorry, don't know what EIT monitoring is; I use FireWire with my HD
cable boxes.

> Adding RAM will not help at all. This is purely a system and disk IO
> issue.

This both gladdens my wallet while dimming my hopes for
always-pristine (subject to the vagaries of cable provider
compression) HD recordings regardless of circumstance.

> Also, the PCI bus can be occupied and cause data from the card to be
> lost if the system IO can't be freed up quick enough to handle the
> data. HD tuner cards have a rather small cache on them. If
> anything occupies the system IO longer than 500ms then you risk
> losing data from the HD tuner, and Myth and the scheduler does this
> at times.

Hmm. If it's not a swapping issue, perhaps I should turn the
ringbuffer size on the frontend back up to the maximum? I thought that
only applied to Live TV. Is there a separate ringbuffer setting I am
not aware of in 0.18.1? (Live TV is set to record locally, but I
wonder if maybe it's better to trade network bandwidth for disk IO
bandwidth and set it to go to the same place the recordings go to?)

> The best way to fix this? Keep the drive for the database on a
> different disk than one used for recordings.

Done. I keep the database on a local SATA drive, but the recordings go
to an Infrant 600 2TB over gigabit Ethernet . . .

> Use XFS or JFS.

. . . and which uses ext3, not JFS (as I use pretty much everywhere
else), as its filesystem. (Although the Infrant folks use
straightforward md-compatible RAID on the 600 [and even their
proprietary X-RAID on the X6 is md-readable], I think they've tweaked
with the ext3 filesystem code a bit. When I delete a recording, the
system gives me back control as quickly as when I recorded to my JFS-based
RAID array, but only for a split second; then there is a distinct
'hang' for a few moments as the deletions occur. And yes, this does
cause a few bursts of IOBOUNDs when I am doing HDTV recording at the
time.)

> Make sure nothing else IO intensive is running when the schedular
> runs.

That's easy; the MythTV box doesn't do anything else but MythTV, and
the Infrant is going to be dedicated solely to Myth (at least once
I've moved my remaining non-Myth video collection off it).

> Intel based systems have less of a problem than AMD from what I've
> seen (Intel systems with hyper threading that is).

Check that; the MythTV box is a 3.0GHz Pentium 4.

> Reiser is a good choice though for the filesystem holding the
> database, since it does a better job with sub GB size files. =

Like I said, I am using JFS pretty much everywhere else, and that
includes the MythTV box's local disk.

I wonder if this is a latency issue? Unfortuantely I can't tweak up
the latency on my PCI Express gigabit Ethernet card with setpci; I've
tried.

--
Yeechang Lee <ylee@pobox.com> | +1 650 776 7763 | San Francisco CA US
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Commflag-related IOBOUND errors [ In reply to ]
On Wed, Jan 18, 2006 at 09:43:37AM -0800, Yeechang Lee wrote:
> Brandon Beattie <brandon+myth@linuxis.us> says:
> > Every time a recording, transcode, commercial detection, and so on start
> > or finish the recordings table is updated.
>
> The IOBOUND errors I mentioned that are occurring at five-minute
> multiples (and one-minute multiples before I changed the "check every
> x seconds" setting in the backend from the default) occur independent
> of any recording starts or stops. As previously mentioned I have set
> commflag jobs to only occur in the early morning, and I never do
> transcode.

That's fine, myth still will perform schedule recording updates from
time to time for a number of reasons. Disk space auto expiring,
commercial detection start/finish, etc.

> > If you use EIT monitoring with a HD tuner than every time this
> > finishes it can cause the recording scheduler to be re-run.
>
> Sorry, don't know what EIT monitoring is; I use FireWire with my HD
> cable boxes.

It's what scans for program guide info on HD channels.

> > Adding RAM will not help at all. This is purely a system and disk IO
> > issue.
>
> This both gladdens my wallet while dimming my hopes for
> always-pristine (subject to the vagaries of cable provider
> compression) HD recordings regardless of circumstance.

No, you just need to outsmart the IO problems on your system. It's not
about what you throw at it, but how.

> > Also, the PCI bus can be occupied and cause data from the card to be
> > lost if the system IO can't be freed up quick enough to handle the
> > data. HD tuner cards have a rather small cache on them. If
> > anything occupies the system IO longer than 500ms then you risk
> > losing data from the HD tuner, and Myth and the scheduler does this
> > at times.
>
> Hmm. If it's not a swapping issue, perhaps I should turn the
> ringbuffer size on the frontend back up to the maximum? I thought that
> only applied to Live TV. Is there a separate ringbuffer setting I am
> not aware of in 0.18.1? (Live TV is set to record locally, but I
> wonder if maybe it's better to trade network bandwidth for disk IO
> bandwidth and set it to go to the same place the recordings go to?)

Each HD tuner card can have a custom ring buffer size, this has nothing
to do with the live-tv buffer. I'm not sure about firewire input though.

> > The best way to fix this? Keep the drive for the database on a
> > different disk than one used for recordings.
>
> Done. I keep the database on a local SATA drive, but the recordings go
> to an Infrant 600 2TB over gigabit Ethernet . . .

May try some different NFS options or look into how your ethernet card
buffers data.

> > Use XFS or JFS.
>
> . . . and which uses ext3, not JFS (as I use pretty much everywhere
> else), as its filesystem. (Although the Infrant folks use
> straightforward md-compatible RAID on the 600 [.and even their
> proprietary X-RAID on the X6 is md-readable], I think they've tweaked
> with the ext3 filesystem code a bit. When I delete a recording, the
> system gives me back control as quickly as when I recorded to my JFS-based
> RAID array, but only for a split second; then there is a distinct
> 'hang' for a few moments as the deletions occur. And yes, this does
> cause a few bursts of IOBOUNDs when I am doing HDTV recording at the
> time.)

Odd, I'd expect this to not be a problem if you're using a remote file
server. This may help myth devs though in fixing the problem. If
you're getting IO Bounds on write, and you're using a remote filesystem
then it's backing up on ethernet/NFS. I don't have a reponse, just a
bunch of hmms for this.

> > Make sure nothing else IO intensive is running when the schedular
> > runs.
>
> That's easy; the MythTV box doesn't do anything else but MythTV, and
> the Infrant is going to be dedicated solely to Myth (at least once
> I've moved my remaining non-Myth video collection off it).

cron jobs could cause problems, depending on which distro you run.
Nightly updatedb's or other tasks too.

> > Intel based systems have less of a problem than AMD from what I've
> > seen (Intel systems with hyper threading that is).
>
> Check that; the MythTV box is a 3.0GHz Pentium 4.

Hmm...

> > Reiser is a good choice though for the filesystem holding the
> > database, since it does a better job with sub GB size files. =
>
> Like I said, I am using JFS pretty much everywhere else, and that
> includes the MythTV box's local disk.
>
> I wonder if this is a latency issue? Unfortuantely I can't tweak up
> the latency on my PCI Express gigabit Ethernet card with setpci; I've
> tried.

It's an IO issue. There's something in Myth that is not playing nice
with system IO. It's brought out when scheduling takes place usually.
Sometimes a large delete or a massive write can cause this. It only
happens if the Myth can't get IO for writing for half a second. As it
happens right now, this problem does occur. There's a bug in the bug
tracking system that you can add addition info to, to help locate and
solve this.

I opened the bug when losing 5-15 minutes per hour of recording. After
ditching reiserFS (I was using it to test as for the two years before I
used XFS/JFS) and additional work from Daniel the loss of data went to
about 5 seconds per 100 hours of HD recording. I currently can record 4
HD shows and watch one without any problems. Since it's been working
fine the bug was closed. I'd recommend re-opening that bug and add in
the info you've found, and/or try some of the profiling and debugging
options mentioned in that bug to help locate what the exact problem is
for you.

--Brandon
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Commflag-related IOBOUND errors [ In reply to ]
Brandon Beattie wrote:

>This is purely a system and disk IO issue. I have my mysql databases on
>a seperate drive than recordings, but I still get IO bound issues once
>in a while. The thread for rescheduling does take all the CPU and IO it
>can get for those few seconds, and with the mysql and disk writing
>threads they don't always work well together. It's a known problem but
>only shows up for HD users more so because HD requires such a large
>amount of bandwidth and it can't really stand a hickup. Also, the PCI
>bus can be occupied and cause data from the card to be lost if the
>system IO can't be freed up quick enough to handle the data. HD tuner
>cards have a rather small cache on them. If anything occupies the
>system IO longer than 500ms then you risk losing data from the HD tuner,
>and Myth and the scheduler does this at times.
>
>
This seems to imply that the mysql database might best be placed on a
separate SCSI drive with a large cache? Any thoughts?

Geoff
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Commflag-related IOBOUND errors [ In reply to ]
On Thu, Jan 19, 2006 at 02:30:41PM -0500, R. G. Newbury wrote:
> This seems to imply that the mysql database might best be placed on a
> separate SCSI drive with a large cache? Any thoughts?

Absolutely.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users