Mailing List Archive

Slow deletes were causing backend to hang and/or cause seg faults
This is continuation of the topic found here:

http://mythtv.org/pipermail/mythtv-users/2008-December/242458.html

I thought that a new title would be beneficial to clarify the cause of my problems and maybe help others.

My system information is:
Linux MythTV2 2.6.27.5-41.fc9.x86_64 #1 SMP Thu Nov 13 20:29:07 EST 2008
x86_64 x86_64 x86_64 GNU/Linux
Dual core CPU E2180 @ 2.00GHz with 3.9GiB

My video partition is a RAID 5 with an ext3 filesystem.

I discovered within the last couple of days that the cause of my problems outlined in the original post (backend hanging/seg faults) were due to "slow delete" being turned on. Once I turned slow deletes off my problems went away and my system became stable.

Long story short, I become convinced after logging using the "-v all" option for both the frontend and backend. While I have not looked into the actual code, I gathered from the log that once you mark a program for slow delete it gets placed into a database for in-use files, and then gets taken out and put back into it again (multiple times) for some reason. Then the log just outputs the following sequence forever:

2008-12-30 23:49:54.111 MythEvent: SCHEDULE_CHANGE
2008-12-30 23:49:54.145 write -> 16 45 BACKEND_MESSAGE[]:[]SCHEDULE_CHANGE[]:[]empty
2008-12-30 23:49:54.150 write -> 21 45 BACKEND_MESSAGE[]:[]

I'm suspecting an infinite loop with some condition never being satisfied. Maybe the system is waiting for a confirmation that the slow delete is finished? The file does actually get deleted from my filesystem, however.

I have edited logs if anyone is interested in looking at them. The original log was 370MB (just overnight)!

-Kenan







_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
Kenan Ezal wrote:
> This is continuation of the topic found here:
>
> http://mythtv.org/pipermail/mythtv-users/2008-December/242458.html
>
> I thought that a new title would be beneficial to clarify the cause of my
> problems and maybe help others.
>
> My system information is: Linux MythTV2 2.6.27.5-41.fc9.x86_64 #1 SMP Thu Nov
> 13 20:29:07 EST 2008 x86_64 x86_64 x86_64 GNU/Linux Dual core CPU E2180 @
> 2.00GHz with 3.9GiB
>
> My video partition is a RAID 5 with an ext3 filesystem.
>
> I discovered within the last couple of days that the cause of my problems
> outlined in the original post (backend hanging/seg faults) were due to "slow
> delete" being turned on. Once I turned slow deletes off my problems went away
> and my system became stable.
>
> Long story short, I become convinced after logging using the "-v all" option
> for both the frontend and backend. While I have not looked into the actual
> code, I gathered from the log that once you mark a program for slow delete it
> gets placed into a database for in-use files, and then gets taken out and put
> back into it again (multiple times) for some reason. Then the log just
> outputs the following sequence forever:
>
> 2008-12-30 23:49:54.111 MythEvent: SCHEDULE_CHANGE 2008-12-30 23:49:54.145
> write -> 16 45 BACKEND_MESSAGE[]:[]SCHEDULE_CHANGE[]:[]empty 2008-12-30
> 23:49:54.150 write -> 21 45 BACKEND_MESSAGE[]:[]
>
> I'm suspecting an infinite loop with some condition never being satisfied.
> Maybe the system is waiting for a confirmation that the slow delete is
> finished? The file does actually get deleted from my filesystem, however.
>
> I have edited logs if anyone is interested in looking at them. The original
> log was 370MB (just overnight)!
>
As a data point, I'm using ext3 (not raid though) with slow deletes turned on
and have not had any problems with my setup.

--

Mike Perkins

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
Kenan Ezal wrote:
> I discovered within the last couple of days that the cause of my
> problems outlined in the original post (backend hanging/seg faults)
> were due to "slow delete" being turned on. Once I turned slow deletes
> off my problems went away and my system became stable.

Exactly what version of mythtv are you using?

Can you uncomment these lines:

//VERBOSE(VB_FILE, QString("Truncating '%1' to %2 MB")
// .arg(filename).arg(fsize / (1024.0 * 1024.0), 0, 'f', 2));

in MainServer::TruncateAndClose() in mainserver.cpp and try again with
-v file (at least) and see how the numbers printed relate to the moment
of crash.

Bolek
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
On Sun, 2009-01-04 at 23:07 -0500, Boleslaw Ciesielski wrote:

> Exactly what version of mythtv are you using?
>

Hello,

MythTV Version : 19114M
MythTV Branch : branches/release-0-21-fixes
Library API : 0.21.20080304-1
Network Protocol : 40
Options compiled in:
linux release using_oss using_alsa using_arts using_jack using_backend
using_dbox2 using_dvb using_firewire using_frontend using_hdhomerun
using_iptv using_ivtv using_joystick_menu using_libfftw3 using_lirc
using_opengl_vsync using_opengl_video using_v4l using_x11 using_xrandr
using_xv using_xvmc using_xvmcw using_xvmc_vld using_glx_proc_addr_arb
using_bindings_perl using_bindings_python using_opengl
using_ffmpeg_threads using_libavc_5_3 using_live


> Can you uncomment these lines:
>
> //VERBOSE(VB_FILE, QString("Truncating '%1' to %2 MB")
> // .arg(filename).arg(fsize / (1024.0 * 1024.0), 0, 'f', 2));
>
> in MainServer::TruncateAndClose() in mainserver.cpp and try again
> with
> -v file (at least) and see how the numbers printed relate to the
> moment
> of crash.
>
> Bolek

Unfortunately, I'm using the packages available from atrmps. Is there
anything else I can do to help figure the root of the problem?

Someone previously suggested I load some debug packages (for Fedora 9).
I never got around to it but I would be willing to do so if it would
help.

-Kenan

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
On 01/05/2009 03:04 AM, Kenan Ezal wrote:
> Unfortunately, I'm using the packages available from atrmps. Is there
> anything else I can do to help figure the root of the problem?
>
> Someone previously suggested I load some debug packages (for Fedora 9).
> I never got around to it but I would be willing to do so if it would
> help.

That + http://mythtv.org/docs/mythtv-HOWTO-22.html#ss22.2 (use the debug
package rather than rebuilding with --compile-type=debug) would be
/very/ helpful and say exactly where the crash is occurring.

Thanks,
Mike
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
---- "Michael T. Dean" <mtdean@thirdcontact.com> wrote:
> That + http://mythtv.org/docs/mythtv-HOWTO-22.html#ss22.2 (use the debug
> package rather than rebuilding with --compile-type=debug) would be
> /very/ helpful and say exactly where the crash is occurring.
>
> Thanks,
> Mike
>

Will do.

However, I have a couple of deadlines at work that will require my attention for the next week 24/7, so it will have to wait for a short while. In the mean time I will start recording some shows I don't care about so that I can delete them at will. You should be hearing from me in about 10 days or less.

-Kenan
_
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
Michael Dean>>
That + http://mythtv.org/docs/mythtv-HOWTO-22.html#ss22.2 (use the debug
package rather than rebuilding with --compile-type=debug) would be
/very/ helpful and say exactly where the crash is occurring.

Thanks,
Mike
<<Michael Dean

I've posted the gdb output under the developers list:

http://mythtv.org/pipermail/mythtv-dev/2009-January/064156.html

There are still quite a few unknown symbols so I hope I did everything
correctly.

-Kenan


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
On 01/08/2009 03:13 PM, Kenan Ezal wrote:
> Michael Dean:
>> That + http://mythtv.org/docs/mythtv-HOWTO-22.html#ss22.2 (use the debug
>> package rather than rebuilding with --compile-type=debug) would be
>> /very/ helpful and say exactly where the crash is occurring.
> I've posted the gdb output under the developers list:
>
> http://mythtv.org/pipermail/mythtv-dev/2009-January/064156.html
>
> There are still quite a few unknown symbols so I hope I did everything
> correctly.

Yeah, it looks like a good backtrace. The unknown symbols are in
non-Myth libs. I haven't gotten a chance to look through it, yet, but
someone should before long.

I noticed that you interrupted it (presumably when it seemed to lock
up)--you didn't actually get a segfault. Was the processor maxed out
(or relatively idle) when you interrupted it?

Any chance you could run with debug symbols for a while longer to try to
catch a segfault?

Thanks for taking the time to get the backtrace.

Mike
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
On Thu, 2009-01-08 at 16:43 -0500, Michael T. Dean wrote:
> On 01/08/2009 03:13 PM, Kenan Ezal wrote:
> > Michael Dean:
> >> That + http://mythtv.org/docs/mythtv-HOWTO-22.html#ss22.2 (use the debug
> >> package rather than rebuilding with --compile-type=debug) would be
> >> /very/ helpful and say exactly where the crash is occurring.
> > I've posted the gdb output under the developers list:
> >
> > http://mythtv.org/pipermail/mythtv-dev/2009-January/064156.html
> >
> > There are still quite a few unknown symbols so I hope I did everything
> > correctly.
>
> Yeah, it looks like a good backtrace. The unknown symbols are in
> non-Myth libs. I haven't gotten a chance to look through it, yet, but
> someone should before long.
>
> I noticed that you interrupted it (presumably when it seemed to lock
> up)--you didn't actually get a segfault. Was the processor maxed out
> (or relatively idle) when you interrupted it?
>

Yes, I hit Control-C as soon as the backend/frontend was non-responsive.
I didn't actually check the CPU load, but my guess would be relatively
idle. I'll check next time I do it.

> Any chance you could run with debug symbols for a while longer to try to
> catch a segfault?
>

Sure. I still have the debug package installed. I just have to turn slow
deletes back on. Unfortunately, it doesn't always result in a seg fault.
I am puzzled, however, because yesterday afternoon I deleted four shows
that I had recorded for this purpose (but that I didn't watch) and
nothing happened. Last night I watched two shows and deleted each one
and both times it crapped out. I only sent the first backtrace but I
have the second one as well.

> Thanks for taking the time to get the backtrace.
>
> Mike

No problem. I wish I could help more but I haven't done too much coding
lately. Seems like I'm just pushing paper around these days.

-Kenan

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
> > Michael Dean:
> I noticed that you interrupted it (presumably when it seemed to lock
> up)--you didn't actually get a segfault. Was the processor maxed out
> (or relatively idle) when you interrupted it?
>


Kenan Ezal >>
Yes, I hit Control-C as soon as the backend/frontend was non-responsive.
I didn't actually check the CPU load, but my guess would be relatively
idle. I'll check next time I do it.
<< Kenan Ezal

Michael Dean >>
> Any chance you could run with debug symbols for a while longer to try to
> catch a segfault?
>
<< Michael Dean

Kenan Ezal >>
Sure. I still have the debug package installed. I just have to turn slow
deletes back on. Unfortunately, it doesn't always result in a seg fault.
I am puzzled, however, because yesterday afternoon I deleted four shows
that I had recorded for this purpose (but that I didn't watch) and
nothing happened. Last night I watched two shows and deleted each one
and both times it crapped out. I only sent the first backtrace but I
have the second one as well.
<< Kenan Ezal

I've been crashing/hanging the system for a week now without a seg fault.
It just hangs. The front-end can't talk to the backend, but the backend is
still there. Before I loaded the debug package it would cause a seg fault
every once in a while. The CPU load is low.

The only other difference I can think of is that I used to run mythbackend
as a daemon:

# /sbin/service mythbackend start

Now I'm running gdb as mythtv user. Otherwise I'm not doing anything
differently, but no seg faults. Any ideas?

Based on the logs I'm pretty sure that somehow the backend is confused.
Every time it hangs I get the message that it couldn't find the record that
it just deleted.

-Kenan



_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
On 01/16/2009 02:35 AM, Kenan Ezal wrote:
> I've been crashing/hanging the system for a week now without a seg fault.
> It just hangs. The front-end can't talk to the backend, but the backend is
> still there.

http://mythtv.org/pipermail/mythtv-dev/2009-February/064565.html

The issue was likely fixed by Chris Pinkham with [19960] in -fixes and
[19962] in trunk. Please let us know if you still have the issue after
updating beyond the appropriate version (i.e. r19960 in -fixes).

Thanks,
Mike
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Slow deletes were causing backend to hang and/or cause seg faults [ In reply to ]
-----Original Message-----
>From: Michael T. Dean [mailto:mtdean@thirdcontact.com]
>Sent: Monday, February 09, 2009 2:51 PM
>To: Discussion about mythtv
>Subject: Re: [mythtv-users] Slow deletes were causing backend to hang
and/or cause seg faults

>On 01/16/2009 02:35 AM, Kenan Ezal wrote:
>> I've been crashing/hanging the system for a week now without a seg
fault.
>> It just hangs. The front-end can't talk to the backend, but the backend
is
>> still there.

>http://mythtv.org/pipermail/mythtv-dev/2009-February/064565.html

>The issue was likely fixed by Chris Pinkham with [19960] in -fixes and
>[19962] in trunk. Please let us know if you still have the issue after
>updating beyond the appropriate version (i.e. r19960 in -fixes).

>Thanks,
>Mike

I was so busy that I only recently turned slow deletes back on. This fix
appears to have solved my problem.

Thank you!

-Kenan


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users