Mailing List Archive

Fragmentation On Recording Disk?
I'd recently been running into performance issues with my combined
frontend/backend. Menu's were slow, hard disk access was slow, and in
general it felt sluggish. When the system was new it did not have these
problems. Recordings and videos are stored on a pair of 250 gig IDE drives,
running in a striped LVM configuration with XFS. They're on different IDE
channels as well, so performance really shouldn't be an issue.

I decided to check the fragmentation on the drive, and xfs_db told me that
it was over 98%. I deleted some recordings that we'd already watched to free
up a bunch of space (114 gigs free space, actually), and ran xfs_fsr in an
infinite loop for a couple of days. I got fragmentation under 3%, and the
system is much improved. I've copied over some new video files and some
shows have been recorded, so fragmentation has increased from that low
again, but is still under 20%. My room mate reports that the system is "3000
percent better." While that's probably a bit of an exaggeration, it is
certainly much more responsive, and the hard disks don't seem to thrash so
much when loading the recordings listing. I've set up a cron script to run
one pass of xfs_fsr each night, so hopefully we can keep the performance
gain for a little while.

root@sixball:~# xfs_db -r /dev/mythtv/mythmedia
xfs_db> frag
actual 3904, ideal 3191, fragmentation factor 18.26%
xfs_db> quit

I just thought I'd share this with the list, and ask if anyone else has
encountered significant performance loss due to disk fragmentation? How
fragmented are your drives?
Re: Fragmentation On Recording Disk? [ In reply to ]
On 3 Apr 2007, at 22:10, Michael MacLeod wrote:

> I just thought I'd share this with the list, and ask if anyone else
> has encountered significant performance loss due to disk
> fragmentation? How fragmented are your drives?

# xfs_db -r /dev/sda6
xfs_db> frag
Segmentation fault

Hmmm.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
Michael MacLeod wrote:
>
> root@sixball:~# xfs_db -r /dev/mythtv/mythmedia
> xfs_db> frag
> actual 3904, ideal 3191, fragmentation factor 18.26%
> xfs_db> quit
>
> I just thought I'd share this with the list, and ask if anyone else has
> encountered significant performance loss due to disk fragmentation? How
> fragmented are your drives?

mythbox ~ # xfs_db -r /dev/mapper/vg-mythtv
xfs_db> frag
actual 60667, ideal 67, fragmentation factor 99.89%
xfs_db>


Hmm, that looks pretty bad.

Robert.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
On 3 Apr 2007, at 22:30, Robert wrote:

> Michael MacLeod wrote:
>>
>> root@sixball:~# xfs_db -r /dev/mythtv/mythmedia
>> xfs_db> frag
>> actual 3904, ideal 3191, fragmentation factor 18.26%
>> xfs_db> quit
>>
>> I just thought I'd share this with the list, and ask if anyone
>> else has
>> encountered significant performance loss due to disk
>> fragmentation? How
>> fragmented are your drives?
>
> mythbox ~ # xfs_db -r /dev/mapper/vg-mythtv
> xfs_db> frag
> actual 60667, ideal 67, fragmentation factor 99.89%
> xfs_db>
>
>
> Hmm, that looks pretty bad.
>
> Robert.

# /usr/local/bin/xfs_db -r -c frag /dev/sda6
actual 940233, ideal 585, fragmentation factor 99.94%

I'd say that's worse!

FWIW, looks like xfs_db that ships with xfsprogs.x86_64 (2.7.3-1.2.1)
is b0rked - mine segfaults. Fix: download source packages for
xfsprogs and xfsdump from http://oss.sgi.com/projects/xfs/
download.html and configure/make/make install
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
fryfrog@dumbledore:~$ sudo xfs_db -r -c frag /dev/md2
actual 63966, ideal 54729, fragmentation factor 14.44%

I'll be honest, I have known about xfs_fsr ever since I noticed that
some of my recordings would delete instantly and some were taking 5+
seconds to delete. It felt like I was back on ext3, I turned on slow
deletes and it took me ages to finally figure out it wasn't a random
problem. The fast to delete files were lightly fragmented, the slow
deleting ones were not.

On 4/3/07, Ben Lancaster <lists@benlancaster.co.uk> wrote:
>
> On 3 Apr 2007, at 22:30, Robert wrote:
>
> > Michael MacLeod wrote:
> >>
> >> root@sixball:~# xfs_db -r /dev/mythtv/mythmedia
> >> xfs_db> frag
> >> actual 3904, ideal 3191, fragmentation factor 18.26%
> >> xfs_db> quit
> >>
> >> I just thought I'd share this with the list, and ask if anyone
> >> else has
> >> encountered significant performance loss due to disk
> >> fragmentation? How
> >> fragmented are your drives?
> >
> > mythbox ~ # xfs_db -r /dev/mapper/vg-mythtv
> > xfs_db> frag
> > actual 60667, ideal 67, fragmentation factor 99.89%
> > xfs_db>
> >
> >
> > Hmm, that looks pretty bad.
> >
> > Robert.
>
> # /usr/local/bin/xfs_db -r -c frag /dev/sda6
> actual 940233, ideal 585, fragmentation factor 99.94%
>
> I'd say that's worse!
>
> FWIW, looks like xfs_db that ships with xfsprogs.x86_64 (2.7.3-1.2.1)
> is b0rked - mine segfaults. Fix: download source packages for
> xfsprogs and xfsdump from http://oss.sgi.com/projects/xfs/
> download.html and configure/make/make install
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>


--
AIM: fryfrog
ICQ: 304825
WWW: http://www.fryfrog.com
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
wow! I need to check this tonight!
It's funny, I thought disk frag was not an issue with linux! guess I was wrong ...
Seriously, are these figures reasonable ? I mean, can one trust what xfs_db spits out ? I never considered checking disk frag on any of my linux boxes, and I've had them for quite a while. Are disks formatted with ext3 showing the same fragmentation factor ? And does one know the growth rate of this factor ? (what are the relevant variables : disk size ? technology ? file system ?)

J.

Ben Lancaster <lists@benlancaster.co.uk> wrote:
On 3 Apr 2007, at 22:30, Robert wrote:

> Michael MacLeod wrote:
>>
>> root@sixball:~# xfs_db -r /dev/mythtv/mythmedia
>> xfs_db> frag
>> actual 3904, ideal 3191, fragmentation factor 18.26%
>> xfs_db> quit
>>
>> I just thought I'd share this with the list, and ask if anyone
>> else has
>> encountered significant performance loss due to disk
>> fragmentation? How
>> fragmented are your drives?
>
> mythbox ~ # xfs_db -r /dev/mapper/vg-mythtv
> xfs_db> frag
> actual 60667, ideal 67, fragmentation factor 99.89%
> xfs_db>
>
>
> Hmm, that looks pretty bad.
>
> Robert.

# /usr/local/bin/xfs_db -r -c frag /dev/sda6
actual 940233, ideal 585, fragmentation factor 99.94%

I'd say that's worse!

FWIW, looks like xfs_db that ships with xfsprogs.x86_64 (2.7.3-1.2.1)
is b0rked - mine segfaults. Fix: download source packages for
xfsprogs and xfsdump from http://oss.sgi.com/projects/xfs/
download.html and configure/make/make install
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users



---------------------------------
No need to miss a message. Get email on-the-go
with Yahoo! Mail for Mobile. Get started.
Re: Fragmentation On Recording Disk? [ In reply to ]
On Wednesday 4 April 2007 8:31 am, James Warden wrote:
> Are disks formatted with
> ext3 showing the same fragmentation factor ? And does one know the growth
> rate of this factor ? (what are the relevant variables : disk size ?
> technology ? file system ?)
>
Using filefrag on my current disk:
/myth/1028_20070223002400.mpg: 2977 extents found, perfection would be 14
extents
/myth/1028_20070322002900.mpg: 1473 extents found, perfection would be 16
extents
/myth/1032_20070220003400.mpg: 2566 extents found, perfection would be 11
extents

I'm in the middle of upgrading my storage and here's what I get on the new
disk:
/mnt/myth/1028_20070223002400.mpg: 30 extents found, perfection would be 14
extents
/mnt/myth/1028_20070322002900.mpg: 33 extents found, perfection would be 16
extents
/mnt/myth/1032_20070220003400.mpg: 22 extents found, perfection would be 11
extents

Both disks are ext3, the files on the current disk were written by myth and
then lossless transcoded and the filesystem is about a year old and MythTV
runs with 1GB min free space. The new disk files were just copied with rsync
and there's acres of space.

MythTV must be one of the worst applications for producing fragmented files.
It fills up the disk and then continues to delete older files and write new
ones while keeping the disk very close to full. The files are all different
sizes so there's no practical file system on earth that can stop
fragmentation happening. I can't think of a better way to stress test
filesystem fragmentation.

If it's a problem then I guess you should keep more disk space free so at
least there's a chance for new recordings to find some contiguous space.
How much free space is enough? Maybe 20 times the largest file? That's a lot
of programmes to lose just in the hope of reducing fragmentation. I haven't
noticed a problem with streaming but this is SD only with a separate BE/FE.

Anyone know of a defragger for ext3/2?

Cheers,
Tim.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
On Wednesday 4 April 2007 9:36 am, I wrote:
> MythTV must be one of the worst applications for producing fragmented
> files. It fills up the disk and then continues to delete older files and
> write new ones while keeping the disk very close to full. The files are all
> different sizes so there's no practical file system on earth that can stop
> fragmentation happening. I can't think of a better way to stress test
> filesystem fragmentation.
>
Here's a better way to frag a disk (from the Debian bug #401622 that is
keeping defrag out of Etch):
==
But 9% non continues files isn't bad. If you really get fragmentation
your disk I/O slows down to below 200K/s. I had this on my P2P
partition which was always 90%+ full. The p2p client created sparse
files and then filled in tiny chunks at random positions in all
(incomplete) files. Total death sentence for an fs. I think fsck
reported over 60% fragmented there.
==
So it looks like MythTV is not that bad after all ;-).
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
Tim Phipps wrote:
> On Wednesday 4 April 2007 8:31 am, James Warden wrote:
>
>> Are disks formatted with
>> ext3 showing the same fragmentation factor ? And does one know the growth
>> rate of this factor ? (what are the relevant variables : disk size ?
>> technology ? file system ?)
>>
>>
> Using filefrag on my current disk:
> /myth/1028_20070223002400.mpg: 2977 extents found, perfection would be 14
> extents
> /myth/1028_20070322002900.mpg: 1473 extents found, perfection would be 16
> extents
> /myth/1032_20070220003400.mpg: 2566 extents found, perfection would be 11
> extents
>
> I'm in the middle of upgrading my storage and here's what I get on the new
> disk:
> /mnt/myth/1028_20070223002400.mpg: 30 extents found, perfection would be 14
> extents
> /mnt/myth/1028_20070322002900.mpg: 33 extents found, perfection would be 16
> extents
> /mnt/myth/1032_20070220003400.mpg: 22 extents found, perfection would be 11
> extents
>
>
>
>
I'm also seeing heavily fragmented transcoded files. I'm also using
ext3 on Ubuntu. Any idea if this causes noticable performance problems?

I tend to see livetv pauses when a transcode complete ... when the
original file is deleted. I wonder if this fragmentation could be
contributing.

I just used the default filesystem when installing Ubuntu. If I knew
more about this before starting, I might have tried XFS. It's a pain
to change now. :(

Matt
Re: Fragmentation On Recording Disk? [ In reply to ]
On 4/4/07, Matt Doran <matt.doran@papercut.biz> wrote:
>
>
> Tim Phipps wrote:
> On Wednesday 4 April 2007 8:31 am, James Warden wrote:
>
>
> Are disks formatted with
> ext3 showing the same fragmentation factor ? And does one know the growth
> rate of this factor ? (what are the relevant variables : disk size ?
> technology ? file system ?)

I thought that with ext2/3 fragmentation was kept in check by regular
fscking. I may be wrong, but I've been using linux for a decade and
have never noticed any performace symptoms that would make me think I
needed a defrag.
-J
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
On 04/04/2007 03:31 AM, James Warden wrote:
> wow! I need to check this tonight!
> It's funny, I thought disk frag was not an issue with linux!
AIUI:

Fragmentation is reduced in various ways by modern filesystems (even
NTFS is significantly better than FAT32). Short of running a perpetual
defrag utility (which would cause more performance loss than
fragmentation) or having orders of magnitude more storage than is
required (and a filesystem that's willing to waste it), there's no way
to completely prevent fragmentation.

> guess I was wrong ...
> Seriously, are these figures reasonable ? I mean, can one trust what
> xfs_db spits out ? I never considered checking disk frag on any of my
> linux boxes, and I've had them for quite a while. Are disks formatted
> with ext3 showing the same fragmentation factor ? And does one know
> the growth rate of this factor ? (what are the relevant variables :
> disk size ? technology ? file system ?)

Myth starts a recording at 8:00pm, it begins writing to that file. The
filesystem has allocated a certain amount of space for the file, and--to
reduce fragmentation--has allocated blocks around the initial blocks for
file growth. What the filesystem doesn't know is that Myth will
continue to write to that file for 30 minutes, 1 hour, 2 hours, ... The
filesystem doesn't allocate gigabytes of additional storage for every
file it creates, so any other files created during that recording are
likely to get written where they prevent the recording from being
allocated in one contiguous block.

When you have one capture card and a large amount of contiguous space
available on your storage and nothing else writing to the storage disk,
fragmentation will be minimal. When you have multiple capture
cards--each starting a recording at the same time (or even one after the
other but still overlapping) and writing to the same
disk/partition/volume/whatever, there's going to be significantly more
fragmentation.

However, things like Myth's (to be available in 0.21) Storage Groups
(which allow writing to multiple directories) will help /if/ you
structure your storage appropriately. Shouldn't we fix it more? Well,
short of preallocating the space before a recording, there's not much
more Myth can do.*** This means it's up to users to defrag every once
in a while. My preferred method is to copy files from one disk to
another, make a new filesystem on the original, and copy them back.

Mike

*** If Myth were to preallocate, the only completely reliable way of
doing so is to write zeros to the disk before recording begins. To see
how much of an issue this would be, run top in one terminal and watch
the wa (I/O wait) value near the top of the screen as you run:

dd if=/dev/zero of=~/testlargefilepreallocation bs=1M count=2048
(for an SDTV show at 2GiB/hr)

or

dd if=/dev/zero of=~/testlargefilepreallocation bs=1M count=8192
(for an HDTV show at 8GiB/hr)

in another terminal. Make sure you adjust the path of "of" to some
location that has space (and to which you have permissions).

And, to make things interesting, try doing anything else that touches
the disks while doing this (even just an "ls ~"). If people think that
the MySQL access while recording is an issue, imagine what this would do
to recordings.

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
On 4/4/07, jason maxwell <decepticon@gmail.com> wrote:
> On 4/4/07, Matt Doran <matt.doran@papercut.biz> wrote:
> >
> >
> > Tim Phipps wrote:
> > On Wednesday 4 April 2007 8:31 am, James Warden wrote:
> >
> >
> > Are disks formatted with
> > ext3 showing the same fragmentation factor ? And does one know the growth
> > rate of this factor ? (what are the relevant variables : disk size ?
> > technology ? file system ?)
>
> I thought that with ext2/3 fragmentation was kept in check by regular
> fscking. I may be wrong, but I've been using linux for a decade and
> have never noticed any performace symptoms that would make me think I
> needed a defrag.
> -J
>

Actually, it appears that linux filesystems have been designed to
minimize fragmentation to begin with. Here's a good, simplified
explanation of how it works:
http://geekblog.oneandoneis2.org/index.php/2006/08/17/why_doesn_t_linux_need_defragmenting
I need to learn to search before announcing my ignorance.
-J
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
On Wednesday 04 April 2007 14:24, jason maxwell wrote:
> On 4/4/07, Matt Doran <matt.doran@papercut.biz> wrote:
> > Tim Phipps wrote:
> >
> > Are disks formatted with
> > ext3 showing the same fragmentation factor ? And does one know the growth
> > rate of this factor ? (what are the relevant variables : disk size ?
> > technology ? file system ?)
>
> I thought that with ext2/3 fragmentation was kept in check by regular
> fscking.

No -- or at least, not as far as I know. To the best of my knowledge, fsck
does nothing about fragmented files. My understanding is that ext2/3 doesn't
fragment much because of its design, or rather the design of its Linux
drivers, which try to place files where they can grow, rather than just
cramming them in one after another, as DOS did.

> I may be wrong, but I've been using linux for a decade and
> have never noticed any performace symptoms that would make me think I
> needed a defrag.

Most Linux systems create files that are far smaller than the multi-gigabyte
files that are common with MythTV. If a 5GB recording of a TV show is
fragmented into 500 parts, that's about 10MB per fragment, which is larger
than most files on most non-MythTV Linux systems.

I don't know the exact algorithms used by the different filesystems for where
to place new files. No doubt those would affect fragmentation, and it might
be useful to change, or at least tune, these algorithms for filesystems that
store particularly large files. I've not looked into this matter, though; I
don't know if there are any tuning parameters for ext3fs, XFS, or any other
filesystem that might be useful for MythTV users.

--
Rod Smith
http://www.rodsbooks.com
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
Michael MacLeod wrote:
> root@sixball:~# xfs_db -r /dev/mythtv/mythmedia
> xfs_db> frag
> actual 3904, ideal 3191, fragmentation factor 18.26%
> xfs_db> quit
>
> I just thought I'd share this with the list, and ask if anyone else has
> encountered significant performance loss due to disk fragmentation? How
> fragmented are your drives?

Robert wrote:
> mythbox ~ # xfs_db -r /dev/mapper/vg-mythtv
> xfs_db> frag
> actual 60667, ideal 67, fragmentation factor 99.89%
> xfs_db>
>
> Hmm, that looks pretty bad.


Very interesting, think it would effect JFS also? I can't seem to locate
an equivalent tool for it.

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
On 4/4/07, catfish <catfish@halfdone.com> wrote:
>
> Michael MacLeod wrote:
> > root@sixball:~# xfs_db -r /dev/mythtv/mythmedia
> > xfs_db> frag
> > actual 3904, ideal 3191, fragmentation factor 18.26%
> > xfs_db> quit
> >
> > I just thought I'd share this with the list, and ask if anyone else has
> > encountered significant performance loss due to disk fragmentation? How
> > fragmented are your drives?
>
> Robert wrote:
> > mythbox ~ # xfs_db -r /dev/mapper/vg-mythtv
> > xfs_db> frag
> > actual 60667, ideal 67, fragmentation factor 99.89%
> > xfs_db>
> >
> > Hmm, that looks pretty bad.
>
>
> Very interesting, think it would effect JFS also? I can't seem to locate
> an equivalent tool for it.
>
>
On my 'Jarod Wilson' Fedora box which uses Jfs I can use filefrag:

[root@mythtv video]# filefrag -v /video/9_20060901222500_20060902011000.nuv
Checking /video/9_20060901222500_20060902011000.nuv
Filesystem type is: 3153464a
Filesystem cylinder groups is approximately 89128
Blocksize of file /video/9_20060901222500_20060902011000.nuv is 4096
File size of /video/9_20060901222500_20060902011000.nuv is 7249782490
(1769967 blocks)
First block: 23647088
Last block: 25417064
Discontinuity: Block 761811 is at 24408904 (was 24408898)
Discontinuity: Block 1518731 is at 25165825 (was 25165823)
Discontinuity: Block 1518734 is at 25165832 (was 25165827)
/video/9_20060901222500_20060902011000.nuv: 4 extents found

I interpret this info as 4 continuus blocks for a 7GB file which seems good.
I tested a few other files and all had a small amount of extents. The system
is now running about 16 months and I did not take any defragmentation
actions. I regularly record two shows at a time with a PVR500 and I see no
weird values.

You can also test multiple files like:

[root@mythtv video]# filefrag /video/*
/video/1001_20070403041300.mpg: 5 extents found
/video/1001_20070403041300.mpg.png: 1 extent found
/video/1011_20070401152300.mpg: 3 extents found
/video/1011_20070401152300.mpg.png: 1 extent found
/video/1069_20061111231036.mpg: 1 extent found
/video/1069_20061111231036.mpg.png: 1 extent found
/video/1070_20061111231041.mpg: 1 extent found
/video/1070_20061111231041.mpg.png: 1 extent found
..
..

or with the -v option to get more info

One program I could find is Fibmap from some research done about the
fragmentation in different filesystems, the site seems to be updated in 2004
for the last time....
http://www.informatik.uni-frankfurt.de/~loizides/reiserfs/fibmap.html#description
I did not test the program.

Martijn
Re: Fragmentation On Recording Disk? [ In reply to ]
On 4 Apr 2007, at 19:37, Michael T. Dean wrote:
> Myth starts a recording at 8:00pm, it begins writing to that file.
> The
> filesystem has allocated a certain amount of space for the file,
> and--to
> reduce fragmentation--has allocated blocks around the initial
> blocks for
> file growth. What the filesystem doesn't know is that Myth will
> continue to write to that file for 30 minutes, 1 hour, 2
> hours, ... The
> filesystem doesn't allocate gigabytes of additional storage for every
> file it creates, so any other files created during that recording are
> likely to get written where they prevent the recording from being
> allocated in one contiguous block.
>
> ...
> *** If Myth were to preallocate, the only completely reliable way of
> doing so is to write zeros to the disk before recording begins. To
> see
> how much of an issue this would be, run top in one terminal and watch
> the wa (I/O wait) value near the top of the screen as you run:
>
> dd if=/dev/zero of=~/testlargefilepreallocation bs=1M count=2048
> (for an SDTV show at 2GiB/hr)
>
> or
>
> dd if=/dev/zero of=~/testlargefilepreallocation bs=1M count=8192
> (for an HDTV show at 8GiB/hr)
>
> in another terminal. Make sure you adjust the path of "of" to some
> location that has space (and to which you have permissions).

Here's a Dumb Question (tm). I'm not sure how far I'm straying off-
topic in asking it, but I'm fairly sure you're knowledgeable enough
to answer it easily, so:

Surely in this case the filesystem doesn't know when `dd` starts
writing its file how much it is going to want to write out, so surely
the big file of zeros will likewise end up fragmented into several
parts?

I'd have expected a way to talk to the operating system and tell it
in advance how much space you need (like malloc for memory). If the
program could say in advance "hi there, I'd like to write a file and
it's going to be 2gig in size" (obviously this isn't always possible)
then the filesystem could better manage space to avoid fragmentation.

Stroller.


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
One thing and I don't know if this has been mentioned or not but maybe
cyclical storage could be of use?

It's used in USENET servers
http://en.wikipedia.org/wiki/News_server_operation (look under spools).

The containers would be created first and never grow or shrink so no
fragmentation would occur. New stuff goes in and older stuff falls out
when full. Probably not what one would want but perhaps this would be
something worthwhile for just the recording Mythtv does on the fly
(thinking live tv). One container per recording stream and a defined how
much time to allot to it.

Not sure how well this would work for mpeg as usenet is just text files.
Probably a crazy of the wall thought anyway.

Jason

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
> Surely in this case the filesystem doesn't know when `dd` starts
> writing its file how much it is going to want to write out, so surely
> the big file of zeros will likewise end up fragmented into several
> parts?

The reason the 'dd' would sort of work is that it would *very quickly*
(as fast as the system will let it) allocate all those 0's to the
disk. So yes, it could still be fragmented. But the (relatively)
short time period vs. the duration of a 30,60 or 90 minute show means
it would be less fragmented. But dd would work for the simple reason
that it totally rapes the system while its running, and if you re-nice
it to not rape it... it'll just take longer and longer, resulting in
the same sort of fragmentation.

Its better to just defrag if you need it... or not worry about it if
it isn't causing problems.
--
AIM: fryfrog
ICQ: 304825
WWW: http://www.fryfrog.com
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
* On Thu Apr 05, 2007 at 08:29:57AM -0400, Jason Portwood wrote:
> The containers would be created first and never grow or shrink so no
> fragmentation would occur. New stuff goes in and older stuff falls out
> when full. Probably not what one would want but perhaps this would be
> something worthwhile for just the recording Mythtv does on the fly
> (thinking live tv). One container per recording stream and a defined how
> much time to allot to it.
>
> Not sure how well this would work for mpeg as usenet is just text files.
> Probably a crazy of the wall thought anyway.

Not that crazy, it's almost exactly how we used to do LiveTV. Then we
wanted it to be more flexible and allow saving of LiveTV recordings, etc.
so the LiveTV was rewritten to be just like a normal recording. Since
video may be compressed at different percentages and to different bitrates,
it would be impossible to say "this block is for 5 minutes", you could
only say "this block is 5 Gigs" or something like that.

All in all, it wouldn't work very well for video and we'd end up implementing
our own "filesystem" on top of the normal filesystem and we'd have the same
fragmentation issues or lots of wasted space trying to eliminate fragmentation.

--
Chris
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
On 04/05/2007 08:33 AM, Donald Webster wrote:
>> Surely in this case the filesystem doesn't know when `dd` starts
>> writing its file how much it is going to want to write out, so surely
>> the big file of zeros will likewise end up fragmented into several
>> parts?
>>

I didn't say the dd would prevent fragmentation. From my post, "Well,
short of preallocating the space before a recording, there's not much
more Myth can do." So, preallocating would be more than we're currently
doing to prevent fragmentation, but still not enough to totally
eliminate it. I also said, "Short of running a perpetual defrag utility
(which would cause more performance loss than fragmentation) or having
orders of magnitude more storage than is required (and a filesystem
that's willing to waste it), there's no way to completely prevent
fragmentation."

Of course, the above is basically the opposite order in which they were
presented in my post. Perhaps this order is easier to follow.

> The reason the 'dd' would sort of work is that it would *very quickly*
> (as fast as the system will let it) allocate all those 0's to the
> disk.

Exactly.

> So yes, it could still be fragmented. But the (relatively)
> short time period vs. the duration of a 30,60 or 90 minute show means
> it would be less fragmented. But dd would work for the simple reason
> that it totally rapes the system while its running, and if you re-nice
> it to not rape it... it'll just take longer and longer, resulting in
> the same sort of fragmentation.
>
> Its better to just defrag if you need it... or not worry about it if
> it isn't causing problems.
>

Right on!

Mike

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
Not that I´m an expert on file systems but,
fragmentation on a large file is natural and not a disaster. There is read
ahead and buffers that will make up for fragments, if 4GB is cut up in 64MB
chunks it probably won´t be a problem, when the fragments get really small
performance will go down the drain though.
An ordinary myth box does sequential IO and not that many parallell streams
to/from disk.


/Henrik
Re: Fragmentation On Recording Disk? [ In reply to ]
Rod Smith wrote:

> I don't know the exact algorithms used by the different filesystems for where
> to place new files. No doubt those would affect fragmentation, and it might
> be useful to change, or at least tune, these algorithms for filesystems that
> store particularly large files. I've not looked into this matter, though; I
> don't know if there are any tuning parameters for ext3fs, XFS, or any other
> filesystem that might be useful for MythTV users.

Reducing the inode count for a mythtv partition might be useful. We
don't record 10,000 files in 100G...we record maybe 100 files in 100G.
Changing the inode percentage can recover an appreciable amount of disk
space..not that does *much* about fragmentation, but it does help *a
little*.

Geoff



_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
> Reducing the inode count for a mythtv partition might be useful. We
> don't record 10,000 files in 100G...we record maybe 100 files in 100G.
> Changing the inode percentage can recover an appreciable amount of disk
> space..not that does *much* about fragmentation, but it does help *a
> little*.

So how do you do that? On JFS for example - as I use JFS :)
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
On Friday 06 April 2007 02:18, Phill Edwards wrote:
> > Reducing the inode count for a mythtv partition might be useful. We
> > don't record 10,000 files in 100G...we record maybe 100 files in 100G.
> > Changing the inode percentage can recover an appreciable amount of disk
> > space..not that does *much* about fragmentation, but it does help *a
> > little*.
>
> So how do you do that? On JFS for example - as I use JFS :)

I just checked the jfs_mkfs and jfs_tune man pages and saw no mention of any
parameters for adjusting the inode count on JFS. It's conceivable that it can
be done using undocumented options or other tools, but if so I don't know
what they are.

When using XFS, the mkfs.xfs man page specifies several options, accessed via
the -i switch, that affect inode allocation; however, I haven't taken the
time to study these options and they're complex enough that I wouldn't want
to speculate on how to use them without further study and/or experimentation.

In ext2fs or ext3fs, the number of inodes can be specified with the -i or -N
options. The -i option sets the number of bytes per inode, with larger values
resulting in fewer inodes per filesystem (assuming a fixed filesystem size).
The man page doesn't specify a default value, but examination of my existing
ext2 and ext3 filesystems suggests the default is between 1,600 to 8,000. (I
got different results for different filesystems; perhaps the default varies
with filesystem size.) The -N option sets the number of inodes as an absolute
value.

AFAIK, ReiserFS doesn't use a fixed number of inodes; I believe it allocates
them dynamically as files are created, but I don't know the details. ReiserFS
is also the filesystem that's least recommended for MythTV, judging by the
documentation I've read.

AFAIK, the number of inodes can't be changed once it's set, at least in JFS,
XFS, or ext2fs/ext3fs. If you want to adjust this parameter for the very
small benefit you'll get, you'll need to move data off the partition you want
to tune, reformat it, and then move data back onto it.

--
Rod Smith
http://www.rodsbooks.com
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Fragmentation On Recording Disk? [ In reply to ]
jason maxwell wrote:
> Actually, it appears that linux filesystems have been designed to
> minimize fragmentation to begin with. Here's a good, simplified
> explanation of how it works:
> http://geekblog.oneandoneis2.org/index.php/2006/08/17/why_doesn_t_linux_need_defragmenting
> I need to learn to search before announcing my ignorance.
>

Yes, but note this phrase from that article:

"Fragmentation thus only becomes an issue on Linux when a disk is so
full that there just aren't any gaps a large file can be put into
without splitting it up. So long as the disk is less than about 80%
full, this is unlikely to happen."

Most MythTV users keep their disks pretty full. On a 200 GB video
drive, keeping it less than 80% full would mean wasting 40 gigabytes, or
nearly a day's worth of recordings at the rate my system uses. So, we
let the disk get nearly full and stay there, and the filesystem fragments.

> -J
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

1 2  View All