Mailing List Archive

Storage groups: How to set 'SGweightLocalStarting' to 0?
I've got 3 BE's each with 2 storage drives. They are all connected to a nice gigabit switch and use NFS and mount points so that all the BE's have access to all recording dirs. All the storage drives are added to the default SG.

I would like myth to balance the recordings across all the drives based on free space rather than 'local' vs. 'remote' storage, so I tried stopping mythBE, opening SQL as root, and executing the following command:

INSERT settings (value, data, hostname) VALUES ("SGweightLocalStarting ", 0, "");

This put the entry into settings, but it doesn't seem to produce the desired effect: that local and remote storage would be judged on an equal footing.

I tested by manually scheduling 2 simultaneous recordings. They should have gone into drives 5&6 based on free space, but they still were placed in the local storage, drives 1&2 (which have the least free space).

Can anyone give me a hand with this?

Jon
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
On Wed, Oct 8, 2008 at 9:27 AM, Jonathan Larson
<jtlarson@u.washington.edu>wrote:

> I've got 3 BE's each with 2 storage drives. They are all connected to a
> nice gigabit switch and use NFS and mount points so that all the BE's have
> access to all recording dirs. All the storage drives are added to the
> default SG.
>
>
>
> I would like myth to balance the recordings across all the drives based on
> free space rather than 'local' vs. 'remote' storage, so I tried stopping
> mythBE, opening SQL as root, and executing the following command:
>
>
>
> INSERT settings (value, data, hostname) VALUES ("SGweightLocalStarting ",
> 0, "");
>

Per the instructions on the wiki, you need to set a hostname when using this
setting

http://www.mythtv.org/wiki/index.php/Storage_Groups_Weighting

Kevin
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
>
>
> So do I need to put in 3 entries—one for each BE, or 6—one for each storage
> dir?
>

Probably one for each backend. Settings are not storage group specific but
host specific.

>
>
> I guess the problem for me is that the post you linked to addresses
> 'SGweightPerDir' not 'SGweightLocalStarting', and I'm just trying to figure
> out which parts translate from one to the other…
>
The principle is the same. Most settings in the DB (if you browse through
it) have a hostname component because they are per host settings, not
global.

Kevin
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
On Wed, Oct 8, 2008 at 12:11 PM, Jonathan Larson
<jtlarson@u.washington.edu>wrote:

>
>
> So do I need to put in 3 entries—one for each BE, or 6—one for each storage
> dir?
>
>
> Probably one for each backend. Settings are not storage group specific but
> host specific.
>
>
>
> I guess the problem for me is that the post you linked to addresses
> 'SGweightPerDir' not 'SGweightLocalStarting', and I'm just trying to figure
> out which parts translate from one to the other…
>
> The principle is the same. Most settings in the DB (if you browse
> through it) have a hostname component because they are per host settings,
> not global.
>
>
> Kevin
>
>
>
>
>
> So I tried entering the settings but got the same result as before—remote
> storage with more free space is still ignored. Here's the entries I made in
> settings:
>

Try running the backend with extra verbosity turned on for VB_FILE and/or
VB_SCHEDULE and you should see a bunch of stuff printed out about the
weighting used.

Kevin
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
On 10/08/2008 04:50 PM, Jonathan Larson wrote:
>> On 10/08/2008 01:24 PM, Kevin Kuphal wrote:
>>
>>> Try running the backend with extra verbosity turned on for VB_FILE and/or
>>> VB_SCHEDULE and you should see a bunch of stuff printed out about the
>>> weighting used.
>>>
>> And note that you'll need to restart Myth (master backend, meaning all
>> other stuff, too) when hand-editing the database.
>>
>> Still, though, you should fix the settings so it's one with NULL
>> hostname.
>>
> Based on Mike's advice, I reduced my three SQL entries to 1:
>
> mysql> select * from settings where value = "SGweightLocalStarting";
> +-----------------------+------+----------+
> | value | data | hostname |
> +-----------------------+------+----------+
> | SGweightLocalStarting | 0 | |
> +-----------------------+------+----------+
> 1 row in set (0.00 sec)
>

And you're sure that hostname is actually NULL and not the empty string?

SELECT * FROM settings WHERE value IS NULL;

Some (esp GUI) clients don't handle NULL well.

> I also turned on verbose logging for file and schedule events and found this:
>
> 2008-10-08 13:30:41.043 Scheduler: FillRecordingDir: Calculating initial FS Weights.
> 2008-10-08 13:30:41.044 AVPC1:/mnt/AVPC1-rec1/recordings is local (-19). initial dir weight = -19
> 2008-10-08 13:30:41.045 AVPC1:/mnt/AVPC1-rec2/recordings is local (-19). initial dir weight = -19
> 2008-10-08 13:30:41.047 AVPC1:/mnt/AVPC2-rec1/recordings is remote (+0). initial dir weight = 0
> 2008-10-08 13:30:41.048 AVPC1:/mnt/AVPC2-rec2/recordings is remote (+0). initial dir weight = 0
> 2008-10-08 13:30:41.050 AVPC1:/mnt/AVPC3-rec1/recordings is remote (+0). initial dir weight = 0
> 2008-10-08 13:30:41.051 AVPC1:/mnt/AVPC3-rec2/recordings is remote (+0). initial dir weight = 0
>
> Notice that the local drives are still set at -19 even with the "SGweightLocalStarting" entry in place....
>
> Any advice would be appreciated. I can include more of the log as an attachment if that would be useful.

If you've got the hostname set right, something is causing it to not
pick up the value. Typically that will only be an issue when updating
the DB while Myth is running. Did you shut down all Myth frontends and
backends, then update the DB, then restart and test?

Mike
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
On Oct 8, 2008, at 2:29 PM, Michael T. Dean wrote:

>> Based on Mike's advice, I reduced my three SQL entries to 1:
>>
>> mysql> select * from settings where value = "SGweightLocalStarting";
>> +-----------------------+------+----------+
>> | value | data | hostname |
>> +-----------------------+------+----------+
>> | SGweightLocalStarting | 0 | |
>> +-----------------------+------+----------+
>> 1 row in set (0.00 sec)
>>
>
> And you're sure that hostname is actually NULL and not the empty
> string?

>
>
> SELECT * FROM settings WHERE value IS NULL;
>
> Some (esp GUI) clients don't handle NULL well.

For future reference, mysql client will display NULL when the value is
null. The above indicates there is an empty value which is not the
same as NULL.

For example:
mysql> select * from settings where hostname IS NULL LIMIT 1;
+------------------------------+------------------+----------+
| value | data | hostname |
+------------------------------+------------------+----------+
| mythfilldatabaseLastRunStart | 2008-10-08 12:53 | NULL |
+------------------------------+------------------+----------+
1 row in set (0.00 sec)

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
> On Oct 8, 2008, at 2:43 PM, Jonathan Larson wrote:
>
> >> I'm new to the SQL CLI, so I didn't even realize there was a
> >> functional difference between "" and NULL. Should the insert
> >> command look like this?:
> >
> > INSERT settings (value, data, hostname) VALUES
> > ("SGweightLocalStarting", 0, NULL);
>
> Since you already have an entry in your database for
> SGweightLocalStarting your command should now look like this:
>
> UPDATE settings SET hostname = NULL WHERE value =
> 'SGweightLocalStarting';

Thanks Brad,

I was just asking about the initial INSERT command because I wanted to add this to the wiki and I wanted to make sure it would be accurate.

Jon
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
On Oct 8, 2008, at 3:07 PM, Jonathan Larson wrote:

>> On Oct 8, 2008, at 2:43 PM, Jonathan Larson wrote:
>>
>>>> I'm new to the SQL CLI, so I didn't even realize there was a
>>>> functional difference between "" and NULL. Should the insert
>>>> command look like this?:
>>>
>>> INSERT settings (value, data, hostname) VALUES
>>> ("SGweightLocalStarting", 0, NULL);
>>
>> Since you already have an entry in your database for
>> SGweightLocalStarting your command should now look like this:
>>
>> UPDATE settings SET hostname = NULL WHERE value =
>> 'SGweightLocalStarting';
>
> Thanks Brad,
>
> I was just asking about the initial INSERT command because I wanted
> to add this to the wiki and I wanted to make sure it would be
> accurate.

Gotcha... yeah, that's right if there is no current entry. Personally,
I'm a fan of REPLACE INTO which will act as an INSERT of the primary
key doesn't exist and an UPDATE if it does. I think this only works in
mysql, though so I can see why SQL purists would frown on it.

Bottom line is yeah, that command is correct. ;)

-Brad

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
> -----Original Message-----
> From: mythtv-users-bounces@mythtv.org [mailto:mythtv-users-
> bounces@mythtv.org] On Behalf Of Brad DerManouelian
> Sent: Wednesday, October 08, 2008 3:15 PM
> To: Discussion about mythtv
> Subject: Re: [mythtv-users] Storage groups: How to set
> 'SGweightLocalStarting' to 0?
>
> On Oct 8, 2008, at 3:07 PM, Jonathan Larson wrote:
>
> >> On Oct 8, 2008, at 2:43 PM, Jonathan Larson wrote:
> >>
> >>>> I'm new to the SQL CLI, so I didn't even realize there was a
> >>>> functional difference between "" and NULL. Should the insert
> >>>> command look like this?:
> >>>
> >>> INSERT settings (value, data, hostname) VALUES
> >>> ("SGweightLocalStarting", 0, NULL);
> >>
> >> Since you already have an entry in your database for
> >> SGweightLocalStarting your command should now look like this:
> >>
> >> UPDATE settings SET hostname = NULL WHERE value =
> >> 'SGweightLocalStarting';
> >
> > Thanks Brad,
> >
> > I was just asking about the initial INSERT command because I wanted
> > to add this to the wiki and I wanted to make sure it would be
> > accurate.
>
> Gotcha... yeah, that's right if there is no current entry. Personally,
> I'm a fan of REPLACE INTO which will act as an INSERT of the primary
> key doesn't exist and an UPDATE if it does. I think this only works in
> mysql, though so I can see why SQL purists would frown on it.
>
> Bottom line is yeah, that command is correct. ;)
>
> -Brad
>

So I stopped all the BEs and Fes and added the entry mentioned above to the settings table. Then I restarted the main BE and ran a test recording.

What I found is that the Null entry did reset the local disk weighting to 0, making it equal to the remote disks. In spite of this, the system still chose to record to one of the local disks with less free space than some of the remote disks. This behavior would seem to be contrary to Chris's documentation, though I'm not sure why that happened. Any more insight from the experts?

Jon

Log exerpt:

--- GetFilesystemInfos directory list end ---
2008-10-09 06:44:29.410 Scheduler: FillDirectoryInfoCache: found 6 unique filesystems
2008-10-09 06:44:29.410 Scheduler: FillRecordingDir: Calculating initial FS Weights.
2008-10-09 06:44:29.411 AVPC1:/mnt/AVPC1-rec1/recordings is local (0). initial dir weight = 0
2008-10-09 06:44:29.412 AVPC1:/mnt/AVPC1-rec2/recordings is local (0). initial dir weight = 0
2008-10-09 06:44:29.413 AVPC1:/mnt/AVPC2-rec1/recordings is remote (+0). initial dir weight = 0
2008-10-09 06:44:29.414 AVPC1:/mnt/AVPC2-rec2/recordings is remote (+0). initial dir weight = 0
2008-10-09 06:44:29.415 AVPC1:/mnt/AVPC3-rec1/recordings is remote (+0). initial dir weight = 0
2008-10-09 06:44:29.417 AVPC1:/mnt/AVPC3-rec2/recordings is remote (+0). initial dir weight = 0
2008-10-09 06:44:29.417 Scheduler: FillRecordingDir: Adjusting FS Weights from inuseprograms.
2008-10-09 06:44:29.418 Scheduler: FillRecordingDir: Adjusting FS Weights from scheduler.
--- FillRecordingDir Sorted fsInfoList start ---
AVPC1:/mnt/AVPC1-rec2/recordings
Location : local
weight : 0
free space : 70778984

AVPC1:/mnt/AVPC1-rec1/recordings
Location : local
weight : 0
free space : 45571944

AVPC1:/mnt/AVPC3-rec2/recordings
Location : remote
weight : 0
free space : 158221328

AVPC1:/mnt/AVPC2-rec1/recordings
Location : remote
weight : 0
free space : 114929048

AVPC1:/mnt/AVPC2-rec2/recordings
Location : remote
weight : 0
free space : 110237936

AVPC1:/mnt/AVPC3-rec1/recordings
Location : remote
weight : 0
free space : 46420320

--- FillRecordingDir Sorted fsInfoList end ---
2008-10-09 06:44:29.430 'SGtest' will record in '/mnt/AVPC1-rec2/recordings' which has 69120 MiB free. This recording could use a max of 72 MiB and the AutoExpirer wants to keep 10240 MiB free.


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
Jonathan Larson <jtlarson@u.washington.edu> says:
> What I found is that the Null entry did reset the local disk
> weighting to 0, making it equal to the remote disks. In spite of
> this, the system still chose to record to one of the local disks
> with less free space than some of the remote disks. This behavior
> would seem to be contrary to Chris's documentation, though I'm not
> sure why that happened. Any more insight from the experts?

I presume you've seen the thread starting at
<URL:http://www.gossamer-threads.com/lists/mythtv/users/346903#346903>
so you know how I stand on the issue.

I believe what happens is something like the following:

* Disk A has 30GB free, but 100GB of expirable/deleted recordings
* Disk B has 35GB free, but 5GB of expirable/deleted recordings
* MythTV is told to keep 30GB minimum free on all disks
* MythTV uses B for a new 10GB recording, as it has more free space
* Once free space on B gets below 30GB, it expires the 5GB of
expirable recordings, even if the 5GB was recorded just two hours
ago

Check out
<URL:http://www.gossamer-threads.com/lists/mythtv/dev/351309#351309>.
I have been running the patch for a week now, but believe the real key
is to have the new pass 2 run before (or instead of) the existing pass
1, not after.

--
Frontend: P4 3.0GHz, 1.5TB software RAID 5 array
Backend: Quad-core Xeon 1.6GHz, 6.6TB sw RAID 6
Video inputs: Four high-definition over FireWire/OTA
Accessories: 47" 1080p LCD, 5.1 digital, and MX-600
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
Re: Storage groups: How to set 'SGweightLocalStarting' to 0? [ In reply to ]
>
> > -----Original Message-----
> > From: mythtv-users-bounces@mythtv.org [mailto:mythtv-users-
> > bounces@mythtv.org] On Behalf Of Brad DerManouelian
> > Sent: Wednesday, October 08, 2008 3:15 PM
> > To: Discussion about mythtv
> > Subject: Re: [mythtv-users] Storage groups: How to set
> > 'SGweightLocalStarting' to 0?
> >
> > On Oct 8, 2008, at 3:07 PM, Jonathan Larson wrote:
> >
> > >> On Oct 8, 2008, at 2:43 PM, Jonathan Larson wrote:
> > >>
> > >>>> I'm new to the SQL CLI, so I didn't even realize there was a
> > >>>> functional difference between "" and NULL. Should the insert
> > >>>> command look like this?:
> > >>>
> > >>> INSERT settings (value, data, hostname) VALUES
> > >>> ("SGweightLocalStarting", 0, NULL);
> > >>
> > >> Since you already have an entry in your database for
> > >> SGweightLocalStarting your command should now look like this:
> > >>
> > >> UPDATE settings SET hostname = NULL WHERE value =
> > >> 'SGweightLocalStarting';
> > >
> > > Thanks Brad,
> > >
> > > I was just asking about the initial INSERT command because I wanted
> > > to add this to the wiki and I wanted to make sure it would be
> > > accurate.
> >
> > Gotcha... yeah, that's right if there is no current entry.
> Personally,
> > I'm a fan of REPLACE INTO which will act as an INSERT of the primary
> > key doesn't exist and an UPDATE if it does. I think this only works
> in
> > mysql, though so I can see why SQL purists would frown on it.
> >
> > Bottom line is yeah, that command is correct. ;)
> >
> > -Brad
> >
>
> So I stopped all the BEs and Fes and added the entry mentioned above to
> the settings table. Then I restarted the main BE and ran a test
> recording.
>
> What I found is that the Null entry did reset the local disk weighting
> to 0, making it equal to the remote disks. In spite of this, the system
> still chose to record to one of the local disks with less free space
> than some of the remote disks. This behavior would seem to be contrary
> to Chris's documentation, though I'm not sure why that happened. Any
> more insight from the experts?
>
> Jon
>
> Log exerpt:
>
> --- GetFilesystemInfos directory list end ---
> 2008-10-09 06:44:29.410 Scheduler: FillDirectoryInfoCache: found 6
> unique filesystems
> 2008-10-09 06:44:29.410 Scheduler: FillRecordingDir: Calculating
> initial FS Weights.
> 2008-10-09 06:44:29.411 AVPC1:/mnt/AVPC1-rec1/recordings is local
> (0). initial dir weight = 0
> 2008-10-09 06:44:29.412 AVPC1:/mnt/AVPC1-rec2/recordings is local
> (0). initial dir weight = 0
> 2008-10-09 06:44:29.413 AVPC1:/mnt/AVPC2-rec1/recordings is remote
> (+0). initial dir weight = 0
> 2008-10-09 06:44:29.414 AVPC1:/mnt/AVPC2-rec2/recordings is remote
> (+0). initial dir weight = 0
> 2008-10-09 06:44:29.415 AVPC1:/mnt/AVPC3-rec1/recordings is remote
> (+0). initial dir weight = 0
> 2008-10-09 06:44:29.417 AVPC1:/mnt/AVPC3-rec2/recordings is remote
> (+0). initial dir weight = 0
> 2008-10-09 06:44:29.417 Scheduler: FillRecordingDir: Adjusting FS
> Weights from inuseprograms.
> 2008-10-09 06:44:29.418 Scheduler: FillRecordingDir: Adjusting FS
> Weights from scheduler.
> --- FillRecordingDir Sorted fsInfoList start ---
> AVPC1:/mnt/AVPC1-rec2/recordings
> Location : local
> weight : 0
> free space : 70778984
>
> AVPC1:/mnt/AVPC1-rec1/recordings
> Location : local
> weight : 0
> free space : 45571944
>
> AVPC1:/mnt/AVPC3-rec2/recordings
> Location : remote
> weight : 0
> free space : 158221328
>
> AVPC1:/mnt/AVPC2-rec1/recordings
> Location : remote
> weight : 0
> free space : 114929048
>
> AVPC1:/mnt/AVPC2-rec2/recordings
> Location : remote
> weight : 0
> free space : 110237936
>
> AVPC1:/mnt/AVPC3-rec1/recordings
> Location : remote
> weight : 0
> free space : 46420320
>
> --- FillRecordingDir Sorted fsInfoList end ---
> 2008-10-09 06:44:29.430 'SGtest' will record in '/mnt/AVPC1-
> rec2/recordings' which has 69120 MiB free. This recording could use a
> max of 72 MiB and the AutoExpirer wants to keep 10240 MiB free.
>

Update: I've just been letting the local drives fill up to see what would happen when they reached my storage limit preset. Now that my first local drive is full (and the second is one recording away), myth is sending recordings to the remote drives that are the least full.

Chris Peterson's post (http://www.mythtv.org/wiki/index.php/Storage_Groups_Weighting) states that "In a tie, the drive with
the highest amount of free disk space is used first"

Based on my experiences, it should be revised as follows: "In a tie, local disks with free space will be used first (1 recording per disk), followed by remote disks with the most free space."

Jon
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users