Mailing List Archive: mythfilldatabase: use of --no-allatonce prevented listings from 8PM on?

mythfilldatabase: use of --no-allatonce prevented listings from 8PM on?

Apr 3, 2024, 11:53 AM

Post #1 of 6 (143 views)

Hi folks,

I've been using MythTV for forever and a half, so some settings/tweaks
that I've done over the years are from before my ability to remember why
I did them.

In my settings table, the MythFillDatabaseArgs is set to
"--no-allatonce". *Why* it is set this way is beyond me, to be honest.
Perhaps to curtail runaway RAM usage?

Anyway, recently I noticed that in the future (starting April 4), my
listings were showing "NO DATA" for the channel starting at 8PM.
Mythweb status showed that there was guide data for 17 days. No other
obvious signs of trouble popped up, other than the fact that I only
coincidentally was looking for a listing that was at 8PM in an upcoming
day where there should definitely have been guide data present.

When I ran mythfilldatabase (with no cmdline arguments), the guide data
eventually populated sucessfully, completely.

My questions at this point are:
- Can anybody think of a reason why I might have set --no-allatonce in
my mythfilldatabase preferences in the first place? (Specifically, I
want to ensure that removing this flag will not have negative consequences)
- Why would --no-allatonce seemingly work just fine all this time, and
starting on April 4, refuse to update listings after 8PM?

Thanks
-WD
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Re: mythfilldatabase: use of --no-allatonce prevented listings from 8PM on? [ In reply to ]

jimmorton01 at gmail

Apr 3, 2024, 12:11 PM

Post #2 of 6 (143 views)

Permalink

I found this nugget in a Gary Buhrmaster post from 1/26/2018

Due to the design of mythfilldatabase (it loads all
the data from the XMLTV grabber into memory before
processing), systems with limited memory may need
to specify --no-allatonce to mythfilldatabase to avoid
excessive swapping (the need will vary depending
on how large many channels your source provides).

HTH,
Jim

On 4/3/2024 2:53 PM, Will Dormann wrote:
> Hi folks,
>
> I've been using MythTV for forever and a half, so some settings/tweaks
> that I've done over the years are from before my ability to remember
> why I did them.
>
> In my settings table, the MythFillDatabaseArgs is set to
> "--no-allatonce". *Why* it is set this way is beyond me, to be
> honest. Perhaps to curtail runaway RAM usage?
>
> Anyway, recently I noticed that in the future (starting April 4), my
> listings were showing "NO DATA" for the channel starting at 8PM.
> Mythweb status showed that there was guide data for 17 days. No other
> obvious signs of trouble popped up, other than the fact that I only
> coincidentally was looking for a listing that was at 8PM in an
> upcoming day where there should definitely have been guide data present.
>
> When I ran mythfilldatabase (with no cmdline arguments), the guide
> data eventually populated sucessfully, completely.
>
> My questions at this point are:
> - Can anybody think of a reason why I might have set --no-allatonce in
> my mythfilldatabase preferences in the first place? (Specifically, I
> want to ensure that removing this flag will not have negative
> consequences)
> - Why would --no-allatonce seemingly work just fine all this time, and
> starting on April 4, refuse to update listings after 8PM?
>
>
> Thanks
> -WD
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Re: mythfilldatabase: use of --no-allatonce prevented listings from 8PM on? [ In reply to ]

f-myth-users at media

Apr 3, 2024, 12:18 PM

Post #3 of 6 (141 views)

Permalink

> Date: Wed, 3 Apr 2024 14:53:54 -0400
> From: Will Dormann <wdormann@gmail.com>

> - Can anybody think of a reason why I might have set --no-allatonce in
> my mythfilldatabase preferences in the first place? (Specifically, I
> want to ensure that removing this flag will not have negative consequences)

I think this was indeed to minimize RAM usage on small machines, if I recall correctly.

> - Why would --no-allatonce seemingly work just fine all this time, and
> starting on April 4, refuse to update listings after 8PM?

As a total guess... You wouldn't happen to be four hours east of UTC,
would you? The timezone in your email certainly claims that, and it
seems reasonable that each "day" in the listings downloaded starts and
ends at a UTC boundary, so if it is somehow only downloading a single
day or is missing days, I'd expect there to be missing data which is
aligned to 2000 in your local timezone.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Re: mythfilldatabase: use of --no-allatonce prevented listings from 8PM on? [ In reply to ]

pb.mythtv at gmail

Apr 3, 2024, 12:57 PM

Post #4 of 6 (141 views)

Permalink

On 4/3/24 15:18, f-myth-users@media.mit.edu wrote:
> > Date: Wed, 3 Apr 2024 14:53:54 -0400
> > From: Will Dormann<wdormann@gmail.com>
>
> > - Can anybody think of a reason why I might have set --no-allatonce in
> > my mythfilldatabase preferences in the first place? (Specifically, I
> > want to ensure that removing this flag will not have negative consequences)
>
> I think this was indeed to minimize RAM usage on small machines, if I recall correctly.

Note that in February 2020, mythfilldatabase was changed to reduce
memory usage. Until then, the entire input file was read and parsed into
memory before anything was processed, resulting in it using a lot of
memory. A test I ran at the time showed it used 5.6 GB

After the change, a test on the same file used 698 MB.

See https://github.com/MythTV/mythtv/commit/a9aa006139

So allatonce is not needed for reducing memory usage any more, as long
as you are using v32 or later.

Peter

Re: mythfilldatabase: use of --no-allatonce prevented listings from 8PM on? [ In reply to ]

wdormann at gmail

Apr 3, 2024, 1:50 PM

Post #5 of 6 (141 views)

Permalink

On 4/3/24 3:57 PM, Peter Bennett wrote:
> See https://github.com/MythTV/mythtv/commit/a9aa006139
>
> So allatonce is not needed for reducing memory usage any more, as long
> as you are using v32 or later.
>

I am indeed running v32 currently, but was definitely running older
versions in the past.

Thanks for putting my mind at ease that I can safely remove this flag
from my preferences!

-WD
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Re: mythfilldatabase: use of --no-allatonce prevented listings from 8PM on? [ In reply to ]

gary.buhrmaster at gmail

Apr 5, 2024, 4:25 AM

Post #6 of 6 (120 views)

Permalink

On Wed, Apr 3, 2024 at 7:58?PM Peter Bennett <pb.mythtv@gmail.com> wrote:

> So allatonce is not needed for reducing memory usage any more, as long as you are using v32 or later.

It is *possible* that for systems with very
limited ram (< 1GB?) and with an EPG
provider with lots of channels (> 500?)
with multiple days of schedules (> 10?)
that --no-allatonce might still be necessary
to avoid excessive swapping and slow
processing some of the time. While
I am sure there are those who are trying
to determine how low (in memory) they
can go, I don't think most users are
going to need --no-allatonce.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org