Mailing List Archive

Reducing frontend startup time
Hi all,

I've put together 5 patches on the devel/lvr/startup branch which
together considerably reduce the startup time on remote frontends. The
saving is especially significant on my Raspberry Pi's where the time is
reduced from around 15 seconds down to 5.

The changes mainly introduce static caches around MySQL select
operations that retrieve static settings. I've also included a patch to
UPnP startup to make the UPnP announcement run in it's own thread and so
not delay the main thread - this saves a second or so.

I've been using these patches on my own systems for a couple of years
now. But before I pushed I wanted to give everyone a chance to comment
on them as they have a significant impact on the database.

Thanks.

--
Lawrence Rust

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
On 1/8/16 1:54 PM, Lawrence Rust wrote:
> Hi all,
>
> I've put together 5 patches on the devel/lvr/startup branch which
> together considerably reduce the startup time on remote frontends. The
> saving is especially significant on my Raspberry Pi's where the time is
> reduced from around 15 seconds down to 5.
>
> The changes mainly introduce static caches around MySQL select
> operations that retrieve static settings. I've also included a patch to
> UPnP startup to make the UPnP announcement run in it's own thread and so
> not delay the main thread - this saves a second or so.
>
> I've been using these patches on my own systems for a couple of years
> now. But before I pushed I wanted to give everyone a chance to comment
> on them as they have a significant impact on the database.
Hi Lawrence,

You've made so many (wonderful!) changes in that branch that it can be
difficult to find just these few changes. Could you please post some links?
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
On Fri, 2016-01-08 at 13:57 -0500, Dan Wilga wrote:
> On 1/8/16 1:54 PM, Lawrence Rust wrote:
> > Hi all,
> >
> > I've put together 5 patches on the devel/lvr/startup branch which
> > together considerably reduce the startup time on remote frontends. The
> > saving is especially significant on my Raspberry Pi's where the time is
> > reduced from around 15 seconds down to 5.
> >
> > The changes mainly introduce static caches around MySQL select
> > operations that retrieve static settings. I've also included a patch to
> > UPnP startup to make the UPnP announcement run in it's own thread and so
> > not delay the main thread - this saves a second or so.
> >
> > I've been using these patches on my own systems for a couple of years
> > now. But before I pushed I wanted to give everyone a chance to comment
> > on them as they have a significant impact on the database.
> Hi Lawrence,
>
> You've made so many (wonderful!) changes in that branch that it can be
> difficult to find just these few changes. Could you please post some links?

Sure,

DB: Load all dbase settings into cache at startup
https://github.com/MythTV/mythtv/commit/37c2c3ce047a442930830060daf8e1faee1a7152

DB: Update settings cache in SaveSettingOnHost
https://github.com/MythTV/mythtv/commit/8d987c9d5baf15cbdd4de18dea7272e8cadc6cd2

SimpleDBStorage: Where possible use Get/SetSetting to speed frontend
https://github.com/MythTV/mythtv/commit/2181ecca4f35ba0dd39182802e74a3a1c6bcb076

UI: Provide dbase cache for RegisterKey and RegisterJump to speed sta
https://github.com/MythTV/mythtv/commit/bea896e448e7a7fd6c6f2b2d46e11f636fe034a3

UPnP: Reduce startup latency by moving blocking code to own thread
https://github.com/MythTV/mythtv/commit/f2226b56c1c03ab76d90b9b081b2c1eb526e71f5

--
Lawrence Rust

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
> Wiadomość napisana przez Lawrence Rust <lvr@softsystem.co.uk> w dniu 08.01.2016, o godz. 20:10:
>
> On Fri, 2016-01-08 at 13:57 -0500, Dan Wilga wrote:
>> On 1/8/16 1:54 PM, Lawrence Rust wrote:
>>> Hi all,
>>>
>>> I've put together 5 patches on the devel/lvr/startup branch which
>>> together considerably reduce the startup time on remote frontends. The
>>> saving is especially significant on my Raspberry Pi's where the time is
>>> reduced from around 15 seconds down to 5.
>>>
>>> The changes mainly introduce static caches around MySQL select
>>> operations that retrieve static settings. I've also included a patch to
>>> UPnP startup to make the UPnP announcement run in it's own thread and so
>>> not delay the main thread - this saves a second or so.
>>>
>>> I've been using these patches on my own systems for a couple of years
>>> now. But before I pushed I wanted to give everyone a chance to comment
>>> on them as they have a significant impact on the database.
>> Hi Lawrence,
>>
>> You've made so many (wonderful!) changes in that branch that it can be
>> difficult to find just these few changes. Could you please post some links?
>
> Sure,
>
> DB: Load all dbase settings into cache at startup
> https://github.com/MythTV/mythtv/commit/37c2c3ce047a442930830060daf8e1faee1a7152 <https://github.com/MythTV/mythtv/commit/37c2c3ce047a442930830060daf8e1faee1a7152>
>
> DB: Update settings cache in SaveSettingOnHost
> https://github.com/MythTV/mythtv/commit/8d987c9d5baf15cbdd4de18dea7272e8cadc6cd2 <https://github.com/MythTV/mythtv/commit/8d987c9d5baf15cbdd4de18dea7272e8cadc6cd2>
>
> SimpleDBStorage: Where possible use Get/SetSetting to speed frontend
> https://github.com/MythTV/mythtv/commit/2181ecca4f35ba0dd39182802e74a3a1c6bcb076 <https://github.com/MythTV/mythtv/commit/2181ecca4f35ba0dd39182802e74a3a1c6bcb076>
>
> UI: Provide dbase cache for RegisterKey and RegisterJump to speed sta
> https://github.com/MythTV/mythtv/commit/bea896e448e7a7fd6c6f2b2d46e11f636fe034a3 <https://github.com/MythTV/mythtv/commit/bea896e448e7a7fd6c6f2b2d46e11f636fe034a3>
>
> UPnP: Reduce startup latency by moving blocking code to own thread
> https://github.com/MythTV/mythtv/commit/f2226b56c1c03ab76d90b9b081b2c1eb526e71f5 <https://github.com/MythTV/mythtv/commit/f2226b56c1c03ab76d90b9b081b2c1eb526e71f5>
>
> --
> Lawrence Rust
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org <mailto:mythtv-dev@mythtv.org>
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev <http://lists.mythtv.org/mailman/listinfo/mythtv-dev>
> http://wiki.mythtv.org/Mailing_List_etiquette <http://wiki.mythtv.org/Mailing_List_etiquette>
> MythTV Forums: https://forum.mythtv.org <https://forum.mythtv.org/>
Lawrence,
I’m using first 4 patches in MiniMyth2 since months with v.good results.
On ION1, FE startup time is reduced from 12-13sec to 7-8sec.
No single stability issue noticed.
IMHO patches are good candidates to be included into official code.

I think current FE has also another area of additional start speed-up: Bonjur startup.
Currently it is in main thread and takes some sec. on startup.

BTW: In my case Bonjur code also has bug: if there is enabled AirPlay video - there is race in concurrent starting audio and video Bonjur services leading to FE crash.
In Minimyth2 crash is average per 10-15 starts.
Re: Reducing frontend startup time [ In reply to ]
On Fri, 2016-01-08 at 20:31 +0100, warpme wrote:
> > Wiadomość napisana przez Lawrence Rust <lvr@softsystem.co.uk> w dniu 08.01.2016, o godz. 20:10:
> >
> > On Fri, 2016-01-08 at 13:57 -0500, Dan Wilga wrote:
> >> On 1/8/16 1:54 PM, Lawrence Rust wrote:
> >>> Hi all,
> >>>
> >>> I've put together 5 patches on the devel/lvr/startup branch which
> >>> together considerably reduce the startup time on remote frontends. The
> >>> saving is especially significant on my Raspberry Pi's where the time is
> >>> reduced from around 15 seconds down to 5.
> >>>
> >>> The changes mainly introduce static caches around MySQL select
> >>> operations that retrieve static settings. I've also included a patch to
> >>> UPnP startup to make the UPnP announcement run in it's own thread and so
> >>> not delay the main thread - this saves a second or so.
> >>>
> >>> I've been using these patches on my own systems for a couple of years
> >>> now. But before I pushed I wanted to give everyone a chance to comment
> >>> on them as they have a significant impact on the database.
> >> Hi Lawrence,
> >>
> >> You've made so many (wonderful!) changes in that branch that it can be
> >> difficult to find just these few changes. Could you please post some links?
> >
> > Sure,
> >
> > DB: Load all dbase settings into cache at startup
> > https://github.com/MythTV/mythtv/commit/37c2c3ce047a442930830060daf8e1faee1a7152 <https://github.com/MythTV/mythtv/commit/37c2c3ce047a442930830060daf8e1faee1a7152>
> >
> > DB: Update settings cache in SaveSettingOnHost
> > https://github.com/MythTV/mythtv/commit/8d987c9d5baf15cbdd4de18dea7272e8cadc6cd2 <https://github.com/MythTV/mythtv/commit/8d987c9d5baf15cbdd4de18dea7272e8cadc6cd2>
> >
> > SimpleDBStorage: Where possible use Get/SetSetting to speed frontend
> > https://github.com/MythTV/mythtv/commit/2181ecca4f35ba0dd39182802e74a3a1c6bcb076 <https://github.com/MythTV/mythtv/commit/2181ecca4f35ba0dd39182802e74a3a1c6bcb076>
> >
> > UI: Provide dbase cache for RegisterKey and RegisterJump to speed sta
> > https://github.com/MythTV/mythtv/commit/bea896e448e7a7fd6c6f2b2d46e11f636fe034a3 <https://github.com/MythTV/mythtv/commit/bea896e448e7a7fd6c6f2b2d46e11f636fe034a3>
> >
> > UPnP: Reduce startup latency by moving blocking code to own thread
> > https://github.com/MythTV/mythtv/commit/f2226b56c1c03ab76d90b9b081b2c1eb526e71f5 <https://github.com/MythTV/mythtv/commit/f2226b56c1c03ab76d90b9b081b2c1eb526e71f5>
> >
> > --
> > Lawrence Rust
> >
> > _______________________________________________
> > mythtv-dev mailing list
> > mythtv-dev@mythtv.org <mailto:mythtv-dev@mythtv.org>
> > http://lists.mythtv.org/mailman/listinfo/mythtv-dev <http://lists.mythtv.org/mailman/listinfo/mythtv-dev>
> > http://wiki.mythtv.org/Mailing_List_etiquette <http://wiki.mythtv.org/Mailing_List_etiquette>
> > MythTV Forums: https://forum.mythtv.org <https://forum.mythtv.org/>
> Lawrence,
> I’m using first 4 patches in MiniMyth2 since months with v.good results.
> On ION1, FE startup time is reduced from 12-13sec to 7-8sec.
> No single stability issue noticed.
> IMHO patches are good candidates to be included into official code.
>
> I think current FE has also another area of additional start speed-up: Bonjur startup.
> Currently it is in main thread and takes some sec. on startup.
>
> BTW: In my case Bonjur code also has bug: if there is enabled AirPlay video - there is race in concurrent starting audio and video Bonjur services leading to FE crash.
> In Minimyth2 crash is average per 10-15 starts.

I must hold my hands up and admit that I don't have the Bonjour
headers/libs installed. I've never needed it so I've never tried. I
must get around to installing the dev package and give it a try.

In all honesty I don't see why, with a bit of effort, we can't get the
time until Myth displays some kind of UI down to just a second or so.
It doesn't inspire confidence in an app when it takes an age to show
just a basic UI. Even on a combined FE/BE there is a significant pause
after starting any Myth program.

Many thanks for your feedback and encouragement.

--
Lawrence Rust

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
>>
>>
>>
>> BTW: In my case Bonjur code also has bug: if there is enabled AirPlay video - there is race in concurrent starting audio and video Bonjur services leading to FE crash.
>> In Minimyth2 crash is average per 10-15 starts.
>
> I must hold my hands up and admit that I don't have the Bonjour
> headers/libs installed. I've never needed it so I've never tried. I
> must get around to installing the dev package and give it a try.
>
> In all honesty I don't see why, with a bit of effort, we can't get the
> time until Myth displays some kind of UI down to just a second or so.
> It doesn't inspire confidence in an app when it takes an age to show
> just a basic UI. Even on a combined FE/BE there is a significant pause
> after starting any Myth program.

I was proposing to move Bonjur init to separate thread - but JYA refused idea with „works for me” answer.
I still think it will be great to have it in separate thread.

BTW: all this startup speed-up is important for Minimyth2 as it is appliance and I’m using S3 sleep-resume.
Unfortunately due https://code.mythtv.org/trac/ticket/12229 I have to quit/re-launch mythfrontend process in sleep-resume cycle.
In such case Your’s start-up speedups are much welcome for minimyth2.
Ideally will be to have #12229 fixed.
In such case mythtv appliance startup(resume in fact) can be instant (i.e. on IntelNUC I’m reaching 2-3 sec from remote power press to mythtv mainmenu).


>
> Many thanks for your feedback and encouragement.

Oh - believe me - it is GREAT PLEASURE for me to work with You!
Re: Reducing frontend startup time [ In reply to ]
On 1/8/16 3:38 PM, warpme wrote:
>
>>> BTW: In my case Bonjur code also has bug: if there is enabled
>>> AirPlay video - there is race in concurrent starting audio and video
>>> Bonjur services leading to FE crash.
>>> In Minimyth2 crash is average per 10-15 starts.
>>
>> I must hold my hands up and admit that I don't have the Bonjour
>> headers/libs installed. I've never needed it so I've never tried. I
>> must get around to installing the dev package and give it a try.
>>
>> In all honesty I don't see why, with a bit of effort, we can't get the
>> time until Myth displays some kind of UI down to just a second or so.
>> It doesn't inspire confidence in an app when it takes an age to show
>> just a basic UI. Even on a combined FE/BE there is a significant pause
>> after starting any Myth program.
>
> I was proposing to move Bonjur init to separate thread - but JYA
> refused idea with „works for me” answer.
> I still think it will be great to have it in separate thread.
>
> BTW: all this startup speed-up is important for Minimyth2 as it is
> appliance and I’m using S3 sleep-resume.
> Unfortunately due https://code.mythtv.org/trac/ticket/12229 I have to
> quit/re-launch mythfrontend process in sleep-resume cycle.
> In such case Your’s start-up speedups are much welcome for minimyth2.
> Ideally will be to have #12229 fixed.
> In such case mythtv appliance startup(resume in fact) can be instant
> (i.e. on IntelNUC I’m reaching 2-3 sec from remote power press to
> mythtv mainmenu).
Interesting. I have always had trouble using MFE after resume from S3
sleep, but with slightly different symptoms. In my case I also
experience the problem after the machine has been asleep for some
unknown length of time greater than a few minutes. But my FE acts as
though it has dropped the UDP port connected to lircd on my BE, so it is
unable to receive remote control keypresses. IIRC, things work perfectly
from the keyboard.
Re: Reducing frontend startup time [ In reply to ]
On 01/08/2016 01:54 PM, Lawrence Rust wrote:
> Hi all,
>
> I've put together 5 patches on the devel/lvr/startup branch which
> together considerably reduce the startup time on remote frontends. The
> saving is especially significant on my Raspberry Pi's where the time is
> reduced from around 15 seconds down to 5.
>
> The changes mainly introduce static caches around MySQL select
> operations that retrieve static settings. I've also included a patch to
> UPnP startup to make the UPnP announcement run in it's own thread and so
> not delay the main thread - this saves a second or so.
>
> I've been using these patches on my own systems for a couple of years
> now. But before I pushed I wanted to give everyone a chance to comment
> on them as they have a significant impact on the database.
>
> Thanks.
>
Hi Lawrence

You have been posting changes for some time now, and I would like to try
them. I have not done a build yet, just used your precompiled archive
files up to now. Can I do a build using the instructions in
mythtv_for_rpi.txt, substituting the github repo for the
www.softsystem.co.uk repo? Must I still use mythbuild.sh or can I use
./configure and make? Is mythbuild.sh now in github? Can the same github
branch be built for x86_64 and for arm?

Thanks
Peter

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
>
> Interesting. I have always had trouble using MFE after resume from S3 sleep, but with slightly different symptoms. In my case I also experience the problem after the machine has been asleep for some unknown length of time greater than a few minutes. But my FE acts as though it has dropped the UDP port connected to lircd on my BE, so it is unable to receive remote control keypresses. IIRC, things work perfectly from the keyboard.
>
Dan,
I think Your’s LIRC issue lays in LIRC HW reinit after S3 resume or… standby power on USB port during S3 sleep (if Your’s IR receiver is on USB port).
What LIRC HW are You using?

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
Michael,
Thx for trying MiniMyth2!

Regarding RPI2 and Minimyth2 - pls see: http://www.gossamer-threads.com/lists/mythtv/dev/593903#593903 <http://www.gossamer-threads.com/lists/mythtv/dev/593903#593903>
Currently I stuck with Qt and EGLFS issue…
Qt5.4.x hasn’t official RPI2 platform def.
Qt5.5.x has issues accordingly to http://www.gossamer-threads.com/lists/mythtv/dev/594209#594209 <http://www.gossamer-threads.com/lists/mythtv/dev/594209#594209>

I can develop mkspec/platform for RPI2 on Qt5.4 - but first I want to prepare myself (and understand) all things related to cross-compile with BCRM IL, RPI2 firmware, U-boot loader and ARM/RPI2 kernel config.

I spent last days on this Qt issue…

Not sure - maybe it will be better to wait for 4.5 kernel as it will have VC4 KMS/DRI infra. included.
MESA already has VC4.
Heh, I was able to get working RPI2 with 3D MESA VC4 accell, kernel KMS and XOrg with 2D glamour based on MESA VC4 3D pipeline.
Even VC4 VDPAU is reported by X.org <http://x.org/> :-)
(which means we can have RPI2 with HW decode via old well knownVDPAU).

Key issue is Qt not ready yet for such future setup (at least I think so)…

Generally, key issue of RPI2 vs. x86 in context Minimyth2 is PXE.
<marketing rant>
For me beauty of diskless, network booted appliance lays in:
- maintenance (single image for all appliances in home)
- S3 sleep/resume for VCR like power on/off
- full PnP (1180 gfx supported today)
</marketing rant>

Anyway - definitely RPI2 is in my plans for Minimyth2.

BTW: You can download new MiniMyth2 master build with current MythTV master code and all Lawrence patches from http://www.gossamer-threads.com/lists/mythtv/dev/594296#594296 <http://www.gossamer-threads.com/lists/mythtv/dev/594296#594296> (and much more) :-)

br
Re: Reducing frontend startup time [ In reply to ]
On Fri, 2016-01-08 at 17:42 -0500, Peter Bennett wrote:
> On 01/08/2016 01:54 PM, Lawrence Rust wrote:
> > Hi all,
> >
> > I've put together 5 patches on the devel/lvr/startup branch which
> > together considerably reduce the startup time on remote frontends. The
> > saving is especially significant on my Raspberry Pi's where the time is
> > reduced from around 15 seconds down to 5.
> >
> > The changes mainly introduce static caches around MySQL select
> > operations that retrieve static settings. I've also included a patch to
> > UPnP startup to make the UPnP announcement run in it's own thread and so
> > not delay the main thread - this saves a second or so.
> >
> > I've been using these patches on my own systems for a couple of years
> > now. But before I pushed I wanted to give everyone a chance to comment
> > on them as they have a significant impact on the database.
> >
> > Thanks.
> >
> Hi Lawrence
>
> You have been posting changes for some time now, and I would like to try
> them. I have not done a build yet, just used your precompiled archive
> files up to now.

There's a pre-compiled archive (for wheezy) here:
http://www.softsystem.co.uk/download/mythtv/mythtv-v0.28-pre-3350-gf2226b5-RPI2.tar.bz2

> Can I do a build using the instructions in
> mythtv_for_rpi.txt, substituting the github repo for the
> www.softsystem.co.uk repo?

Yes, just execute:

# checkout MythTV repository
git clone -b devel/lvr/startup git://github.com/MythTV/mythtv.git
cd mythtv
...

# or if you already have the MythTV repo
cd mythtv
git checkout -b startup devel/lvr/startup

> Must I still use mythbuild.sh or can I use
> ./configure and make?

It's simplest to cross-compile using the mythbuild.sh script. Building
natively (on a RPi or VM) is much slower (RPi >= 12hrs) and more
complex.

> Is mythbuild.sh now in github?

No, get it from my website:
http://www.softsystem.co.uk/download/mythtv/mythbuild.sh

or together with my patches:
http://www.softsystem.co.uk/download/mythtv/mythbuild-CURRENT.zip

> Can the same github
> branch be built for x86_64 and for arm?

Yes, the devel/lvr/startup branch is just current master plus the 5
database patches. Remember to "git clean -fxd ." when switching
targets, or if using mythbuild.sh then append a -E option.

--
Lawrence Rust

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
On Fri, 2016-01-08 at 21:38 +0100, warpme wrote:
> >>
> >>
> >>
> >> BTW: In my case Bonjur code also has bug: if there is enabled AirPlay video - there is race in concurrent starting audio and video Bonjur services leading to FE crash.
> >> In Minimyth2 crash is average per 10-15 starts.
> >
> > I must hold my hands up and admit that I don't have the Bonjour
> > headers/libs installed. I've never needed it so I've never tried. I
> > must get around to installing the dev package and give it a try.
> >
> > In all honesty I don't see why, with a bit of effort, we can't get the
> > time until Myth displays some kind of UI down to just a second or so.
> > It doesn't inspire confidence in an app when it takes an age to show
> > just a basic UI. Even on a combined FE/BE there is a significant pause
> > after starting any Myth program.
>
> I was proposing to move Bonjur init to separate thread - but JYA refused idea with „works for me” answer.
> I still think it will be great to have it in separate thread.
>
> BTW: all this startup speed-up is important for Minimyth2 as it is appliance and I’m using S3 sleep-resume.
> Unfortunately due https://code.mythtv.org/trac/ticket/12229 I have to quit/re-launch mythfrontend process in sleep-resume cycle.
> In such case Your’s start-up speedups are much welcome for minimyth2.
> Ideally will be to have #12229 fixed.
> In such case mythtv appliance startup(resume in fact) can be instant (i.e. on IntelNUC I’m reaching 2-3 sec from remote power press to mythtv mainmenu).

Regarding #12229, have you tried this patch:
[PATCH 273/340] Mythwelcome: Verify BE connection after FE exits

I also use STR-S3 for my combined FE/BE. I haven't seen any problems
since using this patch but I typically don't leave my FE on the EPG
page...

--
Lawrence Rust

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
On 01/09/2016 05:32 AM, Lawrence Rust wrote:
> Yes, just execute: # checkout MythTV repository git clone -b
> devel/lvr/startup git://github.com/MythTV/mythtv.git cd mythtv ... #
> or if you already have the MythTV repo cd mythtv git checkout -b
> startup devel/lvr/startup
>> Must I still use mythbuild.sh or can I use
>> ./configure and make?
> It's simplest to cross-compile using the mythbuild.sh script. Building
> natively (on a Rpi or VM) is much slower (RPi >= 12hrs) and more
> complex.
>
>> Is mythbuild.sh now in github?
> No, get it from my website:
> http://www.softsystem.co.uk/download/mythtv/mythbuild.sh
>
> or together with my patches:
> http://www.softsystem.co.uk/download/mythtv/mythbuild-CURRENT.zip
>
>> Can the same github
>> branch be built for x86_64 and for arm?
> Yes, the devel/lvr/startup branch is just current master plus the 5
> database patches. Remember to "git clean -fxd ." when switching
> targets, or if using mythbuild.sh then append a -E option.
>

Hi Lawrence

I get an error when cross compiling.

I set the path to
tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin
instead of
tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin, because I
am using an x64 system. Without that, the compile was failing much
sooner than this.

I get this error, after it completed extracting the source for qt

/home/peter/work/github.com/MythTV/mythtv/mythbuild/qt-everywhere-opensource-src-5.4.0/qtbase/configure:
3212:
/home/peter/work/github.com/MythTV/mythtv/mythbuild/qt-everywhere-opensource-src-5.4.0/qtbase/configure:
arm-linux-gnueabihf-g++: not found
-reduce-exports was requested but this compiler does not support it
Re-run configure with -v for more information

it says arm-linux-gnueabihf-g++ not found, but it is found when I use
the path that I set as per the instructions

Proof that arm-linux-gnueabihf-g++ exists:

peter@xubuntu1510x64:~/work/github.com/MythTV/mythtv$
PATH="~/work/github.com/raspberrypi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin:$PATH"

peter@xubuntu1510x64:~/work/github.com/MythTV/mythtv$
arm-linux-gnueabihf-g++
arm-linux-gnueabihf-g++: fatal error: no input files
compilation terminated.

Peter

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
On Sat, 2016-01-09 at 14:50 -0500, Peter Bennett wrote:
> On 01/09/2016 05:32 AM, Lawrence Rust wrote:
> > Yes, just execute: # checkout MythTV repository git clone -b
> > devel/lvr/startup git://github.com/MythTV/mythtv.git cd mythtv ... #
> > or if you already have the MythTV repo cd mythtv git checkout -b
> > startup devel/lvr/startup
> >> Must I still use mythbuild.sh or can I use
> >> ./configure and make?
> > It's simplest to cross-compile using the mythbuild.sh script. Building
> > natively (on a Rpi or VM) is much slower (RPi >= 12hrs) and more
> > complex.
> >
> >> Is mythbuild.sh now in github?
> > No, get it from my website:
> > http://www.softsystem.co.uk/download/mythtv/mythbuild.sh
> >
> > or together with my patches:
> > http://www.softsystem.co.uk/download/mythtv/mythbuild-CURRENT.zip
> >
> >> Can the same github
> >> branch be built for x86_64 and for arm?
> > Yes, the devel/lvr/startup branch is just current master plus the 5
> > database patches. Remember to "git clean -fxd ." when switching
> > targets, or if using mythbuild.sh then append a -E option.
> >
>
> Hi Lawrence
>
> I get an error when cross compiling.
>
> I set the path to
> tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin
> instead of
> tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin, because I
> am using an x64 system. Without that, the compile was failing much
> sooner than this.

Yes, using the x64 cross compiler is correct for 64-bit hosts.

> I get this error, after it completed extracting the source for qt
>
> /home/peter/work/github.com/MythTV/mythtv/mythbuild/qt-everywhere-opensource-src-5.4.0/qtbase/configure:
> 3212:
> /home/peter/work/github.com/MythTV/mythtv/mythbuild/qt-everywhere-opensource-src-5.4.0/qtbase/configure:
> arm-linux-gnueabihf-g++: not found
> -reduce-exports was requested but this compiler does not support it
> Re-run configure with -v for more information
>
> it says arm-linux-gnueabihf-g++ not found, but it is found when I use
> the path that I set as per the instructions
>
> Proof that arm-linux-gnueabihf-g++ exists:
>
> peter@xubuntu1510x64:~/work/github.com/MythTV/mythtv$
> PATH="~/work/github.com/raspberrypi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin:$PATH"

NB putting ~ inside double quotes disable its expansion. Use $HOME
instead.

> peter@xubuntu1510x64:~/work/github.com/MythTV/mythtv$
> arm-linux-gnueabihf-g++
> arm-linux-gnueabihf-g++: fatal error: no input files
> compilation terminated.

This is an artefact of the way bash expands $PATH if it finds ~. Qt's
configure script is executed by sh which doesn't expand ~ in $PATH.

--
Lawrence Rust

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
>
> Regarding #12229, have you tried this patch:
> [PATCH 273/340] Mythwelcome: Verify BE connection after FE exits
>
> I also use STR-S3 for my combined FE/BE. I haven't seen any problems
> since using this patch but I typically don't leave my FE on the EPG
> page...
>
> --
> Lawrence Rust
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org <mailto:mythtv-dev@mythtv.org>
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev <http://lists.mythtv.org/mailman/listinfo/mythtv-dev>
> http://wiki.mythtv.org/Mailing_List_etiquette <http://wiki.mythtv.org/Mailing_List_etiquette>
> MythTV Forums: https://forum.mythtv.org <https://forum.mythtv.org/>
Hmm - I see this patch is for myth welcome.
Unfortunately I’m not using mythwelcome.
What I want is to keep all time running mythfrontend - also during S3 cycle.

I’m guessing myth notify connection is TCP type connection.
How active TCP connections are handled in suspend-to-ram case?
Logic says me long sleep cycles are making TCP connections time-outed.
So, after resume, TCP session should be reconnected - if it is time-outed.
I think proper solution should integrate FE with OS in a way that OS resume from sleep should notify FE about resume and FE should check myth notify TCP connection - and reestablish it when time-outed conn. is detected.

Workaround might based on checking notify conn before every usage and quick reconnect when time-outed conn. is detected.
What You think?
Re: Reducing frontend startup time [ In reply to ]
On 1/8/16 6:06 PM, warpme wrote:
>> Interesting. I have always had trouble using MFE after resume from S3 sleep, but with slightly different symptoms. In my case I also experience the problem after the machine has been asleep for some unknown length of time greater than a few minutes. But my FE acts as though it has dropped the UDP port connected to lircd on my BE, so it is unable to receive remote control keypresses. IIRC, things work perfectly from the keyboard.
>>
> Dan,
> I think Your’s LIRC issue lays in LIRC HW reinit after S3 resume or… standby power on USB port during S3 sleep (if Your’s IR receiver is on USB port).
> What LIRC HW are You using?
Here's my setup:

1. I use RF remote controls, which send a signal to a single receiver in
the basement.
2. The receiver converts the RF signal to IR and blasts it into my
HDHomeRun.
3. The HDHR talks to LIRC, running the UDP driver, on the backend machine.
4. The frontend machines all talk to the backend LIRC instance.

It's something to do with the TCP connection between the frontend and
the backend getting dropped if the frontend machine is in S3 idle for
too long. (Note that I said UDP in my original post; I just verified
that it is TCP.)
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
On 1/8/16 2:10 PM, Lawrence Rust wrote:
> On Fri, 2016-01-08 at 13:57 -0500, Dan Wilga wrote:
>> On 1/8/16 1:54 PM, Lawrence Rust wrote:
>>> Hi all,
>>>
>>> I've put together 5 patches on the devel/lvr/startup branch which
>>> together considerably reduce the startup time on remote frontends. The
>>> saving is especially significant on my Raspberry Pi's where the time is
>>> reduced from around 15 seconds down to 5.
>>>
>>> The changes mainly introduce static caches around MySQL select
>>> operations that retrieve static settings. I've also included a patch to
>>> UPnP startup to make the UPnP announcement run in it's own thread and so
>>> not delay the main thread - this saves a second or so.
>>>
>>> I've been using these patches on my own systems for a couple of years
>>> now. But before I pushed I wanted to give everyone a chance to comment
>>> on them as they have a significant impact on the database.
>> Hi Lawrence,
>>
>> You've made so many (wonderful!) changes in that branch that it can be
>> difficult to find just these few changes. Could you please post some links?
> Sure,
>
> DB: Load all dbase settings into cache at startup
> https://github.com/MythTV/mythtv/commit/37c2c3ce047a442930830060daf8e1faee1a7152
>
> DB: Update settings cache in SaveSettingOnHost
> https://github.com/MythTV/mythtv/commit/8d987c9d5baf15cbdd4de18dea7272e8cadc6cd2
>
> SimpleDBStorage: Where possible use Get/SetSetting to speed frontend
> https://github.com/MythTV/mythtv/commit/2181ecca4f35ba0dd39182802e74a3a1c6bcb076
>
> UI: Provide dbase cache for RegisterKey and RegisterJump to speed sta
> https://github.com/MythTV/mythtv/commit/bea896e448e7a7fd6c6f2b2d46e11f636fe034a3
>
> UPnP: Reduce startup latency by moving blocking code to own thread
> https://github.com/MythTV/mythtv/commit/f2226b56c1c03ab76d90b9b081b2c1eb526e71f5
On the order of a "me too", I installed these patches over the weekend
and they definitely improve startup time, with no negative consequences.
Most of my frontends are ION1's.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
On 01/10/2016 04:37 AM, Lawrence Rust wrote:
> peter@xubuntu1510x64:~/work/github.com/MythTV/mythtv$
> PATH="~/work/github.com/raspberrypi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin:$PATH"
> NB putting ~ inside double quotes disable its expansion. Use $HOME
> instead.
>
>> peter@xubuntu1510x64:~/work/github.com/MythTV/mythtv$
>> arm-linux-gnueabihf-g++
>> arm-linux-gnueabihf-g++: fatal error: no input files
>> compilation terminated.
> This is an artefact of the way bash expands $PATH if it finds ~. Qt's
> configure script is executed by sh which doesn't expand ~ in $PATH.
>
Using $HOME solved that. It gets a bit further now, until it fails with
this:

compiling generated/JSNavigator.cpp
arm-linux-gnueabihf-g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://bugs.launchpad.net/gcc-linaro> for instructions.
Makefile.WebCore.Target:129809: recipe for target
'.obj/generated/JSDOMWindow.o' failed
make[3]: *** [.obj/generated/JSDOMWindow.o] Error 4

I tried again, it passed that point then failed on the same thing a bit
later ...

compiling html/HTMLElementsAllInOne.cpp
arm-linux-gnueabihf-g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://bugs.launchpad.net/gcc-linaro> for instructions.
Makefile.WebCore.Target:154262: recipe for target
'.obj/inspector/InspectorAllInOne.o' failed
make[3]: *** [.obj/inspector/InspectorAllInOne.o] Error 4

And tried again - it got a bit further then ...

compiling bindings/js/JSBindingsAllInOne.cpp
arm-linux-gnueabihf-g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://bugs.launchpad.net/gcc-linaro> for instructions.
Makefile.WebCore.Target:155927: recipe for target
'.obj/bindings/js/JSBindingsAllInOne.o' failed
make[3]: *** [.obj/bindings/js/JSBindingsAllInOne.o] Error 4

Sorry to be a nuisance - do you know what is going on here?

Peter


_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
> Date: Mon, 11 Jan 2016 09:43:03 -0500
> From: Dan Wilga <mythtv-dev2@dwilga-linux1.amherst.edu>

> It's something to do with the TCP connection between the frontend and
> the backend getting dropped if the frontend machine is in S3 idle for
> too long. (Note that I said UDP in my original post; I just verified
> that it is TCP.)

That seems reasonable. There are several ways this might happen,
and you may be able to configure them. You may also be able to use
wireshark on the non-sleeping end, or a hub (or a smart switch that
can clone ports, but not a dumb switch) between them, to see what
the actual traffic looks like.

First, one end or both ends may be using TCP keepalives. (ssh will
do this by default, for example, but you can turn it off.) This will
typically cause your connection to die in about two hours, plus some
extra fuzz.

Also, assuming you turn that off, there are a couple different
timeouts that affect how long it takes a connection to die if you've
tried to send a packet to the other end and it hasn't been acknowledged.
You can configure how many retries it takes before the conn fails.
I have no idea if you might have even a single outstanding unacked
packet in either direction when your host sleeps, but it's possible.

(In other words, keepalives send traffic when there hasn't been any
traffic in a long time, and kill the connection if the keepalive
doesn't get acknowledged. TCP timeouts notice if there -is- traffic
but the other end doesn't seem to be answering it.)

This assumes that taking the frontend out of sleep doesn't cause the
network stack to reset its network in some way, of course, which will
kill connections at its end regardless. And that things like arp
don't forget mappings from Ethernet MAC addresses to IP addresses.

If you'd like some pointers as to which values you might try frobbing
with, say so, though I'm unclear whether either end can be configured
to not use keepalives if it is, because I don't know what software is
actually at either end of the TCP connection. It might be a better
idea to explicitly reinitialize the TCP connection at both ends when
coming out of sleep, if you can arrange that.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Reducing frontend startup time [ In reply to ]
Revisiting a very old thread, for the benefit of anyone else following
my problem with LIRC losing its connection upon wake from sleep...

On 1/12/16 12:22 AM, f-myth-users@media.mit.edu wrote:
> > Date: Mon, 11 Jan 2016 09:43:03 -0500
> > From: Dan Wilga <mythtv-dev2@dwilga-linux1.amherst.edu>
>
> > It's something to do with the TCP connection between the frontend and
> > the backend getting dropped if the frontend machine is in S3 idle for
> > too long. (Note that I said UDP in my original post; I just verified
> > that it is TCP.)
>
> That seems reasonable. There are several ways this might happen,
> and you may be able to configure them. You may also be able to use
> wireshark on the non-sleeping end, or a hub (or a smart switch that
> can clone ports, but not a dumb switch) between them, to see what
> the actual traffic looks like.
>
> First, one end or both ends may be using TCP keepalives. (ssh will
> do this by default, for example, but you can turn it off.) This will
> typically cause your connection to die in about two hours, plus some
> extra fuzz.
>
> Also, assuming you turn that off, there are a couple different
> timeouts that affect how long it takes a connection to die if you've
> tried to send a packet to the other end and it hasn't been acknowledged.
> You can configure how many retries it takes before the conn fails.
> I have no idea if you might have even a single outstanding unacked
> packet in either direction when your host sleeps, but it's possible.
>
> (In other words, keepalives send traffic when there hasn't been any
> traffic in a long time, and kill the connection if the keepalive
> doesn't get acknowledged. TCP timeouts notice if there -is- traffic
> but the other end doesn't seem to be answering it.)
>
> This assumes that taking the frontend out of sleep doesn't cause the
> network stack to reset its network in some way, of course, which will
> kill connections at its end regardless. And that things like arp
> don't forget mappings from Ethernet MAC addresses to IP addresses.
>
> If you'd like some pointers as to which values you might try frobbing
> with, say so, though I'm unclear whether either end can be configured
> to not use keepalives if it is, because I don't know what software is
> actually at either end of the TCP connection. It might be a better
> idea to explicitly reinitialize the TCP connection at both ends when
> coming out of sleep, if you can arrange that.
>
In the last few days, work has progressed on this issue:

https://code.mythtv.org/trac/ticket/12229

and the patch at comment 21 works for me.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org