Mailing List Archive

On Mon, Mar 26, 2012 at 05:12:55PM +0100, Nicholas Clark wrote:
> On Wed, Feb 29, 2012 at 10:46:43AM -0500, George Greer wrote:
> > On Wed, 29 Feb 2012, Nicholas Clark wrote:
> >
> > > The thing that *would* have got it is a "build all of CPAN", which we don't
> > > have the resources to do for every change. So it gets down to - how do we
> > > spot which changes need that sort of thing?
> >
> > If I can get a copy of the "build all of CPAN" script that the comparisons
> > from a while ago were done with, I can try to set it up on a rolling
> > basis. That is, it would compare blead from the last pass with whatever
> > was blead at the start of the next pass. So it wouldn't be every change
> > but it would at least be more than never.
>
> I think you'd need to ask Steffen where that is

Oops, I don't have an alias for Steffen as "steffen". Correct cc now there.

Clearly "spell checker" is not the only check I need before hitting "send"

Nicholas Clark

On 03/26/2012 06:15 PM, Nicholas Clark wrote:
> On Mon, Mar 26, 2012 at 05:12:55PM +0100, Nicholas Clark wrote:
>> On Wed, Feb 29, 2012 at 10:46:43AM -0500, George Greer wrote:
>>> On Wed, 29 Feb 2012, Nicholas Clark wrote:
>>>
>>>> The thing that *would* have got it is a "build all of CPAN", which we don't
>>>> have the resources to do for every change. So it gets down to - how do we
>>>> spot which changes need that sort of thing?
>>>
>>> If I can get a copy of the "build all of CPAN" script that the comparisons
>>> from a while ago were done with, I can try to set it up on a rolling
>>> basis. That is, it would compare blead from the last pass with whatever
>>> was blead at the start of the next pass. So it wouldn't be every change
>>> but it would at least be more than never.
>>
>> I think you'd need to ask Steffen where that is
>
> Oops, I don't have an alias for Steffen as "steffen". Correct cc now there.
>
> Clearly "spell checker" is not the only check I need before hitting "send"

It's here:

https://github.com/tsee/cpan_perl_branch_smoke

The README should mostly have step-by-step instructions. There's a whole
bunch of gotchas such as requiring a fair amount of disk (in particular
when running multiple processes for each commit). The other one is that
the tempdir setting seems to be ignored by CPANPLUS. Whether that's due
to CPANPLUS or the surrounding script I did not bother to find out. I
just used TMPDIR=/foo/bar when kicking off the actual smokers. ...

Let me know if you have questions.

Best regards,
Steffen

On Mon, Mar 26, 2012 at 05:12:55PM +0100, Nicholas Clark wrote:
> Also those 1 or 2 characters can make a difference when the most important
> bit is actually the detail of PASS(...) or FAIL(...), which can fall off the
> right

Then wouldn't it be even more useful to have the PASS(...)/FAIL(...) bit
first?

-doy

On Mon, 26 Mar 2012 17:12:55 +0100, Nicholas Clark <nick@ccl4.org>
wrote:

> On Wed, Feb 29, 2012 at 10:46:43AM -0500, George Greer wrote:
> > On Wed, 29 Feb 2012, Nicholas Clark wrote:
> >
> > > The thing that *would* have got it is a "build all of CPAN", which we don't
> > > have the resources to do for every change. So it gets down to - how do we
> > > spot which changes need that sort of thing?
> >
> > If I can get a copy of the "build all of CPAN" script that the comparisons
> > from a while ago were done with, I can try to set it up on a rolling
> > basis. That is, it would compare blead from the last pass with whatever
> > was blead at the start of the next pass. So it wouldn't be every change
> > but it would at least be more than never.
>
> I think you'd need to ask Steffen where that is
>
> > >> Should then *all* changes go through a smoke-me change? I treat all of
> > >
> > > I'd actually be happy if (nearly) all non-doc changes *did*. But it wouldn't
> > > have caught this one.
> > >
> > > (We'd need to improve the smoke-me reporting infrastructure to make this
> > > useful though)
> >
> > What's on your wishlist?
>
> I fear that this isn't complete, as I think I've forgotten something.
>
> It's partly that (as best I can tell) the code you're running locally has
> diverged from the "upstream" code, so it's unclear whether bugs fixes and
> other improvements are getting made in more than one place, which is a
> duplication of effort.
>
> In particular, I'd like everyone else to run your code, because of a couple of
> minor things:
>
> * the subject using the branch name is terser
> Smoke [blead] v5.15.9-20-g15d94df
> vs the tautological
> Smoke [5.15.9] v5.15.9-20-g15d94df
> Also those 1 or 2 characters can make a difference when the most important
> bit is actually the detail of PASS(...) or FAIL(...), which can fall off the
> right

We can change that (too)

> * the smoke-me code mails me directly if the branch fails

That might be amongst several new options

> * I can get the logs

That will probably be possible too in the new setup. The QAH is just a
few days away

--
H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/
using perl5.00307 .. 5.14 porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/
http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/

On Mon, Mar 26, 2012 at 11:39:52AM -0500, Jesse Luehrs wrote:
> On Mon, Mar 26, 2012 at 05:12:55PM +0100, Nicholas Clark wrote:
> > Also those 1 or 2 characters can make a difference when the most important
> > bit is actually the detail of PASS(...) or FAIL(...), which can fall off the
> > right
>
> Then wouldn't it be even more useful to have the PASS(...)/FAIL(...) bit
> first?

I thought about that, but forgot to say that that makes it hard to sort by
subject in a mail client to get all results for the same platform together.
Right now, sorting by subject is a fast way to determine when a platform
started to fail (or pass)

Nicholas Clark

On Mon, Mar 26, 2012 at 06:52:12PM +0200, H.Merijn Brand wrote:
> On Mon, 26 Mar 2012 17:12:55 +0100, Nicholas Clark <nick@ccl4.org>
> wrote:

> > * the subject using the branch name is terser
> > Smoke [blead] v5.15.9-20-g15d94df
> > vs the tautological
> > Smoke [5.15.9] v5.15.9-20-g15d94df
> > Also those 1 or 2 characters can make a difference when the most important
> > bit is actually the detail of PASS(...) or FAIL(...), which can fall off the
> > right
>
> We can change that (too)
>
> > * the smoke-me code mails me directly if the branch fails
>
> That might be amongst several new options
>
> > * I can get the logs
>
> That will probably be possible too in the new setup. The QAH is just a
> few days away

This would all be cool.
It's just is (probably) faster to integrate his code than re-implement it.

Its your valuable Parisian drinking time I care for. Honest :-)

Nicholas Clark

(due to an ADSL outage this is copy/pasted from the web archives)

> On Wed, Feb 29, 2012 at 10:46:43AM -0500, George Greer wrote:
>
> > What's on your wishlist?
>
> I fear that this isn't complete, as I think I've forgotten something.
>
> It's partly that (as best I can tell) the code you're running locally
> has diverged from the "upstream" code, so it's unclear whether bugs
> fixes and other improvements are getting made in more than one place,
> which is a duplication of effort.

My Test::Smoke customizations are:

1. Run 'make minitest' in addition to other tests. (Although I never did
investigate how to get the report matrix to add that as a column...)

2. I moved the "user_note" to the very top of the reports instead of the
bottom. (Currently used for URL to reports but also soon a disclaimer
about my Win32 VM's propensity to have timing issues.)

3. Also allow ".config" suffix on configurations. (Previously "_config".)

4. Get version (what is between the brackets in smoke email subject line)
from $ENV{TEST_SMOKE_BRANCH}, if present, instead of repeating Perl
version. (This is what makes the "Smoke [blead]" in my reports.)

> In particular, I'd like everyone else to run your code, because of a
> couple of minor things:
>
> * the subject using the branch name is terser
> Smoke [blead] v5.15.9-20-g15d94df
> vs the tautological
> Smoke [5.15.9] v5.15.9-20-g15d94df
> Also those 1 or 2 characters can make a difference when the most
> important bit is actually the detail of PASS(...) or FAIL(...),
> which can fall off the right

I thought that was rather silly too. I didn't add that to Test::Smoke
in the best way though (via parsing ".patch") since my "smoke-me" script
already adds the branch name as an environment variable and so that was
fastest for tuit use.

> * the smoke-me code mails me directly if the branch fails

That's a function of the 'smoke-me' script, not Test::Smoke, although I
agree it is a necessity for a 'smoke-me' service to do so. It's an
unconditional email though, not just failures.

My configuration for that is:

driver/smoke-me_clang_quick.config.template:
'to' => 'smokers-reports@perl.org,%COMMITTER_EMAIL%',

and the 'smoke-me' script does some variable replacements before making
the .config file for Test::Smoke.

(For those who may not know, the smoke-me script:
https://github.com/greerga/smoke-me/
)

> * I can get the logs

True, that's necessary for any 'smoke-me' service, Although the dashboard
Tony Cook has is even better. (http://perl.develop-help.com/reports/)

I wonder if he has that on github...

> but also I'd like a couple of visibility bugs in your setup to be fixed:
>
> X X O X X X O X -Uusenm -Duseithreads -Dmad
> | | | | | +- LC_ALL = en_US.utf8 -DDEBUGGING
> | | | | +--- PERLIO = perlio -DDEBUGGING
> | | | +----- PERLIO = stdio -DDEBUGGING
> | | +------- LC_ALL = en_US.utf8
> | +--------- PERLIO = perlio
> +----------- PERLIO = stdio
>
> 8 results vs 6 annotations, or 4 results vs 2 annotations:
>
> O F F F
> O F F F -Duseithreads
> | +--------- -DDEBUGGING
> +----------- no debugging

Yes, my adding 'minitest' to all runs did that. I do need to fix that.


> after which I guess that there are more general skimming issues with the
> smoke output. I'm familiar with it, and I find it easy to read, but
> others are not paying attention to the smoke output because they
> perceive it as impenetrable noise. It would be useful force those people
> to explain what they find most obnoxious about it, fix that, iterate
> until they run out of complaints.

I agree. My biggest wishlist for Test::Smoke is (automatically) keeping
track of when a test first started failing so it can differentiate between
"failure" and "known failure", and while we're at it "sporadic failure"
would be nice. I find the Test::Smoke code...dense...but then I've mostly
only had time for drive-by changes to it and not been able to sit down and
follow its flow.

> First off, I'm not sure whether the line "Summary: PASS" (or FAIL...)
> should be the first line, with 2 (or 3) blank lines beneath it. But I'm
> not the target for such improvements - really the monthly release
> managers are the people whose input we should be getting.

There was a paucity of replies to this particular thread so we may never
know.

--
George Greer

On Wed, 4 Apr 2012 21:02:07 -0400 (EDT), George Greer
<perl@greerga.m-l.org> wrote:

> (due to an ADSL outage this is copy/pasted from the web archives)
>
> > On Wed, Feb 29, 2012 at 10:46:43AM -0500, George Greer wrote:
> >
> > > What's on your wishlist?
> >
> > I fear that this isn't complete, as I think I've forgotten something.
> >
> > It's partly that (as best I can tell) the code you're running locally
> > has diverged from the "upstream" code, so it's unclear whether bugs
> > fixes and other improvements are getting made in more than one place,
> > which is a duplication of effort.
>
> My Test::Smoke customizations are:
>
> 1. Run 'make minitest' in addition to other tests.

If others find that useful too, we should make that optional.

> (Although I never did investigate how to get the report matrix to add
> that as a column...)

That should have been solved in the new setup

> 2. I moved the "user_note" to the very top of the reports instead of the
> bottom. (Currently used for URL to reports but also soon a disclaimer
> about my Win32 VM's propensity to have timing issues.)

That is in a template in the new version, moving it to other places is
easy, but maybe not even necessary in the new database face. Our aim is
to not send reports to a list at all (you can still send mail to
yourself to get a copy).

> 3. Also allow ".config" suffix on configurations. (Previously "_config".)

Abe?

> 4. Get version (what is between the brackets in smoke email subject line)
> from $ENV{TEST_SMOKE_BRANCH}, if present, instead of repeating Perl
> version. (This is what makes the "Smoke [blead]" in my reports.)

As no mails will be send, this is moot

> > In particular, I'd like everyone else to run your code, because of a
> > couple of minor things:
> >
> > * the subject using the branch name is terser
> > Smoke [blead] v5.15.9-20-g15d94df
> > vs the tautological
> > Smoke [5.15.9] v5.15.9-20-g15d94df
> > Also those 1 or 2 characters can make a difference when the most
> > important bit is actually the detail of PASS(...) or FAIL(...),
> > which can fall off the right
>
> I thought that was rather silly too. I didn't add that to Test::Smoke
> in the best way though (via parsing ".patch") since my "smoke-me" script
> already adds the branch name as an environment variable and so that was
> fastest for tuit use.
>
> > * the smoke-me code mails me directly if the branch fails
>
> That's a function of the 'smoke-me' script, not Test::Smoke, although I
> agree it is a necessity for a 'smoke-me' service to do so. It's an
> unconditional email though, not just failures.
>
> My configuration for that is:
>
> driver/smoke-me_clang_quick.config.template:
> 'to' => 'smokers-reports@perl.org,%COMMITTER_EMAIL%',
>
> and the 'smoke-me' script does some variable replacements before making
> the .config file for Test::Smoke.
>
> (For those who may not know, the smoke-me script:
> https://github.com/greerga/smoke-me/
> )
>
> > * I can get the logs

In the new setup, logs are being sent to the database is the final
status was not "PASS". This is configurable.

> True, that's necessary for any 'smoke-me' service, Although the dashboard
> Tony Cook has is even better. (http://perl.develop-help.com/reports/)
>
> I wonder if he has that on github...
>
> > but also I'd like a couple of visibility bugs in your setup to be fixed:
> >
> > X X O X X X O X -Uusenm -Duseithreads -Dmad
> > | | | | | +- LC_ALL = en_US.utf8 -DDEBUGGING
> > | | | | +--- PERLIO = perlio -DDEBUGGING
> > | | | +----- PERLIO = stdio -DDEBUGGING
> > | | +------- LC_ALL = en_US.utf8
> > | +--------- PERLIO = perlio
> > +----------- PERLIO = stdio
> >
> > 8 results vs 6 annotations, or 4 results vs 2 annotations:
> >
> > O F F F
> > O F F F -Duseithreads
> > | +--------- -DDEBUGGING
> > +----------- no debugging
>
> Yes, my adding 'minitest' to all runs did that. I do need to fix that.
>
> > after which I guess that there are more general skimming issues with the
> > smoke output. I'm familiar with it, and I find it easy to read, but
> > others are not paying attention to the smoke output because they
> > perceive it as impenetrable noise. It would be useful force those people
> > to explain what they find most obnoxious about it, fix that, iterate
> > until they run out of complaints.
>
> I agree. My biggest wishlist for Test::Smoke is (automatically) keeping
> track of when a test first started failing so it can differentiate between

The new setup registers start time of every smoke-configuration subset
and the duration thereof

> "failure" and "known failure", and while we're at it "sporadic failure"
> would be nice. I find the Test::Smoke code...dense...but then I've mostly

Sporadic failures should now be detectable, as we store failures per
test file, so you could make a trendline for e.g. op/read.t

> only had time for drive-by changes to it and not been able to sit down and
> follow its flow.
>
> > First off, I'm not sure whether the line "Summary: PASS" (or FAIL...)
> > should be the first line, with 2 (or 3) blank lines beneath it. But I'm
> > not the target for such improvements - really the monthly release
> > managers are the people whose input we should be getting.
>
> There was a paucity of replies to this particular thread so we may never
> know.

We will be on #smoke in irc.perl.org to discuss wishes (when possible)

--
H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/
using perl5.00307 .. 5.14 porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/
http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/

On Thu, Apr 05, 2012 at 09:38:15AM +0200, H.Merijn Brand wrote:
> On Wed, 4 Apr 2012 21:02:07 -0400 (EDT), George Greer
> <perl@greerga.m-l.org> wrote:
>
> > (due to an ADSL outage this is copy/pasted from the web archives)
> >
> > > On Wed, Feb 29, 2012 at 10:46:43AM -0500, George Greer wrote:
> > >
> > > > What's on your wishlist?
> > >
> > > I fear that this isn't complete, as I think I've forgotten something.
> > >
> > > It's partly that (as best I can tell) the code you're running locally
> > > has diverged from the "upstream" code, so it's unclear whether bugs
> > > fixes and other improvements are getting made in more than one place,
> > > which is a duplication of effort.
> >
> > My Test::Smoke customizations are:
> >
> > 1. Run 'make minitest' in addition to other tests.
>
> If others find that useful too, we should make that optional.

Well, I find it useful that at least one smoker is running it.
I don't know what the right way to customise it is.

> > (Although I never did investigate how to get the report matrix to add
> > that as a column...)


> > 3. Also allow ".config" suffix on configurations. (Previously "_config".)
>
> Abe?
>
> > 4. Get version (what is between the brackets in smoke email subject line)
> > from $ENV{TEST_SMOKE_BRANCH}, if present, instead of repeating Perl
> > version. (This is what makes the "Smoke [blead]" in my reports.)
>
> As no mails will be send, this is moot

Not totally. I'd still like to be getting mails for my failed smoke-me
branches, and it's nicer with the branch name. Also, I'm not *sure* if I'd
prefer to also opt-in to get e-mail for failures on blead (or any branch
tested) if it was my fault - ie approximated by "the head commit was me", in
which case it's also useful for it. But that's starting to sound more complex
than just "convention is that smoke-me smokers notify on failure" with the
opt-out being "well, don't create a smoke-me branch then"


> > > * I can get the logs
>
> In the new setup, logs are being sent to the database is the final
> status was not "PASS". This is configurable.

oooh. nice.

> > True, that's necessary for any 'smoke-me' service, Although the dashboard
> > Tony Cook has is even better. (http://perl.develop-help.com/reports/)
> >
> > I wonder if he has that on github...

Good question.

> > > but also I'd like a couple of visibility bugs in your setup to be fixed:
> > >
> > > X X O X X X O X -Uusenm -Duseithreads -Dmad
> > > | | | | | +- LC_ALL = en_US.utf8 -DDEBUGGING
> > > | | | | +--- PERLIO = perlio -DDEBUGGING
> > > | | | +----- PERLIO = stdio -DDEBUGGING
> > > | | +------- LC_ALL = en_US.utf8
> > > | +--------- PERLIO = perlio
> > > +----------- PERLIO = stdio
> > >
> > > 8 results vs 6 annotations, or 4 results vs 2 annotations:
> > >
> > > O F F F
> > > O F F F -Duseithreads
> > > | +--------- -DDEBUGGING
> > > +----------- no debugging
> >
> > Yes, my adding 'minitest' to all runs did that. I do need to fix that.

Yes, please magically find time to fix it :-)

Although it might be better to spend that same time migrating to the new
Test::Smoke code, if it makes it easier to add custom columns.

> > > after which I guess that there are more general skimming issues with the
> > > smoke output. I'm familiar with it, and I find it easy to read, but
> > > others are not paying attention to the smoke output because they
> > > perceive it as impenetrable noise. It would be useful force those people
> > > to explain what they find most obnoxious about it, fix that, iterate
> > > until they run out of complaints.
> >
> > I agree. My biggest wishlist for Test::Smoke is (automatically) keeping
> > track of when a test first started failing so it can differentiate between
>
> The new setup registers start time of every smoke-configuration subset
> and the duration thereof
>
> > "failure" and "known failure", and while we're at it "sporadic failure"
> > would be nice. I find the Test::Smoke code...dense...but then I've mostly
>
> Sporadic failures should now be detectable, as we store failures per
> test file, so you could make a trendline for e.g. op/read.t

Nice

> > > First off, I'm not sure whether the line "Summary: PASS" (or FAIL...)
> > > should be the first line, with 2 (or 3) blank lines beneath it. But I'm
> > > not the target for such improvements - really the monthly release
> > > managers are the people whose input we should be getting.
> >
> > There was a paucity of replies to this particular thread so we may never
> > know.
>
> We will be on #smoke in irc.perl.org to discuss wishes (when possible)

I suspect we (also)(for some value of we) need to be more systematic in
asking monthly release managers what confused them in the smoke reports.

Easiest way to make that happen seems to be to make a question about smoke
reports a regular fixture on the Onionsketch agenda.

Nicholas Clark

On Thu, 5 Apr 2012, H.Merijn Brand wrote:

> On Wed, 4 Apr 2012 21:02:07 -0400 (EDT), George Greer
>
>> 4. Get version (what is between the brackets in smoke email subject line)
>> from $ENV{TEST_SMOKE_BRANCH}, if present, instead of repeating Perl
>> version. (This is what makes the "Smoke [blead]" in my reports.)
>
> As no mails will be send, this is moot

It does keep track of the git branch in the database though, correct?


>> "failure" and "known failure", and while we're at it "sporadic failure"
>> would be nice. I find the Test::Smoke code...dense...but then I've mostly
>
> Sporadic failures should now be detectable, as we store failures per
> test file, so you could make a trendline for e.g. op/read.t

Is there functionality to ignore certain tests failing on particular
servers? For example, my Win32 smoker fails various timing tests due to
it being in a VM with a bouncy clock which aren't otherwise interesting.
My Linux smoker though wouldn't have any clock problems (other than tests
that make assumptions about load average).

--
George Greer