Mailing List Archive

arm64 support for Travis CI testing
I would like to add support for arm64 to httpd/trunk/.travis.yml.
I would then devote some time to getting this support to work.
Are there some steps I should take before adding this commit?

Thanks

Mike Rumph
Re: arm64 support for Travis CI testing [ In reply to ]
On Tue, Jan 07, 2020 at 01:27:33PM -0800, Mike Rumph wrote:
> I would like to add support for arm64 to httpd/trunk/.travis.yml.
> I would then devote some time to getting this support to work.
> Are there some steps I should take before adding this commit?

That'd be great! If you are familiar with github & Travis, you could
try it using a fork of the git repo, following the instructions in
test/README.travis. Otherwise you could just push the change to trunk
and see ;)

I can't remember if I tested arm before, but it seems the non-x86 VMs in
Travis don't have the same set of Debian packages installed as x86, so
some tweaking to the apt: section might be required to get it working.

Regards, Joe
Re: arm64 support for Travis CI testing [ In reply to ]
Thanks Joe,

Since I'm not adding a new feature other than enabling arm64 support,
I'll just add the commit directly so that others can contribute as well
and then revert it if arm64 causes too much noise.

Mike

On Wed, Jan 8, 2020 at 1:00 AM Joe Orton <jorton@redhat.com> wrote:

> On Tue, Jan 07, 2020 at 01:27:33PM -0800, Mike Rumph wrote:
> > I would like to add support for arm64 to httpd/trunk/.travis.yml.
> > I would then devote some time to getting this support to work.
> > Are there some steps I should take before adding this commit?
>
> That'd be great! If you are familiar with github & Travis, you could
> try it using a fork of the git repo, following the instructions in
> test/README.travis. Otherwise you could just push the change to trunk
> and see ;)
>
> I can't remember if I tested arm before, but it seems the non-x86 VMs in
> Travis don't have the same set of Debian packages installed as x86, so
> some tweaking to the apt: section might be required to get it working.
>
> Regards, Joe
>
>
Re: arm64 support for Travis CI testing [ In reply to ]
Okay the first Travis run with arm64 (#201) passed.
The arm64 job (#201.4) took 33 minutes, 5 seconds.
Compared tp ppc64le at 4 minutes, 30 seconds.
This is possibly due to need for extra caching on the first run.
Otherwise, the jobs seem to have the same results.

On Wed, Jan 8, 2020 at 12:13 PM Mike Rumph <mrumph68@gmail.com> wrote:

> Thanks Joe,
>
> Since I'm not adding a new feature other than enabling arm64 support,
> I'll just add the commit directly so that others can contribute as well
> and then revert it if arm64 causes too much noise.
>
> Mike
>
> On Wed, Jan 8, 2020 at 1:00 AM Joe Orton <jorton@redhat.com> wrote:
>
>> On Tue, Jan 07, 2020 at 01:27:33PM -0800, Mike Rumph wrote:
>> > I would like to add support for arm64 to httpd/trunk/.travis.yml.
>> > I would then devote some time to getting this support to work.
>> > Are there some steps I should take before adding this commit?
>>
>> That'd be great! If you are familiar with github & Travis, you could
>> try it using a fork of the git repo, following the instructions in
>> test/README.travis. Otherwise you could just push the change to trunk
>> and see ;)
>>
>> I can't remember if I tested arm before, but it seems the non-x86 VMs in
>> Travis don't have the same set of Debian packages installed as x86, so
>> some tweaking to the apt: section might be required to get it working.
>>
>> Regards, Joe
>>
>>
Re: arm64 support for Travis CI testing [ In reply to ]
On Wed, Jan 08, 2020 at 02:50:17PM -0800, Mike Rumph wrote:
> Okay the first Travis run with arm64 (#201) passed.
> The arm64 job (#201.4) took 33 minutes, 5 seconds.
> Compared tp ppc64le at 4 minutes, 30 seconds.
> This is possibly due to need for extra caching on the first run.
> Otherwise, the jobs seem to have the same results.

Nice, thanks Mike! From the timings in the log a large chunk of the
time (21m) is building and installing the Perl modules, so if the
caching works it should indeed be much faster second time round.

I wonder if the non-x86 builds have more CPUs available, we have
hardcoded "make -j2" at the moment but possibly it makes sense to adjust
this.

Regards, Joe
Re: arm64 support for Travis CI testing [ In reply to ]
Hi,

On Thu, Jan 9, 2020 at 11:01 AM Joe Orton <jorton@redhat.com> wrote:

> On Wed, Jan 08, 2020 at 02:50:17PM -0800, Mike Rumph wrote:
> > Okay the first Travis run with arm64 (#201) passed.
> > The arm64 job (#201.4) took 33 minutes, 5 seconds.
> > Compared tp ppc64le at 4 minutes, 30 seconds.
> > This is possibly due to need for extra caching on the first run.
> > Otherwise, the jobs seem to have the same results.
>
> Nice, thanks Mike! From the timings in the log a large chunk of the
> time (21m) is building and installing the Perl modules, so if the
> caching works it should indeed be much faster second time round.
>
> I wonder if the non-x86 builds have more CPUs available, we have
> hardcoded "make -j2" at the moment but possibly it makes sense to adjust
> this.
>

https://docs.travis-ci.com/user/multi-cpu-architectures/ says:

While available to all Open Source repositories, the concurrency available
for multiple CPU arch-based jobs is limited during the alpha period.

Not sure what exactly they meant with *limited* here.


> Regards, Joe
>
>
Re: arm64 support for Travis CI testing [ In reply to ]
I would like to express my thanks to Luca and Joe and all other people involved in the travis integration! Very nice indeed!

> Am 09.01.2020 um 09:15 schrieb Joe Orton <jorton@redhat.com>:
>
> On Wed, Jan 08, 2020 at 02:50:17PM -0800, Mike Rumph wrote:
>> Okay the first Travis run with arm64 (#201) passed.
>> The arm64 job (#201.4) took 33 minutes, 5 seconds.
>> Compared tp ppc64le at 4 minutes, 30 seconds.
>> This is possibly due to need for extra caching on the first run.
>> Otherwise, the jobs seem to have the same results.
>
> Nice, thanks Mike! From the timings in the log a large chunk of the
> time (21m) is building and installing the Perl modules, so if the
> caching works it should indeed be much faster second time round.
>
> I wonder if the non-x86 builds have more CPUs available, we have
> hardcoded "make -j2" at the moment but possibly it makes sense to adjust
> this.
>
> Regards, Joe
>
AW: arm64 support for Travis CI testing [ In reply to ]
C2 General

> -----Ursprüngliche Nachricht-----
> Von: Joe Orton <jorton@redhat.com>
> Gesendet: Donnerstag, 9. Januar 2020 09:15
> An: dev@httpd.apache.org
> Betreff: Re: arm64 support for Travis CI testing
>
> On Wed, Jan 08, 2020 at 02:50:17PM -0800, Mike Rumph wrote:
> > Okay the first Travis run with arm64 (#201) passed.
> > The arm64 job (#201.4) took 33 minutes, 5 seconds.
> > Compared tp ppc64le at 4 minutes, 30 seconds.
> > This is possibly due to need for extra caching on the first run.
> > Otherwise, the jobs seem to have the same results.
>
> Nice, thanks Mike! From the timings in the log a large chunk of the
> time (21m) is building and installing the Perl modules, so if the
> caching works it should indeed be much faster second time round.
>
> I wonder if the non-x86 builds have more CPUs available, we have
> hardcoded "make -j2" at the moment but possibly it makes sense to adjust
> this.

Would

lscpu -p | grep -c -E ^[0-9]

work on Ubuntu to deliver the number of cores?

So

make -j `lscpu -p | grep -c -E ^[0-9]`

Regards

Rüdiger
Re: arm64 support for Travis CI testing [ In reply to ]
On Thu, Jan 09, 2020 at 09:10:31AM +0000, Pluem, Ruediger, Vodafone Group wrote:
> Would
>
> lscpu -p | grep -c -E ^[0-9]
>
> work on Ubuntu to deliver the number of cores?
>
> So
>
> make -j `lscpu -p | grep -c -E ^[0-9]`

I've got no idea if this DTRT for the mix of VMs & containers that
travis uses, but it's probably worth trying!

Regards, Joe
Re: arm64 support for Travis CI testing [ In reply to ]
The caching does work on arm64 so the build & test only takes 11 minutes
which is reasonable.

https://travis-ci.org/apache/httpd/jobs/634661216

Looks like this one is the next unreliable test to attack -

2425# Failed test 17 in t/apache/expr_string.t at line 87
2426# Failed test 20 in t/apache/expr_string.t at line 87 fail #2
2427t/apache/expr_string.t ..............
2428Failed 2/44 subtests
2429

Can anybody reliably reproduce that failure?
Re: arm64 support for Travis CI testing [ In reply to ]
Il giorno gio 9 gen 2020 alle ore 14:19 Joe Orton <jorton@redhat.com>
ha scritto:
>
> The caching does work on arm64 so the build & test only takes 11 minutes
> which is reasonable.
>
> https://travis-ci.org/apache/httpd/jobs/634661216
>
> Looks like this one is the next unreliable test to attack -
>
> 2425# Failed test 17 in t/apache/expr_string.t at line 87
> 2426# Failed test 20 in t/apache/expr_string.t at line 87 fail #2
> 2427t/apache/expr_string.t ..............
> 2428Failed 2/44 subtests
> 2429
>
> Can anybody reliably reproduce that failure?

Does it happen only for ARM or also for other architectures? Never
seen this before..
Re: arm64 support for Travis CI testing [ In reply to ]
On Thu, Jan 09, 2020 at 03:01:42PM +0100, Luca Toscano wrote:
> Il giorno gio 9 gen 2020 alle ore 14:19 Joe Orton <jorton@redhat.com>
> ha scritto:
> >
> > The caching does work on arm64 so the build & test only takes 11 minutes
> > which is reasonable.
> >
> > https://travis-ci.org/apache/httpd/jobs/634661216
> >
> > Looks like this one is the next unreliable test to attack -
> >
> > 2425# Failed test 17 in t/apache/expr_string.t at line 87
> > 2426# Failed test 20 in t/apache/expr_string.t at line 87 fail #2
> > 2427t/apache/expr_string.t ..............
> > 2428Failed 2/44 subtests
> > 2429
> >
> > Can anybody reliably reproduce that failure?
>
> Does it happen only for ARM or also for other architectures? Never
> seen this before..

Definitely all architectures, it's one Rainer reports as failing
regularly in release testing. Here's another one and I'm pretty sure it
has shown up in more of the recent failures too -

https://travis-ci.org/apache/httpd/jobs/633442262
Re: arm64 support for Travis CI testing [ In reply to ]
I'm interested in taking a look at this t/apache/expr_string.t testcase
failure.
For example, is it the testcase that is in error or is it the code that
it's testing?
Maybe this can be discussed in a separate thread since it is not exclusive
to arm64.
Has a thread been started on this already?

Also, it looks like the arm64 job is taking close to 30 minutes.
- https://travis-ci.org/apache/httpd (Job #213)
But we have seen earlier jobs close to 11 minutes.
Has something changed that can explain this?

On Thu, Jan 9, 2020 at 6:12 AM Joe Orton <jorton@redhat.com> wrote:

> On Thu, Jan 09, 2020 at 03:01:42PM +0100, Luca Toscano wrote:
> > Il giorno gio 9 gen 2020 alle ore 14:19 Joe Orton <jorton@redhat.com>
> > ha scritto:
> > >
> > > The caching does work on arm64 so the build & test only takes 11
> minutes
> > > which is reasonable.
> > >
> > > https://travis-ci.org/apache/httpd/jobs/634661216
> > >
> > > Looks like this one is the next unreliable test to attack -
> > >
> > > 2425# Failed test 17 in t/apache/expr_string.t at line 87
> > > 2426# Failed test 20 in t/apache/expr_string.t at line 87 fail #2
> > > 2427t/apache/expr_string.t ..............
> > > 2428Failed 2/44 subtests
> > > 2429
> > >
> > > Can anybody reliably reproduce that failure?
> >
> > Does it happen only for ARM or also for other architectures? Never
> > seen this before..
>
> Definitely all architectures, it's one Rainer reports as failing
> regularly in release testing. Here's another one and I'm pretty sure it
> has shown up in more of the recent failures too -
>
> https://travis-ci.org/apache/httpd/jobs/633442262
>
>
Re: arm64 support for Travis CI testing [ In reply to ]
Okay Job #214 has the arm64 run back close to 11 minutes.

On Fri, Jan 10, 2020 at 9:55 AM Mike Rumph <mrumph68@gmail.com> wrote:

> I'm interested in taking a look at this t/apache/expr_string.t testcase
> failure.
> For example, is it the testcase that is in error or is it the code that
> it's testing?
> Maybe this can be discussed in a separate thread since it is not exclusive
> to arm64.
> Has a thread been started on this already?
>
> Also, it looks like the arm64 job is taking close to 30 minutes.
> - https://travis-ci.org/apache/httpd (Job #213)
> But we have seen earlier jobs close to 11 minutes.
> Has something changed that can explain this?
>
> On Thu, Jan 9, 2020 at 6:12 AM Joe Orton <jorton@redhat.com> wrote:
>
>> On Thu, Jan 09, 2020 at 03:01:42PM +0100, Luca Toscano wrote:
>> > Il giorno gio 9 gen 2020 alle ore 14:19 Joe Orton <jorton@redhat.com>
>> > ha scritto:
>> > >
>> > > The caching does work on arm64 so the build & test only takes 11
>> minutes
>> > > which is reasonable.
>> > >
>> > > https://travis-ci.org/apache/httpd/jobs/634661216
>> > >
>> > > Looks like this one is the next unreliable test to attack -
>> > >
>> > > 2425# Failed test 17 in t/apache/expr_string.t at line 87
>> > > 2426# Failed test 20 in t/apache/expr_string.t at line 87 fail #2
>> > > 2427t/apache/expr_string.t ..............
>> > > 2428Failed 2/44 subtests
>> > > 2429
>> > >
>> > > Can anybody reliably reproduce that failure?
>> >
>> > Does it happen only for ARM or also for other architectures? Never
>> > seen this before..
>>
>> Definitely all architectures, it's one Rainer reports as failing
>> regularly in release testing. Here's another one and I'm pretty sure it
>> has shown up in more of the recent failures too -
>>
>> https://travis-ci.org/apache/httpd/jobs/633442262
>>
>>
Re: arm64 support for Travis CI testing [ In reply to ]
On Fri, Jan 10, 2020 at 09:55:24AM -0800, Mike Rumph wrote:
> I'm interested in taking a look at this t/apache/expr_string.t
> testcase failure. For example, is it the testcase that is in error or
> is it the code that it's testing? Maybe this can be discussed in a
> separate thread since it is not exclusive to arm64. Has a thread been
> started on this already?

I don't think there's another thread.

I'm guessing this is some kind of timing issue in the test case, which
tries to read an error_log entry immediately after the request which
generates it. I've added a sleep which will maybe mitigate this -
http://svn.apache.org/r1872705 so let's see how this one goes

Regards, Joe