Mailing List Archive

1 2  View All
Re: Migrate to git? [ In reply to ]
On Tue, Oct 8, 2019 at 3:13 AM Joe Orton <jorton@redhat.com> wrote:

> On Sat, Oct 05, 2019 at 04:09:34PM -0400, Jim Jagielski wrote:
> > Various PMCs have made their default/de-facto SCM git and have seen an
> > increase in contributions and contributors...
> >
> > Is this something the httpd project should consider? Especially w/ the
> > foundation officially supporting Github, it seems like time to have a
> > discussion about it, especially as we start thinking about the next 25
> > years of this project :)
>
> Can we use Travis CI as well? If so I am +1 on moving to github, being
> able to easily configure a consistent CI across branches and PRs will be
> a major improvement over the status quo. (I have no idea how buildbot
> works or how to improve it and it's unusuable before commits)
>

Travis CI is possible *today* ... since the svn commits are replicated over
to github, Travis can pick them up and run tests. Just file an INFRA ticket
to enable it.

Cheers,
-g
Re: Migrate to git? [ In reply to ]
> On 7 Oct 2019, at 15:06, Daniel Gruno <humbedooh@apache.org> wrote:
>
> On 06/10/2019 17.59, Nick Kew wrote:
>>
>> OK, I've just dug up an example in an Apache/Github project. A simple renaming
>> of a source file, that with "svn mv" would have preserved history, seems to have
>> essentially wiped its past. 'History' is highly misleading, 'Blame' is 100% wrong!
>
> It would be 100% wrong in svn as well if the same operation had been performed there, as it wasn't a move - the number of lines don't match up. There is a `git mv` just like `svn mv` that preserves history, AIUI. A file where `svn mv` was actually used [1] shows that the history is preserved through the mv operation and blame works as intended, even in git.

Which points to another problem. The committer concerned wasn't a git newbie,
he's been happily using it for (I know not how long) before the project ever came
to Apache. And I can't believe the renaming was malicious.

So why did he get it wrong in this manner? Perhaps it points to the real-life
complexity of git, and its real-life consequences?

I've got caught on that myself (in a non-Apache context) before: things like
git merge of a branch having spectacularly unintended consequences for
what history subsequently looks like.

Indeed, perhaps the cause of the screw-up wasn't really the renaming, but some
subsequent change whose author had never even looked at the source file in question?
If you look at history, many of the changes there seem to be mass formatting-exercises.

The line count appears to be an artifact of the committer's toolset, which includes
some auto-formatting such as
-extern "C" void TSPluginInit(int argc, const char *argv[])
+extern "C" void
+TSPluginInit(int argc, const char *argv[])

> As Eric alluded to, it's much less about svn versus git than it is about tapping into the community on GitHub. If there was an svnhub, I'd be all for that as an easier way to do this, but alas no.

So the Apache community isn't good enough and we need Microsoft?

OK, that's not quite fair. But isn't that what the github mirror is for?

--
Nick Kew
Re: Migrate to git? [ In reply to ]
On Tue, Oct 8, 2019 at 4:45 AM Nick Kew <niq@apache.org> wrote:
>...

> OK, that's not quite fair. But isn't that what the github mirror is for?
>

Not even close to fair, stop that.

The github mirror is readonly. As noted upthread, that means merging PRs
and other activities are limited. It cannot act as a road to
expanding/growing our community (and again: -0 in any case).

Cheers,
-g
Re: Migrate to git? [ In reply to ]
Il giorno mar 8 ott 2019 alle ore 11:04 Greg Stein <gstein@gmail.com>
ha scritto:
>
> Travis CI is possible *today* ... since the svn commits are replicated over to github, Travis can pick them up and run tests. Just file an INFRA ticket to enable it.
>

Thanks for the pointer, will file a task to infra to enable it :)

Luca
Re: Migrate to git? [ In reply to ]
On 10/08/2019 10:44 AM, Greg Stein wrote:
> On Tue, Oct 8, 2019 at 3:13 AM Joe Orton <jorton@redhat.com
> <mailto:jorton@redhat.com>> wrote:
>
> On Sat, Oct 05, 2019 at 04:09:34PM -0400, Jim Jagielski wrote:
> > Various PMCs have made their default/de-facto SCM git and have seen an
> > increase in contributions and contributors...
> >
> > Is this something the httpd project should consider? Especially w/ the
> > foundation officially supporting Github, it seems like time to have a
> > discussion about it, especially as we start thinking about the next 25
> > years of this project :)
>
> Can we use Travis CI as well?  If so I am +1 on moving to github, being
> able to easily configure a consistent CI across branches and PRs will be
> a major improvement over the status quo.  (I have no idea how buildbot
> works or how to improve it and it's unusuable before commits)
>
>
> Travis CI is possible *today* ... since the svn commits are replicated over to
> github, Travis can pick them up and run tests. Just file an INFRA ticket to
> enable it.
>
> Cheers,
> -g
>
Hi Greg,

That does not cover Joe's note "...and PRs...". Basically having a transparent,
dead simple set of gate smoke tests
on a handful of major platforms and config flavours/layouts. Linux and Windows
can be used in this capacity for free (as in free beer).

It makes almost no sense unless all committers agree that all code commits pass
through PR gate, i.e.
no direct commits.

Almost all concerns about git and its presumed complexity can be addressed by
adhering
to a fixed workflow. GitHub PR workflow is one of the options.

Reading the email thread, I get the vibe that the community would have to
put out the SVN vs. Git flame first though :)

K.

Michal Karm Babacek

--
Sent from my Hosaka Ono-Sendai Cyberspace 7
Re: Migrate to git? [ In reply to ]
On Wed, Oct 9, 2019 at 7:59 AM Michal Karm <michal.babacek@gmail.com> wrote:

> On 10/08/2019 10:44 AM, Greg Stein wrote:
> > On Tue, Oct 8, 2019 at 3:13 AM Joe Orton <jorton@redhat.com
> > <mailto:jorton@redhat.com>> wrote:
> >
> > On Sat, Oct 05, 2019 at 04:09:34PM -0400, Jim Jagielski wrote:
> > > Various PMCs have made their default/de-facto SCM git and have
> seen an
> > > increase in contributions and contributors...
> > >
> > > Is this something the httpd project should consider? Especially w/
> the
> > > foundation officially supporting Github, it seems like time to
> have a
> > > discussion about it, especially as we start thinking about the
> next 25
> > > years of this project :)
> >
> > Can we use Travis CI as well? If so I am +1 on moving to github,
> being
> > able to easily configure a consistent CI across branches and PRs
> will be
> > a major improvement over the status quo. (I have no idea how
> buildbot
> > works or how to improve it and it's unusuable before commits)
> >
> >
> > Travis CI is possible *today* ... since the svn commits are replicated
> over to
> > github, Travis can pick them up and run tests. Just file an INFRA ticket
> to
> > enable it.
> >
> > Cheers,
> > -g
> >
> Hi Greg,
>
> That does not cover Joe's note "...and PRs...".


I understand. Just noting that Travis is (already) possible, even if PR
handling/testing/merging is uneven.

Basically having a transparent,
> dead simple set of gate smoke tests
> on a handful of major platforms and config flavours/layouts. Linux and
> Windows
> can be used in this capacity for free (as in free beer).
>
> It makes almost no sense unless all committers agree that all code commits
> pass
> through PR gate, i.e.
> no direct commits.
>

Nope. Won't happen.

The httpd project has been "commit-then-review" for over two decades. "Must
past tests before merge" is antithetical, and I cannot possibly imagine
this community changing to that position.

>...

> Reading the email thread, I get the vibe that the community would have to
> put out the SVN vs. Git flame first though :)
>

FUD. That is not happening here at all. I'm one of the initial SVN
developers, but you won't see any flames from me, about git. I said "-0"
because I believe our community won't see the related growth that some
other projects see. It would be a change for little, if any, benefit. And I
already stated else-thread that I really like GitHub. ... it isn't about a
git/svn flame; it is about benefit/cost.

Cheers,
-g
Re: Migrate to git? [ In reply to ]
On Wed, Oct 9, 2019, 07:59 Michal Karm <michal.babacek@gmail.com> wrote:

> On 10/08/2019 10:44 AM, Greg Stein wrote:
> > On Tue, Oct 8, 2019 at 3:13 AM Joe Orton <jorton@redhat.com
> > <mailto:jorton@redhat.com>> wrote:
> >
> > On Sat, Oct 05, 2019 at 04:09:34PM -0400, Jim Jagielski wrote:
> > > Various PMCs have made their default/de-facto SCM git and have
> seen an
> > > increase in contributions and contributors...
> > >
> > > Is this something the httpd project should consider? Especially w/
> the
> > > foundation officially supporting Github, it seems like time to
> have a
> > > discussion about it, especially as we start thinking about the
> next 25
> > > years of this project :)
> >
> > Can we use Travis CI as well? If so I am +1 on moving to github,
> being
> > able to easily configure a consistent CI across branches and PRs
> will be
> > a major improvement over the status quo. (I have no idea how
> buildbot
> > works or how to improve it and it's unusuable before commits)
> >
> >
> > Travis CI is possible *today* ... since the svn commits are replicated
> over to
> > github, Travis can pick them up and run tests. Just file an INFRA ticket
> to
> > enable it.
> >
> > Cheers,
> > -g
> >
> Hi Greg,
>
> That does not cover Joe's note "...and PRs...". Basically having a
> transparent,
> dead simple set of gate smoke tests
> on a handful of major platforms and config flavours/layouts. Linux and
> Windows
> can be used in this capacity for free (as in free beer).
>
> It makes almost no sense unless all committers agree that all code commits
> pass
> through PR gate, i.e.
> no direct commits.
>

This approach implies a test driven codebase, where every exception to
deliberate behavior at the functional level is called out by the test
schema.

That doesn't describe the scope of our test-framework, which consists only
of behavioral run-time testing against a small and discrete list of
expectations. No automated testing will substitute at this particular
project for attentive code review, although it would be very kind to run
all PR's through a compile validation for the benefit of the submitter and
reviewers.
Re: Migrate to git? [ In reply to ]
Fwiw, Apache Allura has this workflow for svn. And a bunch of other tooling
for github-like functionality around svn.



On Wed, Oct 9, 2019, 08:59 Michal Karm <michal.babacek@gmail.com> wrote:

> On 10/08/2019 10:44 AM, Greg Stein wrote:
> > On Tue, Oct 8, 2019 at 3:13 AM Joe Orton <jorton@redhat.com
> > <mailto:jorton@redhat.com>> wrote:
> >
> > On Sat, Oct 05, 2019 at 04:09:34PM -0400, Jim Jagielski wrote:
> > > Various PMCs have made their default/de-facto SCM git and have
> seen an
> > > increase in contributions and contributors...
> > >
> > > Is this something the httpd project should consider? Especially w/
> the
> > > foundation officially supporting Github, it seems like time to
> have a
> > > discussion about it, especially as we start thinking about the
> next 25
> > > years of this project :)
> >
> > Can we use Travis CI as well? If so I am +1 on moving to github,
> being
> > able to easily configure a consistent CI across branches and PRs
> will be
> > a major improvement over the status quo. (I have no idea how
> buildbot
> > works or how to improve it and it's unusuable before commits)
> >
> >
> > Travis CI is possible *today* ... since the svn commits are replicated
> over to
> > github, Travis can pick them up and run tests. Just file an INFRA ticket
> to
> > enable it.
> >
> > Cheers,
> > -g
> >
> Hi Greg,
>
> That does not cover Joe's note "...and PRs...". Basically having a
> transparent,
> dead simple set of gate smoke tests
> on a handful of major platforms and config flavours/layouts. Linux and
> Windows
> can be used in this capacity for free (as in free beer).
>
> It makes almost no sense unless all committers agree that all code commits
> pass
> through PR gate, i.e.
> no direct commits.
>
> Almost all concerns about git and its presumed complexity can be addressed
> by
> adhering
> to a fixed workflow. GitHub PR workflow is one of the options.
>
> Reading the email thread, I get the vibe that the community would have to
> put out the SVN vs. Git flame first though :)
>
> K.
>
> Michal Karm Babacek
>
> --
> Sent from my Hosaka Ono-Sendai Cyberspace 7
>
>
>
Re: Migrate to git? [ In reply to ]
On Sat, Oct 5, 2019 at 10:09 PM Jim Jagielski <jim@jagunet.com> wrote:
>
> Various PMCs have made their default/de-facto SCM git and have seen an increase in contributions and contributors...
>
> Is this something the httpd project should consider? Especially w/ the foundation officially supporting Github, it seems like time to have a discussion about it, especially as we start thinking about the next 25 years of this project :)

+0.5

I quite like the gh tooling for reviews, ci and merges (provided there
are rules/enforcements about PRs based off the targeted branch, i.e.
no rebase, force-push..), and I think that httpd can gain attractivity
and increase contributions by doing this.

My "concerns" (and missing .5) are about the interraction with our
mailing list(s), though it's probably easy to address (and well done
by other Apache projects).
I wouldn't want to have to browse/track gh to be aware of new PRs,
comments or merges/pulls (including for things I didn't participate to
and thus I'm not subscribed to, yet). Committers need to be aware of
them by the usual way: their mailer (central point for me). I suppose
gh can thread everything smoothly...

So what could be the plan regarding gh activity sent to our lists? New
issues (and their activity) sent to bug@, new PRs (and their activity)
sent to dev@, and merges/pulls to cvs@ (or git@)? How do other Apache
projects manage this?
Finally, what about commenting on gh by replying to emails, can this work?

Regards,
Yann.
Re: Migrate to git? [ In reply to ]
On Tue, Oct 08, 2019 at 02:17:12PM +0200, Luca Toscano wrote:
> Il giorno mar 8 ott 2019 alle ore 11:04 Greg Stein <gstein@gmail.com>
> ha scritto:
> >
> > Travis CI is possible *today* ... since the svn commits are
> > replicated over to github, Travis can pick them up and run tests.
> > Just file an INFRA ticket to enable it.
> >
>
> Thanks for the pointer, will file a task to infra to enable it :)

This would be awesome, did you file something? If not I can.

Like others I am no fan of git, but so long as we can get the process
right, I think using PRs with CI would be a significant benefit for the
project. At minimum we could avoid some of the trivial build breakage
type issues which have delayed 2.4 releases recently.

Even for trunk I would like to be able to develop new features and have
a full test suite run (e.g. w/pool-debug, with different APR releases,
etc etc) easily available without having to wait an hour with laptop
fans giving me a headache.

At the moment I think we have a quality control problem for 2.4.x, yet I
find it hard to justify spending much time on writing test cases because
that stuff is run so rarely. How many tests proposed in 2.4.x STATUS
have had a full test suite run? I certainly don't always do it. But if
we get the tests running all the time automatically it's much easier to
see a return on investment for improving test coverage.

Regards, Joe
Re: Migrate to git? [ In reply to ]
On Mon, Oct 14, 2019 at 03:27:31PM +0100, Joe Orton wrote:
> At the moment I think we have a quality control problem for 2.4.x, yet I
> find it hard to justify spending much time on writing test cases because
> that stuff is run so rarely. How many tests proposed in 2.4.x STATUS
> have had a full test suite run? I certainly don't always do it. But if
> we get the tests running all the time automatically it's much easier to
> see a return on investment for improving test coverage.

I see there's already a buildbot job for httpd trunk:
https://ci.apache.org/waterfall?tag=httpd-trunk
It seems this build job is configured to run a compile but it does not
run the test suite? Putting Github/Travis questions aside, an easy way
to get automated tests going with minimal effort today could be to run
the existing tests inside httpd's existing buildbot job.

The Subversion project has been doing this for years.
See https://subversion.apache.org/buildbot/all
Re: Migrate to git? [ In reply to ]
Hi Joe,

Il giorno lun 14 ott 2019 alle ore 16:27 Joe Orton <jorton@redhat.com>
ha scritto:
>
> On Tue, Oct 08, 2019 at 02:17:12PM +0200, Luca Toscano wrote:
> > Il giorno mar 8 ott 2019 alle ore 11:04 Greg Stein <gstein@gmail.com>
> > ha scritto:
> > >
> > > Travis CI is possible *today* ... since the svn commits are
> > > replicated over to github, Travis can pick them up and run tests.
> > > Just file an INFRA ticket to enable it.
> > >
> >
> > Thanks for the pointer, will file a task to infra to enable it :)
>
> This would be awesome, did you file something? If not I can.
>
> Like others I am no fan of git, but so long as we can get the process
> right, I think using PRs with CI would be a significant benefit for the
> project. At minimum we could avoid some of the trivial build breakage
> type issues which have delayed 2.4 releases recently.
>
> Even for trunk I would like to be able to develop new features and have
> a full test suite run (e.g. w/pool-debug, with different APR releases,
> etc etc) easily available without having to wait an hour with laptop
> fans giving me a headache.
>
> At the moment I think we have a quality control problem for 2.4.x, yet I
> find it hard to justify spending much time on writing test cases because
> that stuff is run so rarely. How many tests proposed in 2.4.x STATUS
> have had a full test suite run? I certainly don't always do it. But if
> we get the tests running all the time automatically it's much easier to
> see a return on investment for improving test coverage.


I still haven't filed the request to infra, I wanted to fix my docker
images first. I have created
https://github.com/elukey/httpd_integration_testing with support for
Debian and Centos (for the moment), to address two use cases:

1) run of the perl/http2 test suites against a new httpd release
2) run of the perl/http2 test suites for the latest version of trunk or 2.4.x

The docker images seem to work fine, but probably there are some stuff
to change/improve. I tried to document myself about Travis and GH, but
still didn't come up with a clear picture. I hope to have something
ready during the next days, but if you have a better
idea/understanding please go ahead and contact infra :)

Thanks!

Luca
Re: Migrate to git? [ In reply to ]
On 14/10/2019 09.51, Stefan Sperling wrote:
> On Mon, Oct 14, 2019 at 03:27:31PM +0100, Joe Orton wrote:
>> At the moment I think we have a quality control problem for 2.4.x, yet I
>> find it hard to justify spending much time on writing test cases because
>> that stuff is run so rarely. How many tests proposed in 2.4.x STATUS
>> have had a full test suite run? I certainly don't always do it. But if
>> we get the tests running all the time automatically it's much easier to
>> see a return on investment for improving test coverage.
>
> I see there's already a buildbot job for httpd trunk:
> https://ci.apache.org/waterfall?tag=httpd-trunk
> It seems this build job is configured to run a compile but it does not
> run the test suite? Putting Github/Travis questions aside, an easy way
> to get automated tests going with minimal effort today could be to run
> the existing tests inside httpd's existing buildbot job.

Infra is in the midst of setting up a new buildbot 2 service with
automated builds sort of like what travis does. It's not ready for prime
time just yet, but within a month or so, we should be able to directly
configure and trigger dynamic builds/tests via a yaml file in our repo.

>
> The Subversion project has been doing this for years.
> See https://subversion.apache.org/buildbot/all
>
Re: Migrate to git? [ In reply to ]
On Mon, Oct 14, 2019 at 04:51:56PM +0200, Stefan Sperling wrote:
> On Mon, Oct 14, 2019 at 03:27:31PM +0100, Joe Orton wrote:
> > At the moment I think we have a quality control problem for 2.4.x, yet I
> > find it hard to justify spending much time on writing test cases because
> > that stuff is run so rarely. How many tests proposed in 2.4.x STATUS
> > have had a full test suite run? I certainly don't always do it. But if
> > we get the tests running all the time automatically it's much easier to
> > see a return on investment for improving test coverage.
>
> I see there's already a buildbot job for httpd trunk:
> https://ci.apache.org/waterfall?tag=httpd-trunk
> It seems this build job is configured to run a compile but it does not
> run the test suite? Putting Github/Travis questions aside, an easy way
> to get automated tests going with minimal effort today could be to run
> the existing tests inside httpd's existing buildbot job.
>
> The Subversion project has been doing this for years.
> See https://subversion.apache.org/buildbot/all

Yes, we have a buildbot which is great, but compared to a GitHub+Travis
workflow it has two downsides: buildbot is horribly complicated, and it
only runs *after* commits. Seeing test results before merging a feature
to trunk to 2.4.x is a world from waiting for a buildbot run.

(To the first: adding a new build with different ./configure arguments
is a typically trivial copy&paste with .travis.yml even for somebody who
doesn't know Travis syntax; I spent an hour reading our bb config once
and was not confident to try changing it)

Regards, Joe
Re: Migrate to git? [ In reply to ]
On 10/06/2019 05:06 AM, Daniel Gruno wrote:

> There is also, as you mention, the risk of force-pushing to rewrite history, but as I understand it, we can disable this
> by requiring PRs for each change to the canonical branch(es).

Maybe a git dummy question, but how do you adjust log messages for already pushed commits without force-pushing?
In Subversion you can just do svn propedit svn:log --revlog -r<revision>, but for git?

I have edited my own log message recently to improve it and I know that others do it to indicate that a commit was
reverted by a future revision or to add CVE numbers after vulnerabilities are public.
Would we loose this possibility with git?

Regards

Rüdiger
Re: Migrate to git? [ In reply to ]
On Tue, Oct 15, 2019 at 1:42 AM Ruediger Pluem <rpluem@apache.org> wrote:
>...

> Would we lose this possibility [of editing log messages] with git?
>

Yes.

The log message is part of the commit hash. You can effectively delete the
"tip" commit of a line-of-development, and replace it with a new commit
(ie. same code changes, but edited log msg). That new commit would have a
different hash. If people sync'd the old commit/hash, their clones are
basically broken post-edit.

It is effectively impossible to edit a log message if it is not-tip. That
is because each ongoing commit has a "parent [hash]" incorporated into its
hash signature. So if you go back in time to edit C1, then you're gonna to
regenerate hashes for C2, C3, and C4 that came after it. With the
corresponding breakage of clones.

Arguments can be made for whether this is pro/con, but I think is not worth
discussing here.

HTH,
-g
Re: Migrate to git? [ In reply to ]
On Sat, Oct 05, 2019 at 04:09:34PM -0400, Jim Jagielski wrote:
> Various PMCs have made their default/de-facto SCM git and have seen an increase in contributions and contributors...
>
> Is this something the httpd project should consider? Especially w/ the foundation officially supporting Github, it seems like time to have a discussion about it, especially as we start thinking about the next 25 years of this project :)
>

Hi,

At W3C, all the Working Group spec as well as some developer work has
moved on over to github. We use github's hooks API to connect actions
on github repositories with our mailing lists and other aspects of the
main server.

Yes, git and github may have a steeper learning curve that svn,
but people now seem satisfied with the current workflow.

Yes, having this kind of work hosted at github has increased
its visibility and getting unexpected external contributions.

For the record, part of this change of system required migrating
CVS repositories from our servers at W3C to github, while preserving
all the history of commits and names of commiteers, using readily
available tools. There were some incompatibilities between CVS unmerged
branches, though, and we had to take decisions in that case. This may
not be the case for SVN.

Another important reason that pushed for this migration is that CVS
clients are becoming less available across platforms. I'm not sure
if that's the same situation for SVN, although I see there are still
recent releases of it. It helps it's hosted at ASF.

For simplicity, from here one I will use "apache" to refer to the
httpd server.

I think that one step for migrating apache to github would require
your writing down a list of things you need from a DRCS and then
working on how to map them to github.

One thing that I find important is that you evaluate how you're
going to handle contributions, or, in git parlance, merge requests (MR).
These are public visible in the repository. Having too many hanging
out without closure doesn't speak well for a project. I talk of this
from experience of seeing that bug reportss and proposed bug fixes are
quickly fixed and integrated into apache (Thanks Yann!), however,
proposal for extensions of existing modules, even when developed in
consultation with the list, just go dormant and require multiple
prodding today and even like that, the hit ratio is really low in my
opinion. After I while this just demotivated me from continuing proposing
them to this list and just stick to the ocassional bug report / patches..
See for example the latest one, dating from 2015 [1]. When this
happens in github, the result may be that someone forks your project
and makes a new one out of it.

Another thing from the migration that I think you should consider
is what you're going to do with your existing bugzilla content.
Are you going to keep using it or use the github issues for that?
Are you going to migrate open issues there? How about the links
in reports pointing to your svn repository?

Finally, as much as migrating to github can be interesting, you must
also consider not being tied to github in case any future policy changes
don't please you. You must be prepared to be able to migrate to another
system if needed and see how much of the github environment can follow you.
Can the issues follow you? How about the discussions in github? etc.
What parts of github are you going to use, which ones disable (and can
they be disabled?).

Other alternatives for github are gitlab. You can install an instance
of gitlab locally or use their repository. Upon the announcement of
MS buying github, some projects migrated to gitlab.

If it may help, I can put you in contact with the people in our
team that can describe more in detail how W3C is using github today,
the hooks, CI, how we backup it, and the advantages and limitations we
have found so far.

Hope this feedback helps,

--jos?

[1] https://bz.apache.org/bugzilla/show_bug.cgi?id=58025

1 2  View All