Mailing List Archive

Do we need Selenium for anything anymore? (was: [Selenium] IDE test for regressing bug)
On Thu, Jul 7, 2011 at 11:23 AM, Sumana Harihareswara
<sumanah@wikimedia.org> wrote:
> Thanks for the test & test suite!  Just wanted to update you that it seems
> like fewer and fewer MediaWiki developers are interested in Selenium tests,
> and many are moving to other tools for automatic testing:
>

That being said: is there any reason to leave the Selenium support in
core anymore?

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Do we need Selenium for anything anymore? (was: [Selenium] IDE test for regressing bug) [ In reply to ]
On Thu, Jul 7, 2011 at 8:28 AM, Chad <innocentkiller@gmail.com> wrote:

> On Thu, Jul 7, 2011 at 11:23 AM, Sumana Harihareswara
> <sumanah@wikimedia.org> wrote:
> > Thanks for the test & test suite! Just wanted to update you that it
> seems
> > like fewer and fewer MediaWiki developers are interested in Selenium
> tests,
> > and many are moving to other tools for automatic testing:
> >
>
> That being said: is there any reason to leave the Selenium support in
> core anymore?
>

I believe work still needs to be done to provide a test harness that uses
QUnit for 'live-wiki' tests -- the current QUnit tests are standalone and
run from static files, so can't test interaction with server-side behavior.

With our current QUnit tests you can see if your JS code does something on
some input data or HTML you construct, but you can't, say, confirm that
issuing a click event on the watch tab submits an API request and comes back
with the expected result.

Interaction tests also tend to modify data -- by performing edits, creating
accounts, changing settings, deleting things, undeleting things, etc. So,
like the parser & phpunit tests that are potentially destructive, they
either need to run on dedicated testing sites where that's acceptible, or
the test harness needs to be able to set up / tear down a temporary wiki
copy. Unlike the parser & unit tests, that temporary copy has to be actually
exposed via the web so client-side tests can run.

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: Do we need Selenium for anything anymore? (was: [Selenium] IDE test for regressing bug) [ In reply to ]
Hello everybody,

that's also the way I see it. Selenium tests do test the application as a whole. While I agree that a lot of things that are currently tested with Selenium can be substituted with QUnit or similar, I still think that selenium tests should be used to check more complex interaction patterns. In BlueSpice, an extension I am maintaining, we use Selenium e.g. to test a file uploader or GUI elements that interact via AJAX with the server.

Also, keep in mind that selenium tests are quite easy to record. While experienced developers like us probably tend to write the tests instead of recording, I think that we can actually reduce the barrier for others.

Regarding the core code, I think we could easily remove the autoloader references from the core since they can be put into the SeleniumFramework itself. However, there is a call to includes/SeleniumWebSettings.php right after LocalSettings are loaded. This file's task is to reconfigure the wiki and its database and files when it is under test in order to start with a fresh or otherwise defined state. This mechanism is not specific to selenium, although currently only used by SeleniumFramework. For Selenium tests to make sense, I suggest this mechanism should stay in core. However, we could rename it to something more descriptive.

Btw, there will be a panel in Haifa where we want to discuss the test environment as a whole, if anyone's interested :)

Cheers,
Markus


-----Ursprüngliche Nachricht-----
Von: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] Im Auftrag von Brion Vibber
Gesendet: Donnerstag, 7. Juli 2011 19:43
An: Wikimedia developers
Betreff: Re: [Wikitech-l] Do we need Selenium for anything anymore? (was: [Selenium] IDE test for regressing bug)

On Thu, Jul 7, 2011 at 8:28 AM, Chad <innocentkiller@gmail.com> wrote:

> On Thu, Jul 7, 2011 at 11:23 AM, Sumana Harihareswara
> <sumanah@wikimedia.org> wrote:
> > Thanks for the test & test suite! Just wanted to update you that it
> seems
> > like fewer and fewer MediaWiki developers are interested in Selenium
> tests,
> > and many are moving to other tools for automatic testing:
> >
>
> That being said: is there any reason to leave the Selenium support in
> core anymore?
>

I believe work still needs to be done to provide a test harness that uses QUnit for 'live-wiki' tests -- the current QUnit tests are standalone and run from static files, so can't test interaction with server-side behavior.

With our current QUnit tests you can see if your JS code does something on some input data or HTML you construct, but you can't, say, confirm that issuing a click event on the watch tab submits an API request and comes back with the expected result.

Interaction tests also tend to modify data -- by performing edits, creating accounts, changing settings, deleting things, undeleting things, etc. So, like the parser & phpunit tests that are potentially destructive, they either need to run on dedicated testing sites where that's acceptible, or the test harness needs to be able to set up / tear down a temporary wiki copy. Unlike the parser & unit tests, that temporary copy has to be actually exposed via the web so client-side tests can run.

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l