Options for Data Driven testing with Selenium IDE and RC

Tags: 

Listed below are some options for how to handle data driven testing with Selenium IDE or RC

Option 1 – “Sel Blocks” Add-on for Firefox

Your first option for data driven testing with Selenium IDE is downloading a Firefox plug-in called “Sel Blocks”, found here: https://addons.mozilla.org/es/firefox/addon/selenium-ide-sel-blocks/?src=search. That’s a very easy way to get started with Static, data driven testing. Some documentation on it can be found here: http://web.archive.org/web/20121011165150/http://wiki.openqa.org/display/SEL/SelBlocks. Basically, not only does it provide the IDE the much needed logic/condition functions it needs for more advanced scripting (like if/else, for/foreach/while, continue/break) it allows you to read in data from a static XML file of data for use in the IDE.

An example of the XML file might be a couple of entries defining users and their roles. For example:

                  <testdata>

                                <vars userid="dilbert" role="superuser" />

                                <vars userid="wally"   role="lazyuser" />

                </testdata>

A very simple and easy static XML file that any tester can format/control. In fact, the XML file could be automatically generated by a Stored Procedure, or many other applications, or even code. The limitation here, is that it is an add-on, not a JavaScript file that can be appended to the existing user-extension.js file for the IDE AND RC (or even WebDriver for that matter). However, consider what you are doing first before you feel you have a limitation here. For one, it is very unlikely that you’re going to gain ANY value in executing data driven tests on multiple platforms and multiple browsers. Data testing, 99.9% of the time, has little to nothing to do with the UI/interface testing. So … in general, a great deal of data driven tests can be executed quickly with the IDE in Firefox alone. However, it is still a limitation, and, depending upon what and how often the XML file is being generated, may be using or maintaining static data. But again, maybe you’re not … maybe you have a SPROC generating that XML file every day, every hour, or off of a trigger.  That’s really up to how it is implemented. So the only true limitation here, is that it is run in Firefox only. However, there is a work around for this too … in that the JS script that runs in the browser is totally accessible via the open-source project the developer created on github.com. It is conceivable that you can access the open source components, pop that script into your user-extensions.js file for both the IDE AND RC, and now you can run those functions cross platform and cross browser.

Option 2 – “DataDriven” user-extension for IDE AND RC

You can download this extension, and its additionally required extensions here: http://web.archive.org/web/20120928080130/http://wiki.openqa.org/display/SEL/datadriven. This is a pure JavaScript user-extension, in that you can append it directly into your user-extension for the IDE and RC. Note though that there is one version of the script for the IDE, and one version for RC, so make sure you append the right JavaScript to the right applications user-extension file. I don’t think I need to say much here, since the URL above should explain and provide examples. Basically, it reads in an XML file, same as Option 1. Either the XML could be static, or semi-static depending upon how it is generated.

Option 3 – Create your data files online in a Content Management System, or Share Point page

Share Point and TFS have the ability to generate data driven pages from database sources. It’s basically the same concept as the debug data Connexus generates, in that it is simply data exposed on a web page. Selenium IDE AND RC can read that data into an array, and then assert or verify on that data. I do this all the time. In fact, I do this as my test setup file or configuration file for the test, and store that in my Drupal Content Management System, so that I don’t have to edit the scripts to set the variables. I simply tell the script at the start of the run, to open the page with the variables, read them in, and use them in my test setup. Some ideas for what data I store there:

  • Target URL or domain
  • Login User Name and Password

A quick screenshot of one of the test suites setup data from a web page from the Drupal CMS I use for the Test Manager application I created:

 

In this example, I create some variables called “playtests” and “skipReporting”. Also note in this example that I can have different setup pages for each test environment. Very handy … so I don’t just have one configuration file for the script. Depending upon where the script is being run (which it knows by simply storing the URL of where the script was told to start, and depending upon that string, open a specific setup page), is how it determines which setup file to read. In this case, my batch file that kicks the script off in Selenium RC, was told to start in a particular domain (DEV). So I have multiple batch files for each domain. In my selenium script, I simply tell it to open this page and store the value for each of those variables:

And the value of those variables is used by Selenium to determine which test cases in the script I want to execute (for each test case file I give them a TCID, or test case ID value at the top of the script so that they have an identity and are basically parameterized). In this example suite, I have over 30 test case scripts. In this example test setup page, I have told the script to play them all, via the value “~ALL~” … or, I can have listed out which ones I want to play. Very helpful when in debug mode, or if you want to leave out lower level tests and allow the script to complete faster. I also setup a variable called “skipReporting”. I use that when I am developing scripts or debugging so that the script does not generate any reporting information in my Test Manager CMS. I don’t want junk data from debugging or prototyping to be collected because I use the pass/fail information from the reporting, as well as metrics on how fast/slow the test executed, and other things so that I can spot trends. I have a template reporting process that after each test case executes, it reports on its success, and other meta information data. Here is a screenshot of a view of that data that Selenium wrote to the CMS after executing each of its test cases:

As you can see, in this view, I show for each test case ran (row), it’s unique runID (which is really just a UNIX Epoch time stamp that increments by the millisecond, so the values will NEVER be the same, AND I can always determine the exact time in which the entry was made), When the report was posted (in case there was any latency between run time of the test case and report time), the environment the script ran in (which is also parameterized in other test setup pages I have, but not this one), the Feature Path of what the test case is executing against, the name of the Selenium Test Suite, the name of the test set (i.e., test case file), and of course whether or not the test completed. On a lower level, I have a related view that shows the status of each test case executed in the test case file, so I can pin-point exactly where the failure is.

And because this is done in a Content Management System like Drupal, I have the ability to use rules on content creation that says if any row reported is incomplete or a failure (on the test case level), to email me all of the details. That way, I get instant notification, can use Outlook rules to organize those emails, and still have all the data in a database to aggregate the data and spot trends.

For this example, I do not mean to imply (with the static variables I have setup for each suite setup) that you would be limited to static data. You are not. If SharePoint is generating the data for the page from a database then it is clearly dynamic. Real-time database data.

Option 4 – Acquire an off-the-shelf Web Query tool

If you want access to the raw database data, and don’t want to develop something, and don’t want to use a CMS or Share Point, than acquire a web-based query tool. There are many out there. Some are free, some are not. Having this as a separate application from Connexus, or any other application, means no integration or critical path questions about security or the priority of when it can get done … find a home, install it, lock it down and go.

Some examples:

Just be persistent in your Google search. There are many. An example of what one might look like:

Because all of this is web-based, Selenium can access it on-screen. You can created canned reports and call them directly (very fast) then read in the data you want as variables to use throughout your test, or paste in a canned query and execute it, or actually use the query builder. Because many of these tools are admin software, they have MANY extended caopabilities … like running SPROCS, comparing data, inserting, editing data, etc … right there in the browser. So not only can you setup and manipulate the data, you can quickly call a canned (pre-written) SQL query page directly in your Selenium IDE or RC or Web-Driver script, read those values and use them in testing. Now that data is real-time and dynamic.

Option 5 – Create a widget in your project that exposes the data to a web form so that Selenium can read it

We just did that in a project I am working on right now, and am expanding its usage across other projects. Here is a screenshot of its implementation:

Note that the links circled in red only appear if you are a developer/tester. Clicking on one of those links pops up a page for a canned SQL query of raw data from the database:

In this case, it is a list of all the phone numbers for each of the organizations they are related to (and was the “Debug All Phone Numbers” link from the first screenshot above it). Since its on a web page, selenium IDE or RC or whatever, can actually read the values, count the columns, rows, etc … and use those values in testing. It is the exact data returned if you used SSMS and did a query on “Select * from phoneNumber where blah blah blah”… just exposed on a popup page so Selenium can see it. This code never goes to production, and the links are not exposed or even in the HTML code, because security has prevented them from showing if you don’t have the Developer role. Even if you did … the files that do the query are not included in the production release.

These are only some of the options you have for doing data driven testing using a tester-friendly automation tool.

Related Articles: