Using Drupal AND Selenium for data driven testing


This may sound a bit strange to many of you ... but, we have engineered a way to create a framework for managing scheduled selenium launches, and providing data into each launch (data driven) using a Content Management System (Drupal). For  our needs, we needed the ability to have a separation between data and the scripts themselves. I should menton that we are a performance engineering team;although, this method has also been used for functional testing. We were looking for browser-based (not synthetic) single instance performance testing metrics. We wrote our scripts using object oriented test design methods and therefore one test case can be used to drive multiple test transactions. For example, for a normal menu system there may be the following items:

  • Home
  • About
  • Shop
  • Contact
  • Help

Rather then writing one test case for each item in the menu (and having to maintain five different test cases and thier code over time), we write one test case for clicking any item in the menu and parameterize the data going into the test. So for example, our topMenu-click test case accepts the following data: home, about, shop, contact, help. We can run the topMenu-click test case many times, providing data each time for the desired result. Rather then put the target data (i.e., any of the menu items in the bullet list above) into the script and having to change the script each time we want to use a different menu item, we create a webpage where we indicate what data we want to go into the script at runtime. In fact, we break the data up into two groups:

  • Suite Configuration Data: Which contains information on the description of the suite, whether or not to provide repoting information (because we may be in debug mode and dont want test transaction reporting data), and when to expire (from the database) and test data (since often, after the test, we dont want thousands of transactions hanging around for years ... the database would grow too large). This data also acts as a key to all of the test configuration data (explained below) associated with this suite configuration. For any selenium suite we create, we cann have one or more suite configurations ... so the same suite can serve mutiple purposes. Our performance testing team works with specific end-user workflows or use cases. An end user may go through our system (just like any system) in many different ways (i.e., multiple use cases and work flows). Those use cases/workflows would each get one suite configuration with specific tests defined. Maybe the first time through the system, the end-user needs to load the home page, login, go to the contact page, submit a contact request and log out. Maybe the second time through the system the end-user needs to load the home page, login, go to the shop, buy some items, and checkout. Perhaps the third time the end-user needs to do the same as the second, but cancel thier order. We would setup Suite Configuration 1 with the appropriate test cases, and name it something like "Contact". Suite Configuration 2 would be something like "Checkout", and Suite Configuration 3 might be something like "Cancel Order". However, all of them would use the same test objects, but with different data and in different order.
  • Test Configuration Data: This allows us to  provide dynamic data into the generic test object as well as set some runtime values for the test. For example, we allow for an SLA value, or Service Level Agreement value, where the object under test should respond in X number of seconds or faster, or have the result marked as a failure. By the way, test results are written to the database as well using a "Result" content type that we created to collect the data. This way, we are not bound to the Selenium RC HTML result fle, nor do we have to setup a third party like testNG just to gain control over the result information. We simply created a content type on our Drupal website, and had the selenium instance actually write the data to the webform. We then created a view to the data and could easily wire-in graphs. Anyway, here are some additional parameters we allow for the test configuration:
    • Test Name (used for pretty reporting)
    • Skip Test (whether or not we want to skip this test on any particular run)
    • Skip Reporting (We may not want metrics out of a particular test case)
    • Think Time (number of seconds the script should loop in idle simulating and end-user taking the time to read a page or piece of content)
    • Data Source (whether or not the data provided will be static or dynamic). Static data is a single piece of data that will not change for this suite configuration. An example of static data might be the value "Home" for the topMenu-Click test case above. In this suite configuration, we always want the first test case to be "Home" menu item to click on. Dynamic data would be something like a long list and randomized user id's to login the website with.
    • Precondition: A value to look for on the page to verify that the test is worth proceeding. If I want to execute the "Logout" test case, I would check for the precondition that the "Logout" link is available (which on our system, is only available to logged in users). Failure aborts the test and provides a notificaiton.
    • Value: This is the actual data value that would be used in the topMenu-click test case. It is the value for the actual test action being performed, whether that is a clickAndWait, VerifyElementPresent, or whatever may be the case for that particular test case. To execute the topMenu-click test case, and click on the "Shop" menu item, the value "Shop" would be entered here and passed to the selenium command: clickAndWait | css=a:contains("${field_tc_value_value}") |
    • Postcondition: Same as precondition, but the check is made AFTER the test to verify that you landed in the right area after executing the test. Failure aborts the test and provides a notificaiton.

We use the CallWebService extension during runtime (i.e., during the actual execution of the test case file) to call to our Drupal MySQL database and request these values by a Test Configuration ID (TCID) value provided on the Suite Configuration page for each test configuration.

Below is a screen-shot example of one of our Suite Configuration pages. You will also see the Test Configurations that are associated with the suite on the same page at the bottom:

 Once we have the Suite and test cases created and paramterized, and we have a Suite COnfiguration and associated Test Configurations created on the framework website, thenwe are able to create a reservation to schedule our launch. Our "Reservation" content type (Drupals term for the content architecture and the webform and fields for entering the content) has the following fields:

  • Status: Used to track the progress of the launch (New, Starting, In Progress, Completed AND Recur)
  • Suite Configuration: An automcompplete lookup field to relate the Suite Configuration to the Reservation content
  • Environment: Used to determine which SVN folders/files to use for this launch (Development, Test or Production)
  • Date/Time Start: The date and time the reservation should start
  • End Time: The time the reservation should end
  • End Date: The date the reservation should end
  • Runtime Settings:
    • Machines: What machines we want the reservation to launch on (and no, we dont use GRID to do that)
    • Vusers: The number of selenium instances we want to launch (so if we wanted to launch 10 users to run at the same time to generate a small laod on the system)
    • Interval: How far between in seconds each new selenium instance will start from the other
    • Floodgate: The number of failed instances before an email notification is fired off
    • Restart Delay Time: Once all of the initial instances of selenium are launched, how long in seconds before the restarted instances will start from each other

Here is a screenshot of our Reservation content type (form):

We have a tabelized view of all reservations where the date/time, status elements are exposed. A selenium script (checkSchedule.html) runs every 15 minutes to check for reservations that are scheduled within the next 15 minute timeframe (not before, and not after, but within the 15 minute window). If it finds a reservation, it views its content (node) page and clicks on a donwload link. Drupal has a nifty feature module called "Views Data Export" which allows database values to be tokenized and formed to create new content, and then provide that content in a configurable file format. We chose TXT, because we later want to convert that to a batchfile once the file has been downloaded to the target machine(s). Our test browsers (FireFox) are configured to autodownload files to a local directory (c:\selenium\2\downloads\) without any user interaction. The view is configured to name the file sch${nid}.sch, where nid is Drupals Node ID value (automatically assigned by the content management system) so that no two files will ever hav the same name, and therefore no files named sch1234 - copy(1).sch (as Windows does when it creates a copy of a file in the same directory).

Inside the text file (whihc will later be converted to a batch file on the local machine) is a batch command, like so:

call c:\selenium\2\batchfiles\startSelenium.bat 188 seleniumRecipesTrainingSuiteConfiguration suite-seleniumrecipes-trainingSuite.html development 1 1 1

Every minute, a batchfile on the local machine, named processDownloads.bat (see the attached file), will execute and move files with the mask sch*.sch to a "schedule" folder, rename the file to sch*.bat, and then execute it. In turn, that file will call into the startSelenium.bat file (see the attached file) with the arguments:

  1. 188 (reservation ID)
  2. seleniumRecipesTrainingSuiteConfiguration (Suite Configuration name)
  3. suite-seleniumrecipes-trainingSuite.html (suite file name)
  4. development (environment to run in)
  5. 1 (number of vusers)
  6. 1 (iteration)
  7. 1 (interval)

and that is how we start selenium without even being there.

That's just the tip of what you can do when you pair selenium with a content management system.


Related Articles: