Automated Web Testing Bliss: Integrating Selenium IDE/RC and FitNesse

Strategies for developing web application tests vary, but most are difficult to implement, maintain, or automate. In this entry, I show how to take a no-new-code approach to defining new testcases with 2 popular open source testing tools. Tests are recorded and executed in Selenium, but saved, maintained, and controlled in wiki-based FitNesse.

I've been a big fan of the Fit and FitNesse testing frameworks for about a year. As good as Fit is for writing and collaboratively maintaining tests, testing web app UIs was still very challenging. When I first tried Selenium and the Selenium IDE recorder application, I was excited, but not sold... I didn't want to give up what FitNesse was giving me in terms of collaborative, story-based test development that could include non-UI tests and database setup in the same page. I believe the most important quality of a testing framework is how hard it is to maintain it, and FitNesse made things very, very easy.

After seeing Jeffrey Palermo's blog entry on integrating Selenium RC and FitNesse to create Domain-Specific languages, I was inspired. I wanted the opposite goal, however: to create an entirely domain-generic combination of selenium and FitNesse, so that I could record web app tests in Selenium IDE, save them to FitNesse, and run them from FitNesse without having to write any new testing code (easy maintenance!). I wanted a Fit Fixture that could understand the output of Selenium IDE (click this link, verify this text, etc), and use it to drive Selenium RC automatically. Here's how I accomplished this with 1 Fit Fixture and a bit of JavaScript:

The Solution

Here's a summary of the goals of the system:

1. Use Selenium IDE to record test cases... no manual coding allowed!
   


2. Save the test cases to FitNesse, allowing easy team collaboration, documentation, and updates as well as inclusion data-driven operations like database setup defined in one browser-accessible place.


3. Execute the tests as suites in FitNesse, either manually from the wiki or automatically from one of the many interfaces to FitNesse (maven, ant, cruisecontrol, scripts)


4. Use Selenium RC automatically from FitNesse to run the UI portions of tests in both IE and FireFox


5. See the results in FitNesse


6. Never have to write or build test-specific code for UI changes

And here are the steps to get to that very scenario:


STEP 1: Create a (Single) FIT Fixture

The first step (after installing FitNesse, Selenium IDE, and Selenium RC if you haven't already, of course) was to define a DoFixture class to act as a proxy to Selenium commands. I picked a common subset of functionality to start out with, including opening a page, clicking links/buttons, setting form values, and verifying text. A partial listing of the fixture in C# is shown below.

public class SeleniumFixture : DoFixture {
public static ISelenium SELENIUM
{
  get { return _selenium; }
  set { _selenium = value; }
}

//...

public void StartBrowserPointingToHostOnPortWithURL(String browser, String host, int port, String baseURL)
{
  BASEURL = baseURL;
  BROWSER = browser;
  HOST = host;
  PORT = port;

  SELENIUM =
      new DefaultSelenium(HOST,
          PORT,
          "*" + BROWSER,
          BASEURL);
          SELENIUM.Start();
}

public void Stop()
{
  SELENIUM.Stop();
}

public void open(String url)
{
  SELENIUM.Open(url);
}

public void SetTo(String control, String value)
{
  SELENIUM.Type(control, value);
}

public void ClickAndWait(String where)
{
  SELENIUM.Click(where);
  SELENIUM.WaitForPageToLoad(15000);
}

public void Click(String where)
{
  SELENIUM.Click(where);
}

public bool VerifyTextPresent(String what)
{
  return SELENIUM.IsTextPresent(what);
}
}

This fixture class is just a very thin wrapper around the Selenium RC functionality. It contains no app-specific code, making it reusable across multiple web apps. To make the transition from Selenium IDE to FitNesse fixture easier, I generally created method names to exactly match the command names generated by Selenium IDE. In certain cases (for the sake of readability in the FitNesse pages) I chose different namings, such as the Selenium 'type' command, which fills in text fields by name with a value. In a FitNesse table by default, this would read as 'Type Username Josh'... good, but not completely obvious. I renamed this function in my Fixture as: setTo(String control, String value), which will become a much more readable "Set Username To Josh" in FitNesse.

To test this out, create a FitNesse wiki page with the following test table (substitute your environment-specific values for host/port/etc:

!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnet\FitServer.exe}
!path dotnet\*.dll
!|start browser|firefox|pointing to host|localhost|on port|4444|with url|http://www.google.com|
|open|/webhp?hl=en||
|Set|q|To|fitnesse+wiki|
|clickAndWait|btnG||
|verifyTextPresent|fitnesse.org|

Set the properties of the page to indicate that it is a test, and then run by clicking the 'Test' link. You should see a Firefox window pop up in the background and run a search, verifying that fitnesse.org is among the top 10 results returned.



You can see how easy it is to modify this test in Fitnesse... say that you want to also verify that the high-level categorical links are shown in the search result. Add the following lines to the end of your fitnesse test and re-run it... you've changed the test entirely from the wiki!

|verifyTextPresent|Download - fitnesse.org/FitNesse.DownLoad|
|verifyTextPresent|FitNesse.UserGuide - fitnesse.org/FitNesse.UserGuide|
|verifyTextPresent|A Two-Minute Example - fitnesse.org/FitNesse.TwoMinuteExample|

STEP 2: Extend Selenium IDE

All this is good, but now we need to bring Selenium IDE into the mix so that we can just record browser interactions instead of having to write all those wiki tables by hand. Selenium IDE has built-in support for saving recorded testcases as HTML as well as several variants of code snippets for Selenium RC (C#, Java, Perl, Python, and Ruby). Lucky for me, it also allows you to extend this set by defining your own formats. I'm going to define a 'FitNesse' format that will output the wiki tables for me directly. To do this, we need to write a bit of JavaScript code:




Go to 'Options->Options...' and click on the 'Formats' tab. From here, notice that all the supported language formats are listed. Each format is defined by some implementing some methods in JavaScript... click on 'HTML' and hit the 'Source' button at the bottom of the page to see an example of the code necessary for a format. Several methods are defined defined here, but just remember that the basic operations are to convert testcases to your format, and convert your format to testcases. Additionally, you can define XUL interface elements to control options for your format. Since I only need to get wiki text from my testcases, I'll write a partial implementation that does only the testcases->format conversion.

Click the 'Add' Button and type 'FitNesse' for your format name. Notice that Selenium provides the method stubs and comments for the methods you need to implement.




In the source pane, I defined the following JavaScript methods for the testcase->FitNesse conversion. If you get stuck, or want to implement more of the functionality, check out the HTML format code again or the online docs for reference. Of course, swap in your own specific values where appropriate:

 var verifyTextPresentCommandTemplate = "|${command.command}|${command.target}|";
var defaultCommandTemplate = "|${command.command}|${command.target}|${command.value}|";
var selectCommandTemplate = "|Choose|${command.value}|from select|${command.target}|";

/**
* Format TestCase and return the source.
*
* @param testCase TestCase to format
* @param name The name of the test case, if any. It may be used to embed title into the source.
*/
function format(testCase, name) {
 return formatCommands(testCase.commands);
}

function getCommandTemplate(commandStr) {
 if (commandStr == 'type')
     return typeCommandTemplate;
 else if (commandStr == 'verifyText')
     return verifyTextCommandTemplate;
 else if (commandStr == 'verifyTextPresent')
     return verifyTextPresentCommandTemplate;
 else if (commandStr == 'select')
     return selectCommandTemplate;
 return defaultCommandTemplate;
}

function getSourceForCommand(commandObj) {
 var command = null;
 var comment = null;
 var text = '';

 if (commandObj.type == 'command') {
     command = commandObj;
     command = commandObj.createCopy();
     var template = getCommandTemplate(commandObj.command);
     text = template.replace(/\$\{([a-zA-Z0-9_\.]+)\}/g,
         function(str, p1, offset, s) {
             result = eval(p1);
             return result != null ? result : '';
         });
 }

 return text;
}

/**
* Format an array of commands to the snippet of source.
* Used to copy the source into the clipboard.
*
* @param The array of commands to sort.
*/
function formatCommands(commands) {
 var commandsText = '';
 commandsText = commandsText + '!|josh.SeleniumFixture|\n';
 commandsText = commandsText + '|start browser|firefox|pointing to host|localhost|on port|5555|with url|http://localhost/|\n';
 commandsText = commandsText + '!';

 for (i=0;i<commands.length;i++) {
     var text = getSourceForCommand(commands[i]) + "\n";
     commandsText = commandsText + text;
 }

 return commandsText;
}

With that defined, we're really ready to go. Go to Google in your browser and start a new testcase in Selenium IDE. Under 'Options->Format', choose your newly created FitNesse format. Click around on Google in the advanced options, generate a search page, and highlight some text, right-click, and choose 'verifyTextPresent...' from the context menu. Now stop the test and click the 'Source' tab of Selenium IDE. Notice that it has nicely generated complete wiki table entries ready to insert into FitNesse!






Copy the text, create a new FitNesse wiki page, and paste it in (be sure to define the command and path variables and add a command to start the browser, or modify the script to do so)... from here on out you can create new tests or update old ones in FitNesse just by recording your browser sessions with Selenium IDE.

Having the combination of the Selenium IDE recorder and FitNesse wiki-editable formatting is a great thing. When the UI changes or grows, all that is required to add or update the tests is to record using the changed pages or edit a wiki table directly. FitNesse lets me collaborate with other team members, write extensive inline documentation, change tests independently of build cycles, do data-driven test setup, and reuse test fragments via suites, includes, and symbolic links. Finally, driving Selenium RC with FitNesse gives you multi-browser code coverage with a wide variety of hooks for automation and continuous integration tools. There's still work to be done to fill out the fixture with full Selenium RC functionality, but you only have to add functionality according to your needs from Selenium, not when your web app changes.