Functional testing

From Complete Cyclos documentation wiki
Jump to: navigation, search

There are a collection of functional tests that are run to ensure a minimal behavior of the application. This section describes how to install the required environment and the steps needed to run and extend this tests. There are two approaches, the Selenium environment is used to test core functionality surfing Cyclos web front end from a browser; the second approach (using SoapUI) is to test core functionality that is exposed through web services.

Selenium Environment


  • Open firefox and go to the following URL: to install the plugin.
  • It is also important to have the Firebug plugin, you can install it from within firefox.
  • Optionally, to run the tests that have already been created, install the database located in cyclos3_testbed_X.X/selenium/db folder and change the file of your cyclos installation to point to this database.
  • Add the user extensions to the Selenium IDE. To do this, open the Selenium IDE (Firefox: Tools > Selenium IDE), Options > Options ..., add the path to the user-extensions.js file cyclos3_testbed_X.X/selenium/user-extensions.js in the Selenium Core Extensions section.
  • Install the MKP- Selenium IDE tools (that installs a set of selenium ide plugins)


The basics

Selenium IDE

Open the Selenium IDE: In Firefox, Tools > Selenium IDE.

  • Left Panel

By default, Selenium starts with a Test Suite containing a unique Test Case named 'Untitled'. You can create new test cases through the top menu or the context menu of the left panel.

  • Right Panel

At the right panel Selenium displays the content of the selected test case. To select a test case you must double click the test case, otherwise the test case will be highlighted and the content of the test case won't appear at the right panel. The right panel contains two tabs, Table and Source. The Table tab (when it is enabled) displays the list of commands to be executed by the browser. The Source tab displays the list of commands in the selected format. By default, the selected format is HTML. You can switch between different formats through the Options menu. Note that the Table tab will be enabled only when the HTML format is selected.

  • Base URL

At the top of the IDE, there is a text box which you need to fill with a base URL (e.g: http://localhost:8080/cyclos/do/). Certain commands need an URL as an argument. The URL will be relative to the base URL. e.g:

    Command      Target    
    open         login        (login is the relative path, the absolute URL would be: http://localhost:8080/cyclos/do/login).

The base URL is specific to each test case and it's not global to all the test cases. Modifying the base URL located at the top will modify only the selected test case's base URL.

  • Recording browser events

To start recording click on the recording button (red bullet) and start browsing. After a few actions with the browser click again on the recording button and look at the commands that have been detected.

That's it!. To replay your actions click on the green play button (there are two, one to replay the entire test suite and other to replay the selected test case).

  • Saving test suites and test cases.

Once the test case is finished it can be saved to a file through the menu or the context menu over the test case. A test suite only contains references to test case files. Saving a test suite prompts the user to save every unsaved test case, and after all, prompts to save the test suite file.

Not everything is so easy

Sometimes Selenium IDE doesn't catch all the actions done in the browser and some commands have to be added manually in order to create a complete test case. To do this, open the Table tab and add them in the places you need, helped by the context menu.

Selenium can't handle alerts that rise in the onLoad event of a page. If this happens, the test gets stucked waiting for something to "consume" the alert. In Cyclos this alerts are everywhere, the workaround is to change the javascript function showMessage (head.jsp) so that a div is displayed instead of an alert. Yes, your Cyclos must be patched.

   function showMessage() {
       if(booleanValue(readCookie("showMessage"))) {
           alert('${message}');        -----> this line has to be changed by: showMessageDiv('${message}');
           messageDiv = null;          -----> this line should be added.


Many commands need to refere to an element in the page. The reference is done giving a locator. For example: the click command needs to be done on some element. To write the locator it is very useful to inspect the element you want to locate. To inspect the element, right click the element in the browser and choose inspect element (firebug must be installed first). Locators are ways of locating an element in the page. For example you could locate an element giving an XPATH: (//input[@id='username']). Click on the left panel examples to learn basic XPATH constructions in this tutorial: There are serveral other ways to locate an element, someones much more easier. They are all explained here: Reeding this reference is a must!.

Tips & tricks

  • Execute a single command in the Selenium IDE by double clicking the command row.
  • Use the find button on the command tab to verify if the written locator is correct. If it is, the element in the browser will be highlighted; if it's not, an error will be notified in the log tab of the selenium ide.
  • Use the context menu on the browser over an element to add commands related to that element directly to the Selenium IDE. For example, you could select some text in the browser and add an assert to that text through the context menu.
  • *AndWait: All the commands that ends with AndWait tells selenium to wait for a page to load after the action is done. For example, clickAndWait when submitting a form. If you only use click command, the test will continue without waiting for the page to load and surely your test will fail.
  • typeKeys, waitForValue: There are many pages that uses an input element where the user starts typing the code of something and the system autocompletes that element and loads the name in another input element. It is useful the typeKeys and waitForValue commands for this cases. The typeKeys is different from the type command. The typeKeys is exactly what a user does. However, the type command is faster and almost always work as it's expected. The waitForValue tells Selenium that it has to wait for some value in some element. To be more specific: if the input component hears key events to do something, use typeKeys. Otherwise, you could use type. Note that in some versions of Firefox, a direct call of typeKeys over an element doesn't work. To ensure the correct behavior insert a focus command before the typeKeys to gain the focus of the element.
  • assertTextPresent: Asserts a text in the whole page.
  • assertValue: Asserts an element value.
  • select: Use select command for combo boxes instead of simulating the select with clicks.
    Command      Target       Value
    select       age          label=30
  • storeValue: Use the store commands to save in memory particular values that can be referenced in the following commands. For example, when a new ad is created, the page contains in a hidden input element (look at the page source code) the ad's id given by the system. Store this value to locate further in the list of ads which remove button you must click to remove the recently created ad.
  • You can execute javascript within commands (using javascript{<javascript expression>}) or by the runScript command. There is also a shortcut to embed a variable value within a command expression, this is done using the ${<variableName>} expression.
    Command      Target       
    clickAndWait //img[@class='remove' and @id=${adId}]
  • runScript: Rich text editors can't be handled by selenium directly but you can manage it by accessing the components API with javascript. For example, the description of an ad is introduced in a rich text editor. The following command writes in the component.
    Command      Target
    runScript    FCKeditorAPI.GetInstance( 'descriptionText' ).SetHTML( 'The new ad description' )
  • storeEval: Use this command to store in a variable the result of a javascript expression. Example: If you type 'albert' as the username when creating a member, in the next execution the test will fail cause there is another member with the same username. You need to put something that is generated each time you execute the test.
    The solution is to create a variable with the long representation of the actual date. This will be unique each time you execute it.
    Command      Target                  Value
    storeEval    new Date().getTime()    varUsername
    type         username                member${varUsername}     this will produce user names like: member126436363543

Custom Commands

This commands doesn't come as built-in commands, they are part of our own command set and are implemented in the user-extensions.js file.

  • storeURLParameterValue: It's aim is to store a parameter value extracted from the actual URL. Generally, when a form is submitted, a new page is loaded and the id of the saved element appears as a parameter of the URL. In this cases the command becomes useful.
    Command                   Target                Value
    storeURLParameterValue    parameterName         variableName
  • storeActualDate: This command stores in a variable the actual date in the locale format.
    Command                   Target      
    storeActualDate           variableName
  • storeTestCaseFolderPath: This command stores in a variable the absolute path of the folder where the current test case is stored.
    Command                   Target      
    storeTestCaseFolderPath   variableName
  • login: This command logs in an user. It's assumed that the user being logged in has 1234 as password. Takes two arguments: user and member. In case of administrators and members only the user is used. In case of operators, the member is used as the operator's owner.
    Command      Target                  Value
    login        admin
    login        member1
    login        operator1               member1

Note: If you want to export test cases that have custom commands to a certain language like java, the appropriate implementation for this language must be added, but for now this explanation is beyond the scope.

Test standards

  • Cyclos should be switched to English language before recording your tests. It's very common to assert text after an action, the text should be always in English. The best would be to make assertions on elements and not on text. Text is more prone to change than element names for example.
  • Write comments (option in the command context menu) so that the commands get tighted in explained sections. Probably, this comments will be extracted automatically by a tool to create a doc test reference. Think that the comments should say exactly what the test does.
  • Write tests that can be executed any number of times. The tests should be smart enough to test that the functionality is correct and that a new run of the same test needs not to be run on a "reverted" database. For example, if the test asserts the text present "The ad with id 7 was created successfully", it won't succeed the next time it runs because the system will give id 8 to the new ad. You could assert some part of the text helped by regular expressions.
  • Try to make simple command arguments. Sometimes when recording, Selenium creates commands with locators that are unreadable or more prone to brake with a little change in the structure of the page.


   Command                   Target
   assertElementPresent      css=table.defaultTable > tbody > tr:nth(10) > td:nth(1) > span.customFieldContainer > input[name=member(customValues).value]
   Inspecting the element you'll find that the locator can be much more simple, it only takes a few seconds more and you'll make your tests more maintainable. 
   The above example could be easily changed by the following command:
   Command                   Target
   assertElementPresent      //input[@fieldname='city']
  • If the test case needs to open a certain page, don't record the clicks you do over the menu to open it. Instead use the open command with the direct URL. If Selenium records the clicked elements done in the menu, probably it will locate the menu items by it's position on the menu, and we all know that the items in the menu vary depending on the configuration.

The login example

  • Go to the browser and open the Cyclos login page (e.g: http://localhost:8080/cyclos/do/login)
  • Create a new test case.
  • Start recording
  • Go to the browser and type the login, password and submit the form.
  • The following commands should be added automatically:
   command         target                value
   open            /cyclos/do/login
   type            cyclosUsername        admin
   type            cyclosPassword        1234
   clickAndWait    css=input.button

Sometimes the autocomplete of the browser interferes with the recording and the type cyclosPassword command could have been skipped. Manually insert the command right clicking in the commands window and choosing the appropriate option.

Created tests

There are many tests that have already been created, they can give you an idea of how to do some other things not covered in this introduction. There is a tool (jar) that can collect all the test cases from a folder and it's subfolders and make a single test suite to run them all. This jar is called testSuiteGenerator.jar and is located under the tests folder. Execute: java -jar testSuiteGenerator.jar -help from a command line in the cyclos3_testbed_X.X/selenium/tests folder to see all the generator options. The test cases are proven to be successful with the database located in the db folder.

User extensions

To add new commands append the implementations in the user-extensions.js file. Some considerations:

  • the name of the function needs to follow this structure:<NameOfTheCommand> = function([firstArgument[, secondArgument]]){
       //implementation goes here
  • when a change to the user-extensions.js file is done you must restart Selenium IDE for the changes to take effect.
  • You can have multiple javascript extensions. Just add them separated by commas.

SoapUI Environment


SoapUI is an open source functional testing tool. To download and learn SoapUI follow this link:


SoapUI by itself doesn't provide the easiest way for us to create web service tests, but offers all the necessary tools for us to do it. So part of this document describes how to write web service tests in a simple way just using an OpenOffice Calc spreadsheet and generating a file that consists on the input that SoapUI takes to generate and run the tests. When designing this tool we focus directly on the target of what we wanted, and that was to specify tests that:

  • are easy and fast to implement
  • are easy to maintain
  • can be run any number of times without having to restore the database
  • are easy to run

In essence, we need to fill input values of a request, execute the request, make some assertions on the response and probably get some data from the response to be used in other requests.

Specifying tests

As we mentioned in the introduction a spreadsheet is used to write tests. In this section you'll learn how to specify a web service test in the spreadsheet and generate the properties file that SoapUI needs. file

Global configuration is established in the file:

  • ts.execution is a property that indicates wich are the filenames (without the extensions) that take place in the generation. It's a list separated by commas.
  • ts.cyclosURL is a property that indicates the base url of the cyclos instance.
  • ts.numberOfLastCasesToGenerate is a property that indicates what number of last cases the generation will gather.
 ts.execution=PaymentsTestSuite, POSTestSuite, MemberTestSuite

In the above example, there should exist three spreadsheets with some tests in each. Only the last two test cases will be generated for each spreadsheet.

The spreadsheet

The spreadsheet contains a macro, so a dialog message appears when the spreadsheet is opened asking if you want to enable macros. Please select “yes”. If no message appears probably the macro will be disabled and you won't be able to generate the properties file. To change this settings go to Tools > Options... > | Security | Macro Security button and select the Medium security level. Then try opening the spreadsheet again.

Two main sheets are in the spreadsheet: Input and Configurations. Configuration contains general properties for all the tests we will be specifying in the Input sheet.

The Configuration sheet

Here you can specify general configurations for all the test cases.

Figure 1: Configuration sheet

Comments can be added by adding a “#” character behind the name of the property in the appropriate row of the Property column. In the above example only three interface URLs are defined, the first group of URLs will be not taken into account since the “#” prefix precedes the property names.

Interfaces or WSDL URL properties

In the figure 1 we declare some WSDL URL properties. This properties follow a specific syntax. The WSDL URL property name consists on three sections separated by a dot “.”: “conf” meaning its a configuration property “url” meaning its an URL property wsdl definition name

The wsdl definition name can be taken from the wsdl definition.

Figure 2: WSDL definition accessed with a browser

The value of the property consists on the relative URL of the WSDL.

Number of decimal places property

This property declares the decimal places when formatting a number. See Format References for more on formatting numbers.

The Input sheet
Generating the property file

To generate the properties file (the final input file for SoapUI) click on the button Generate Properties in the toolbars section. The number of test cases that will be generated is established in the file described before, if this property is omitted all test cases form the sheet will be generated.

Figure 3: Generate properties toolbar
General terms

We will go through an example to understand all the concepts. But first we'll introduce some terms.

  • Operations

An interface or WSDL may contain several operations. For example, the payments interface has the doPayment and the reverse operations.

Figure 4: An interface with its operations in SoapUI
  • Input parameters

An operation can have multiple input parameters. For example, the doPayment operation has amount currency description etc... So whenever an operation is invoked (a request is done), all the input parameters of the request must have the appropriate values.

This kind of parameters are declared in the Input sheet using an “in” prefix as we'll see further on.

  • Output parameters

The result of a request operation is a response. The response comes with certain data on certain fields. The values of this fields are what we want to hold in output parameters. For example, the field status on the response of a doPayment tell us if the operation has succeeded or not. If we would like to store this value we need to declare an out parameter in the Input sheet.

This kind of parameters are declared in the Input sheet using an “out” prefix as we'll see further on.

  • Authentication parameters

Operations needs to carry authentication information on the request headers. This kind of parameters are declared in the Input sheet using an “auth” prefix.

  • Assertion parameters

When the response is received, validation stuff can be done. There are different kinds of validations: Validate there is a response from the server. Validate that the response is in the format (xml) that we expect. Validate that in the response doesn't come a soap fault message. Validate that the response contains certain data (e.g: a SUCCESS text in the status field).

The first two validations are done without having to explicitly declare them in the Input sheet but the last ones needs to be specified because it depends on the test context.

This kind of parameters are declared in the Input sheet using the rude “ass” prefix as we'll see further on.

Parameter conventions

Lets look at an example to understand how to specify the above parameters in the Input sheet. Suppose our first test case consists on making a payment. An user member2 wants to pay to member1 1 dollar.

We will specify some parameters for the doPayment operation in the Input sheet.

Figure 5: Do payment specification

Some parameter values will be explained in the next sections, don't worry if you don't understand everything by now.

  • Operation name prefix in parameter names

Each operation must have a unique name that identifies the operation. In the above case, dp1 is the operation identifier.

  • auth parameters

When a Cyclos web service client is configure to use http credentials, the operation request must carry this credentials. To accomplish this, authentication parameters must be specified for the request. In this case, the web service client's user and password are 'member1'.

The parameter type is “auth” because it's an authentication parameter.

   The syntax for auth parameters is:

The values are taken from the sheet and placed encrypted in the header of the operation request.

  • in parameters

The values of the input parameters are placed in the operation request.

Figure 6: Fragment of request with some parameters
   To specify input parameters the syntax is:
       in.<Name of the parameter>
   where <Name of the parameter> is the exact name of the request parameter. If an invalid parameter name is specified the SoapUI will detect the problem.

Sometimes the requests have multiple section parameters as in doBulkPayment. In this operation you can specify more than one section of params. So we could have something like this:


To refere to the second field amount you need to be more specific in the naming of the parameters.

   The general syntax for in parameters is:
       in.<tag1[X1]>.<>...<>.<tagN[Xn]>.<Name of the parameter>
       in.params[2].amount to refere to the second amount parameter.
   The parser looks for the following xpath in the request: //params[2]/amount.
   This aplies to all in parameters.
  • out parameters

The out parameters are used to hold values from responses and that can potentially be used for filling input parameters or storing important information to be processed later.

Unlike the in parameters, out parameters have an arbitrary name.

   The syntax for out parameters is:
       out.<Name of the parameter>
   A name example:
   The value of the property is the xpath of the tag from where to take the value to fill the property. The xpath is specified between “#{xpath: ” and “}” like in the following example:
   A value example:
       #{xpath: //transfer/id}
  • ass parameters

The ass parameters are used to make validations over an operation response. Following the example described above, we would like to halt the test if we find that the response is a soap fault or if the status code isn't “PROCESSED”.

   The syntaxes for ass parameters are:
       ass.notSoapFaultAssertion If the value is true, then a response will be valid if the response isn't a soap fault.

This two properties are combined and the above one doesn't have to be necessarily an xpath expression but can be a whole complex expression. Let's describe some examples:


The last row of the above examples shows how we can make references to other properties. For more information on xpath assertions follow this link:

General notes on parameters
  • NULL value in properties

If we specify the NULL value for a parameter, then: if it is an in parameter, the tag in the request will be removed. If it is an auth parameter, the Authentication header will be removed. If it is an out parameter, then the parameter will not be created for that case.

  • Parameter references #{p: <content>}

In all property values we can make references to other property values. For example, we could reference the trace number property of the doPayment operation in the reverse operation. The property should contain the value: #{p:} #{p: <content>} indicates that the content should be a property name and it's replaced by the value of the specified property.

  • Script references #{s: <content>}

After evaluating in references, script references are evaluated. Script references are available to compute math expressions or whatever groovy script you may want. Example: To test the credit limit, we could ask for the available credit and make a payment of that amount plus one cent. The system should return a credit exceeded message. So property should contain the value: #{s: 0.01 + #{p: sah.out.availableBalance}} #{s: <content>} indicates that the content should be evaluated as a groovy script.

sah.out.availableBalance has the available balance amount returned by the search account history operation invoked before.

  • Format references #{f: <content>}

Sometimes we need to format a number with a certain amount of decimal places. For this purpose we can use the format reference like in the following example: #{f: #{p: sah.out.availableBalance}} if sah.out.availableBalance contain 0.0466 and the declared decimal places is 2, then the result will be 0.05. The decimal places for the format reference needs to be declared in the configuration sheet in the conf.numberOfDecimalPlaces property. #{f: <content>} indicates that the content is numeric and should be formatted following the conf.numberOfDecimalPlaces property.

  • Date references #{d: long}

Sometimes we need a generated unique number. For example, if a test case registers Member 1 with username member1, the first time the test will succeed, but in the followings the member will be already registered and the test case will fail if the test case asserts always the status "Member registered". The test case must be smarter and register a new member each time it runs. To accomplish this, the username could be an incremental number computed each time. This incremental number can be obtained with a Date reference. There are many created tests that use this approach. Check them to see how this is easily done.

Running tests

SoapUI application

In this section we'll explain the main application components and what objects we need to be created before and during the generation and execution of tests. Lets look at a project example:

Figure 7: Different objects of a project

The interfaces are the WSDL definitions, contains the URL of the definition and you can update an interface definition by accessing the contextual menu stepped on the interface. By convention, the name of the interface is the name that is specified in the WSDL appended by the “SoapBinding” string. This interfaces are created automatically based on data entered in the Configuration Sheet as we shall see in the next sections.


An operations is a definition containing input and output types of data it has to carry to and from the server, and a name to identify the business logic the server has to invoke.

Test suites

A test suite is a collection of test cases.

Test cases

A test case is a collection of test steps.

Test steps

A test step can be of several types. The most frequent we'll be using is the “Test Request” step.

Figure 8: A test case with six request test steps and a property step

In the above image, the SOAP steps are executable requests. The first step is a Properties step and is used to store and load properties that are used by the SOAP steps.


Scrip test step

There is another type of step: the Script Test step. The “Generator” scrip test step is located in the “All Test Case” test case and we use it to lunch the generation and execution of steps.

Figure 9: Generator script test step

In the above image, when you double click the Generator step, a window is opened with the script. To lunch the script click on the green arrow button at the upper left corner.

How generation works

There is a test suite generator called 'TS - Generator Test Suite'. This is a special test suite that generates and executes the rest of the test suites. To run this test suite (and generate and run the rest of the test suites) double click the test suite and click on the green arrow button at the upper left corner. For each test suite name declared in the file, a test suite is generated.

Figure 10: Generator scripts in each generated test suite

If you want to generate and run a particular test suite, just run the test suite. If you want to run a particular test case, just run the test case.

Test steps are run in descending order, so in the above example the first step to execute is the Generator step and then the first SOAP step below the properties step (properties steps cannot be executed).

Figure 11: Run of generated test cases and test steps

After running the Generator test step, a test case is created for each column specified in the Input sheet. There is also a test step request for each specified operation and a properties step with all the properties for that particular test case. The generator test step window has an output log where we can read the messages of the overall run. The test steps start in white colour indicating that the step didn't run yet. When the step is run it turns to green if there where no errors or red otherwise. To see the request and the response of a step just double click on the step and a window (like the above in the image) will show up. You can double click on the assertions to know it's content.