Executing external test scripts from Matrix

There is a way to automate the calls to external scripts. The general idea is that the tests steps to automate will have a text starting with run:, followed by the name of a script to launch on the client side, and optional arguments.

There's a utility that must run on the client side, which will be able to:

  • retrieve all the tests (XTC) that need to run, under some conditions
  • launch the execution of the tests and send back the results to MatrixALM


Details

  1. The user set up some test steps with the syntax run: followed by the name of a script to run.
    Example in Linux:

    run: /home/scripts/script1.sh x y z

    Example in Windows:

    run: d:\scripts\script1.cmd x y z

  2. The user generates a test run containing some of the above. Let's say the test run has tests number from XTC-50 yo XTC-60

  3. The user calls the TestRunner program to create the test file (big list of all the tests to run):

    java -jar TestRunner.jar --grab --project=PROJ --category=XTC --start=50 --end=60

    this will generate a file called test.xml on the local directory, with the test steps detected in items XTC-50 to XTC-60.

  4. The actual run of the tests can now take place. The user enters the following command:

    java -jar TestRunner.jar --run --project=PROJ

    This will read the test.xml file and call each script. The script must generate on return a response file called response.xml (see syntax below).
    The TestRunner program will decode the response and send the ok or error code to MatrixALM, including some comments.
    It is possible to stop the whole execution at a particular step if a pre-condition is not met by specifying continue=no in the response file

TestRunner syntax

java -jar TestRunner.jar [--help][--grab|--run][--project=X][--category=X][--start=n][--end=n][--item=XTC-n][--config=configfile][--test=testfile]
  --help: displays this help
  --grab: generates a configuration file from a (set of) test execution items
  --run: read the test execution items from the testfile and run the tests
  --project=P: mandatory
  --category=X: if specified, changes the default category (XTC)
  --start=n --end=n: if both are present, the test will grab all XTC items from XTC-start to XTC-end
  --item=XTC-n: will grab a single item instead of a start-end
  --config=configile: the name of the config file to use (defaults to config.xml on the local folder)
  --test=testfile: the name of the test file to generate (defaults to test.xml on the local folder)

Syntax of the XML response we are expecting in the reponse.xml file

<execution result='ok' user='xyz' date='2015/10/25' continue='yes|no'>
   <details>(text for details)</details>
   <global>(optional global test result)</global>
</execution>

Syntax of the configuration file

In order to avoid having to pass the same identification data to the test line all the time (and for security) the credentials have to be stored in a configuration file called config.xml by default:

<runner>
    <instance>https://mycompany.matrixreq.com</instance>
    <user>mylogin</user>
    <password>mypassword</password>
</runner>

TestRunner.jar

Here is the utility (you may have to rename the downloaded file to TestRunner.jar):

TestRunner.jar

Search