Test Cases¶

Overview¶

Test cases are the means by which a specific testing scenario is implemented. One or more test cases form the content of a test suite. The following example represents a complete, simple test case for the validation of an invoice that is uploaded by a user.

<?xml version="1.0" encoding="UTF-8"?>
<testcase id="UBL_invoice_validation_test_1" xmlns="http://www.gitb.com/tdl/v1/" xmlns:gitb="http://www.gitb.com/core/v1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.gitb.com/tdl/v1/ ../../gitb_tdl.xsd">
    <metadata>
        <gitb:name>UBL_invoice_validation_test_1</gitb:name>
        <gitb:type>CONFORMANCE</gitb:type>
        <gitb:version>1.0</gitb:version>
        <gitb:description>Test case to verify the correctness of a UBL invoice. The invoice is provided manually through user upload.</gitb:description>
    </metadata>
    <imports>
        <artifact type="schema" encoding="UTF-8" name="UBL_Invoice_Schema_File">UBL_invoice_validation/artifacts/UBL/maindoc/UBL-Invoice-2.1.xsd</artifact>
        <artifact type="object" encoding="UTF-8" name="BII_CORE_Invoice_Schematron_File">UBL_invoice_validation/artifacts/BII/BII_CORE/BIICORE-UBL-T10-V1.0.xsl</artifact>
    </imports>
    <variables>
        <var name="file_content" type="object"/>
    </variables>
    <actors>
        <gitb:actor id="User" name="User" role="SUT"/>
    </actors>
    <steps>
        <!--
            Step 1. Request the user to upload the UBL invoice.
        -->
        <interact desc="UBL invoice upload" with="User">
            <request desc="Upload the UBL invoice to validate" with="User" contentType="BASE64">$file_content</request>
        </interact>
        <!--
            Step 2. Validate the uploaded invoice.
        -->
        <verify handler="XSDValidator" desc="Validate invoice against UBL 2.1 Invoice Schema">
            <input name="xmldocument">$file_content</input>
            <input name="xsddocument" source="$UBL_Invoice_Schema_File"/>
        </verify>
        <verify handler="SchematronValidator" desc="Validate invoice against BII2 CORE restrictions for Invoice Transaction">
            <input name="xmldocument">$file_content</input>
            <input name="schematron" source="$BII_CORE_Invoice_Schematron_File"/>
        </verify>
    </steps>
</testcase>

The following table provides an overview of the attributes and child elements that a testcase may have. A more detailed discussion per case follows in the subsequent sections.

Name	Required?	Description
@id	yes	A string to uniquely identify the test case by. This is referenced in the test suite XML.
metadata	yes	A block containing the metadata used to describe the test case.
namespaces	no	An optional set of namespaces to define the expression languages used in the test case.
imports	no	An optional set of imports used to load additional resources from the test suite.
preliminary	no	An optional set of user interaction steps to display before the test session starts.
variables	no	An optional set of variables that are used in the test case.
actors	yes	The set of actors that this test case refers to.
steps	yes	The sequence of steps that this test case foresees.
scriptlets	no	Optional named groups of test steps which can be used within the test case.

Elements¶

We will now see how a test case breaks down into its individual sections and discuss the purpose of each.

Metadata¶

The metadata element is basically the same as the one defined for the test suite Metadata. Its purpose is to provide basic information about the test case to help users understand its purpose. Its structure is as follows:

Name	Required?	Description
name	yes	The name of the test case that is used to identify it to users.
type	no	Either “CONFORMANCE” (the default) or “INTEROPERABILITY”. “INTEROPERABILITY” is used when more than one actor are defined as SUTs.
version	yes	A string that indicates the test case’s version.
authors	no	A string to indicate the test case’s authors.
description	no	A string to provide a user-friendly description of the test case that is displayed to users.
published	no	A string acting as an indication of the test case’s publishing time.
lastModified	no	A string acting as an indication of the last modification time for the test case.

Note

GITB software support: Contrary to a test suite’s name, the name of the test case is recorded but not otherwise used. Matching of test cases and display uses the test case’s id attribute. Regarding the test case type this must currently be set to “CONFORMANCE” (the default value) as the “INTEROPERABILITY” type is not supported. Finally, the version, authors, published and lastModified values are recorded but never used or displayed.

Namespaces¶

The namespaces optional element is used to define one or more expression languages that are used in test case constructs that support them. This needs to be done when expressions are used that should not be processed using the default XPath 1.0 language. A detailed discussion on GITB expressions as well as where and how you can use them is provided in Expressions. Each referred expression language is defined in a namespace element with the following structure:

Name	Required?	Description
ns	yes	One or more elements to define the expression languages used in the test case. Each `ns` element must specify a `prefix` attribute to define the namespace prefix.

The values specified in the prefix attributes define how each expression language is referenced within the test case. As an example consider a test case that needs to check a number variable var and set a result variable with “result1” if var is 1 or with “result2” otherwise. Simple if-else constructs like this are a good example of a shortcoming of XPath 1.0 as they are not natively supported and need a non-intuitive workaround based on string concatenation. A simpler approach would be to use a scripting language more adapted to this kind of operation such as JavaScript. The following test case illustrates this assignment using both the default XPath 1.0 and JavaScript:

<testcase>
    <namespaces>
        <ns prefix="JavaScript"/>
    </namespaces>
    <variables>
        <var name="var" type="number"/>
        <var name="result" type="string"/>
    </variables>
    <steps>
        <!--
            Assignment using the default XPath 1.0.
        -->
        <assign to="$result">concat(substring('result1', 1 div number(boolean($var = 1))), substring('result2', 1 div number(not(boolean($var = 1)))))</assign>
        <!--
            Assignment using JavaScript.
        -->
        <assign to="$result" lang="JavaScript">if ($var == 1) { return 'result1' } else { return 'result2' }</assign>
    </steps>
</testcase>

Needless to say the assignment using JavaScript is much more natural and easy to understand. However use of alternative expression languages, and their definition through the namespaces element, is tricky because we need to know exactly how the target test bed refers to the language (i.e. “JavaScript” in our case) to correctly identify it. Furthermore it must be clear how the test bed will process the expression and how session context variables are looked up. In the above example we assume that context variables (e.g. “$var”) are looked up in exactly the same way as with XPath 1.0 expressions and that the entire expression will be evaluated by first wrapping it in a function, the result of which is returned as the assignment output. Apart from actually supporting JavaScript for expressions, these additional details need to first be defined unambiguously by the test bed and made known to its users. Only then can we use them in a deterministic and portable manner.

Note

GITB software support: Expression languages other than the default XPath 1.0 are not supported. As such the namespaces element is currently ignored.

Imports¶

The imports element allows the use of arbitrary resources that are present in the test suite. This can be very useful when a test case needs to send messages based on a template or load a binary file that is needed as input by a messaging, processing or validation handler (e.g. a certificate). The imports element defines one or more artifact children with the following structure:

Name	Required?	Description
@name	yes	The name with which this artefact will be associated to the test session context for subsequent lookups.
@type	yes	The type as which the artefact needs to be loaded.
@encoding	no	In case the artefact is to be treated as text	this is the character encoding to apply when reading its bytes (default is “UTF-8”).

The text value of the artifact element is the path within the test suite from which the relevant resource will be loaded. Regarding the type attribute, this needs to refer to an appropriate type from the GITB type system (see Types). Given that in this case we are referring to a file being loaded, the types that can be used are:

binary: Load the artefact as a set of bytes without additional processing.
object: Load the artefact as a XML Document Object Model. In this case it is best to also explicitly provide the encoding to consider.
schema: Load the artefact as a XML Schema. As in the object case it is best to explicitly provide the encoding to consider.

Regarding the path to the resource, this considers as the root the name of the test suite, followed by the resource’s path within the test suite archive. As an example consider the following test case fragment where a XML schema is loaded and set in the session context as a variable of type schema that is named “ublSchema”. The path specified suggests that this test case is part of a test suite with name “UBL_invoice_validation” (the path root) and the file itself is named “UBL-Invoice-2.1.xsd” and exists in a folder within the test suite archive named “resources”.

<testcase>
    <imports>
        <artifact type="schema" encoding="UTF-8" name="ublSchema">UBL_invoice_validation/resources/UBL-Invoice-2.1.xsd</artifact>
    </imports>
    <steps>
        <verify handler="XSDValidator" desc="Validate invoice against UBL 2.1 Invoice Schema">
            <!--
                Variable $fileContent is loaded in another step.
            -->
            <input name="xmldocument">$fileContent</input>
            <input name="xsddocument" source="$ublSchema"/>
        </verify>
    </steps>
</testcase>

Note

Test module import: The GITB TDL schema allows also for module elements to be defined for the import of test modules (validation, messaging and processing handlers). This approach is no longer supported as it required the handler implementations to be bundled within the test bed itself. The preferred and simpler approach now is to simply define the handler in the respective test step (e.g. the verify step’s handler attribute for validators) without previously importing it.

Preliminary¶

The preliminary element allows the test case to interact with the user before the test session begins. The purpose here is to allow the user to provide preliminary input or be informed of certain actions that need to take place before the test session starts. In terms of structure and use, the preliminary element is a UserInteraction element (see interact). The difference is that the interaction takes place before the test session actually starts.

The following example shows a test case that prompts the user before starting to initialise their server and upload a configuration file.

<testcase>
    <preliminary desc="Prepare your system" with="User">
        <instruct desc="Preparation instructions" with="User" type="string">"Make sure your system is up and running"</instruct>
        <request desc="Provide your configuration file" with="User" contentType="BASE64">$sutConfigFile</request>
    </preliminary>
    <variables>
        <var name="sutConfigFile" type="binary"/>
    </variables>
    <actors>
        <gitb:actor id="User" name="User" role="SUT"/>
    </actors>
    <steps>
        <!--
            The provided file can be referenced as $sutConfigFile
        -->
    </steps>
</testcase>

Actors¶

The actors element is where the test case defines the actors involved in its steps and, importantly, their role. It contains one or more actor children with the following structure:

Name	Required?	Description
@id	yes	The actor’s unique (within the specification) ID. This must match an actor ID specified in the test suite.
@name	yes	The name to display for the actor. This can differ from the ID to display an actor name specific to the test case.
@role	yes	The actor’s role in the test case. This is “SUT” if the actor is the focus of the test case, “SIMULATED” if the actor is simulated by the test bed, or “MONITOR” if the actor is present for monitoring purposes.
endpoint	no	An optional sequence of configuration endpoints if the actor is simulated.

The endpoint elements require a bit more explanation to understand their use. A specification may foresee actors that are all valid selections for conformance statements. Imagine a specification that defines “sender” and “receiver” actors that can both be the SUTs depending on the actor a system selects to conform to. As such, a test suite focusing on the sender will include test cases with the sender as the SUT and the receiver as being simulated. Similarly, a test suite focusing on the receiver will define the receiver as SUT and the sender as simulated. In terms of configuration properties, the sender might need to define a “replyToAddress” to receive replies, whereas the receiver simply needs to define his “deliveryAddress” which is where messages are expected. In terms of actor configuration in the test suite this would look like this:

<testsuite>
    <gitb:actor id="sender">
        <gitb:name>Sender</gitb:name>
        <gitb:endpoint name="info">
            <gitb:config name="replyToAddress" desc="The address to return replies to" kind="SIMPLE"/>
        </gitb:endpoint>
    </gitb:actor>
    <gitb:actor id="receiver">
        <gitb:name>Receiver</gitb:name>
        <gitb:endpoint name="info">
            <gitb:config name="deliveryAddress" desc="The address to receive messages on" kind="SIMPLE"/>
        </gitb:endpoint>
    </gitb:actor>
</testsuite>

Depending on the test case at hand, the user will be expected to provide the appropriate configuration parameters. For example, in a conformance statement for the sender, the applicable test cases will be those defining the sender actor as the SUT, and the “replyToAddress” parameter will need to be entered before starting the test. How is the “deliveryAddress” then provided for the simulated receiver actor? This can be achieved in two ways:

Dynamically by the simulated actor’s messaging handler. Using this approach, the test bed, while in its initiation phase, will request configuration properties from the handler that will be mapped to the SUT’s corresponding endpoint (see Endpoints to map simulated actor configuration).
Statically by defining the endpoint and one or more of its parameters within the test case itself.

The second option is why we are able to configure endpoint elements as part of the test case. The values configured here will be used only if not already specified by the response of the simulated actor’s handler. The below snippet illustrates this considering the sender as the SUT:

<testcase>
    <gitb:actor id="sender" name="sender" role="SUT"/>
    <gitb:actor id="receiver" name="receiver" role="SIMULATED">
        <gitb:endpoint name="info">
            <gitb:config name="deliveryAddress">SIMULATED_ADDRESS</gitb:config>
        </gitb:endpoint>
    </gitb:actor>
    <steps>
        <!--
            receiver's address can be referenced by $sender{receiver}{deliveryAddress}
        -->
    </steps>
</testcase>

Note

GITB software support: The “MONITOR” value for the actor role is currently not supported.

Variables¶

The variables element can be defined to create one or more variables that will be used during the test case’s execution. It contains one or more var elements, one per variable, with the following structure:

Name	Required?	Description
@name	yes	The name of the variable. It is with this name that the variable can be referenced.
@type	yes	The type of the variable. One of the GITB data types can be used (see Types).
value	no	One or more values for the variable. More than one values are applicable in case of a `map` or `list` type.

Variables can be used to record arbitrary information for the duration of the test session. These can be fixed values defined along with the variable’s definition or dynamically produced values resulting from test steps. The way to reference variables is defined based on the expression language in place. Using the default XPath 1.0 expression language a variable named myVar is referenced as $myVar. More information on expressions to reference variable values is provided in Expressions.

The following example shows use of two variables, one to store a user-uploaded file and another to store a part of it, extracted via XPath:

<testcase>
    <imports>
        <artifact type="schema" encoding="UTF-8" name="schemaFile">testSuite/artifacts/UBL/maindoc/UBL-Invoice-2.1.xsd</artifact>
    </imports>
    <variables>
        <var name="fileContent" type="object"/>
        <var name="targetElement" type="object"/>
    </variables>
    <steps>
        <!--
            Store the uploaded result in the fileContent variable.
        -->
        <interact desc="UBL invoice upload" with="User">
            <request desc="Upload the UBL invoice to validate" with="User" contentType="BASE64">$fileContent</request>
        </interact>
        <!--
            Extract a part of it and store in the targetElement variable.
        -->
        <assign to="$targetElement" source="$fileContent">/*[local-name() = 'testcase']/*[local-name() = 'steps']</assign>
        <!--
            Pass the targetElement for validation.
        -->
        <verify handler="XSDValidator" desc="Validate content">
            <input name="xmldocument">$targetElement</input>
            <input name="xsddocument" source="$schemaFile"/>
        </verify>
    </steps>
</testcase>

Setting a variable’s initial value is achieved using the value element, with one or more being used in case of a map or list type. The following example illustrates setting values for different variable types:

<testcase>
    <variables>
        <var name="aList" type="list[string]">
            <value>List value 1</value>
            <value>List value 2</value>
        </var>
        <var name="aMap" type="map">
            <value name="key1" type="string">Map value 1</value>
            <value name="key2" type="string">Map value 2</value>
        </var>
        <var name="aString" type="string">
            <value>A string value</value>
        </var>
    </variables>
</testcase>

Note

List variables: When a list is defined as a variable it also needs to specify its internal element type. To do this you need to specify the type attribute as list[INTERNAL_TYPE]. For example a list of string elements is defined as <var name="myList" type="list[string]"/>.

Note

GITB software support: Currently values for variables are defined in a fixed manner. Using expressions to set variable values is not supported.

Steps¶

The steps element is where the test case’s testing logic is implemented. It consists of a sequence of test steps that use a GITB TDL step construct per test step. The available test steps that can be defined are:

Step name	Description
send	Send a message to an actor
receive	Receive a message from an actor
listen	Listen for exchanged messages between actors
btxn	Begin a messaging transaction
etxn	End a messaging transaction
process	Process a set of inputs to get an output
bptxn	Begin a processing transaction
eptxn	End a processing transaction
if	Apply if-else logic to conditionally execute test steps
while	Loop over a set of steps while a condition is true
repuntil	Repeat a set of steps while a condition is true (executing at least once)
foreach	Execute a set of steps a fixed number of times
flow	Execute sets of steps concurrently
exit	Immediately terminate the test session
assign	Process an expression and assign its output to a variable
group	Display a set of steps as a logical group
verify	Validate content
call	Call a scriptlet
interact	Trigger an interaction with the user

Scriptlets¶

The scriptlets element is meant to define reusable blocks of steps that can be called during the test case’s execution. They are similar to function blocks in programming languages considering that:

They receive inputs and produce outputs.
They define a nested context for their own variables and processing that is isolated from the test session context.
They can access variables from the parent test session context.

The main benefit of using scriptlets is really one of test step organisation to avoid copying (and then maintaining) sequences of steps that might be executed multiple times and at different locations in the test case. Each scriptlet is defined in a scriptlet element with the following structure:

Name	Required?	Description
@id	yes	The ID of the scriptlet used to refer to it in `call` steps (see call).
metadata	no	Optional scriptlet metadata. Structurally this matches exactly the `metadata` for the test case (see Metadata).
namespaces	no	Optional namespaces for contained expressions. Structurally this matches exactly the `namespaces` for the test case (see Namespaces).
imports	no	Optional artefacts to import for the scriptlet. Structurally this matches exactly the `imports` for the test case (see Imports).
params	no	An optional set of parameters that the scriptlet will expect from a `call` step (see call).
variables	no	An optional set of local variables. Structurally this matches exactly the `variables` for the test case (see Variables).
steps	yes	The sequence of steps to be executed in this scriptlet. Can contain any supported test steps (see Steps).
output	yes	One or more output values resulting from the scriptlet’s execution.

The params and variables for the scriptlet define their individual elements as var elements. These follow the same structure and logic as the test cases variables (see Variables).

A scriptlet is called using its id in a call step (see call). As part of this call the test case needs to pass as inputs any parameters that the scriptlet expects. Concerning the scriptlet’s output, each defined output value will be evaluated once the scriptlet completes and the overall result will be returned as a map named using the id used in the relevant call step that triggered it’s execution. Regarding output values:

If the call does not specify named output elements then all the scriptlet’s outputs will be returned.
If the call does specify named output elements only these will be returned.

The following example shows a scriptlet that will validate a file passed as an input parameter. Once completed it will also return a string result for a value provided by the user. Calling the scriptlet occurs for two different files and results in the display of their output values.

<testcase>
    <steps>
        <!--
            Request two files to be uploaded.
        -->
        <interact desc="Upload files" with="User">
            <request desc="Upload the first file" with="User" contentType="BASE64">$fileContent1</request>
            <request desc="Upload the second file" with="User" contentType="BASE64">$fileContent2</request>
        </interact>
        <!--
            Call the scriptlet for the first file and store the result under variable "call1".
        -->
        <call id="call1" path="script1">
            <input name="docToValidate">$fileContent1</input>
            <output name="outputMessage"/>
        </call>
        <!--
            Call the scriptlet for the second file and store the result under variable "call2".
        -->
        <call id="call2" path="script1">
            <input name="docToValidate">$fileContent2</input>
            <output name="outputMessage"/>
        </call>
        <!--
            Display the results from both calls to the user.
        -->
        <interact desc="Scriptlet results" with="User">
            <instruct desc="Output one" with="User" type="string">$call1{outputMessage}</instruct>
            <instruct desc="Output two" with="User" type="string">$call2{outputMessage}</instruct>
        </interact>
    </steps>
    <scriptlets>
        <scriptlet id="script1">
            <!--
                This parameter has to be provided when calling the scriptlet.
            -->
            <params>
                <var name="docToValidate" type="object"/>
            </params>
            <!--
                These variables are only locally visible.
            -->
            <variables>
                <var name="userMessage" type="string"/>
                <var name="outputMessage" type="string"/>
                <var name="anotherOutputMessage" type="string">
                    <value>Another value</value>
                </var>
            </variables>
            <steps>
                <!--
                    Ask the user to enter a string value.
                -->
                <interact desc="Scriptlet call" with="User">
                    <request desc="Give me a value" with="User" type="string">$userMessage</request>
                </interact>
                <!--
                    Validate the file passed as a parameter.
                -->
                <verify handler="XSDValidator" desc="Validate content">
                    <input name="xmldocument">$docToValidate</input>
                    <!--
                        The schemaFile variable is retrieved from the global session context
                        (its declaration is ommitted from the example).
                    -->
                    <input name="xsddocument" source="$schemaFile"/>
                </verify>
                <assign to="$outputMessage">$userMessage</assign>
            </steps>
            <!--
                This output is returned as it is specified in the call.
            -->
            <output name="outputMessage" type="string">$outputMessage</output>
            <!--
                This output is not specified in the call and is thus ignored.
            -->
            <output name="anotherOutputMessage" type="string">$anotherOutputMessage</output>
        </scriptlet>
    </scriptlets>
</testcase>

Note

GITB software support: Currently the GITB software requires that scriptlets have at least one variable and one parameter. In addition, declared outputs must also exist as declared scriptlet variables.