Test Cases¶

Overview¶

Test cases are the means by which a specific testing scenario is implemented. One or more test cases form the content of a test suite. The following example represents a complete, simple test case for the validation of an invoice that is uploaded by a user.

<?xml version="1.0" encoding="UTF-8"?>
<testcase id="UBL_invoice_validation_test_1" xmlns="http://www.gitb.com/tdl/v1/" xmlns:gitb="http://www.gitb.com/core/v1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.gitb.com/tdl/v1/ ../../gitb_tdl.xsd">
    <metadata>
        <gitb:name>UBL_invoice_validation_test_1</gitb:name>
        <gitb:type>CONFORMANCE</gitb:type>
        <gitb:version>1.0</gitb:version>
        <gitb:description>Test case to verify the correctness of a UBL invoice. The invoice is provided manually through user upload.</gitb:description>
    </metadata>
    <imports>
        <artifact type="schema" encoding="UTF-8" name="UBL_Invoice_Schema_File">artifacts/UBL/maindoc/UBL-Invoice-2.1.xsd</artifact>
        <artifact type="object" encoding="UTF-8" name="BII_CORE_Invoice_Schematron_File">artifacts/BII/BII_CORE/BIICORE-UBL-T10-V1.0.xsl</artifact>
    </imports>
    <variables>
        <var name="file_content" type="object"/>
    </variables>
    <actors>
        <gitb:actor id="User" name="User" role="SUT"/>
    </actors>
    <steps>
        <!--
            Step 1. Request the user to upload the UBL invoice.
        -->
        <interact desc="UBL invoice upload" with="User">
            <request desc="Upload the UBL invoice to validate" with="User" contentType="BASE64">$file_content</request>
        </interact>
        <!--
            Step 2. Validate the uploaded invoice.
        -->
        <verify handler="XSDValidator" desc="Validate invoice against UBL 2.1 Invoice Schema">
            <input name="xmldocument">$file_content</input>
            <input name="xsddocument" source="$UBL_Invoice_Schema_File"/>
        </verify>
        <verify handler="SchematronValidator" desc="Validate invoice against BII2 CORE restrictions for Invoice Transaction">
            <input name="xmldocument">$file_content</input>
            <input name="schematron" source="$BII_CORE_Invoice_Schematron_File"/>
        </verify>
    </steps>
</testcase>

The following table provides an overview of the attributes and child elements that a testcase may have. A more detailed discussion per case follows in the subsequent sections.

Name	Required?	Description
@id	yes	A string to uniquely identify the test case by. This is referenced in the test suite XML.
metadata	yes	A block containing the metadata used to describe the test case.
namespaces	no	An optional set of namespaces to define the expression languages used in the test case.
imports	no	An optional set of imports used to load additional resources from the test suite.
preliminary	no	An optional set of user interaction steps to display before the test session starts.
variables	no	An optional set of variables that are used in the test case.
actors	yes	The set of actors that this test case refers to.
steps	yes	The sequence of steps that this test case foresees.
scriptlets	no	Optional named groups of test steps which can be used within the test case.

Elements¶

We will now see how a test case breaks down into its individual sections and discuss the purpose of each.

Metadata¶

The metadata element is basically the same as the one defined for the test suite Metadata. Its purpose is to provide basic information about the test case to help users understand its purpose. Its structure is as follows:

Name	Required?	Description
name	yes	The name of the test case that is used to identify it to users.
type	no	Either “CONFORMANCE” (the default) or “INTEROPERABILITY”. “INTEROPERABILITY” is used when more than one actor are defined as SUTs.
version	yes	A string that indicates the test case’s version.
authors	no	A string to indicate the test case’s authors.
description	no	A string to provide a user-friendly description of the test case that is displayed to users.
published	no	A string acting as an indication of the test case’s publishing time.
lastModified	no	A string acting as an indication of the last modification time for the test case.
documentation	no	Rich text content that provides further information on the current test case.

The documentation element complements the test case’s description by allowing its author to include extended rich text documentation. This documentation can provide further information on the context of the test case, diagrams or reference information that are useful to understand how it is to be completed or its purpose within the overall specification. The content supplied supports several HTML features:

Structure elements (e.g. headings, text blocks, lists).

In-line styling.

Tables.

Links.

Images.

The simplest way to provide such information is to enclose the HTML content in a CDATA section to ensure the XML remains well-formed. The following sample provides an example of this approach:

<testcase id="TS1-TC1" xmlns="http://www.gitb.com/tdl/v1/" xmlns:gitb="http://www.gitb.com/core/v1/">
    <metadata>
        <gitb:name>TS1-TC1</gitb:name>
        <gitb:version>1.0</gitb:version>
        <gitb:description>A short description of the test case to offer a short summary of its purpose.</gitb:description>
        <gitb:documentation><![CDATA[
            <p>Extended documentation for test case <b>TS1-TC1</b></p>
            <p>This is an example to support the <a href="https://www.itb.ec.europa.eu/docs/tdl/latest">GITB TDL docs</a>.</p>
        ]]></gitb:documentation>
    </metadata>
    ...
</testcase>

Note that documentation such as this is also supported for:

The overall test suite.

Individual test case steps.

Note

GITB software support: The test case type must currently be set to “CONFORMANCE” (the default value) as the “INTEROPERABILITY” type is not supported. Finally, the version, authors, published and lastModified values are recorded but never used or displayed.

Namespaces¶

The namespaces optional element is used to define one or more expression languages that are used in test case constructs that support them. This needs to be done when expressions are used that should not be processed using the default XPath 1.0 language. A detailed discussion on GITB expressions as well as where and how you can use them is provided in Expressions. Each referred expression language is defined in a namespace element with the following structure:

Name	Required?	Description
ns	yes	One or more elements to define the expression languages used in the test case. Each `ns` element must specify a `prefix` attribute to define the namespace prefix.

The values specified in the prefix attributes define how each expression language is referenced within the test case. As an example consider a test case that needs to check a number variable var and set a result variable with “result1” if var is 1 or with “result2” otherwise. Simple if-else constructs like this are a good example of a shortcoming of XPath 1.0 as they are not natively supported and need a non-intuitive workaround based on string concatenation. A simpler approach would be to use a scripting language more adapted to this kind of operation such as JavaScript. The following test case illustrates this assignment using both the default XPath 1.0 and JavaScript:

<testcase>
    <namespaces>
        <ns prefix="JavaScript"/>
    </namespaces>
    <variables>
        <var name="var" type="number"/>
        <var name="result" type="string"/>
    </variables>
    <steps>
        <!--
            Assignment using the default XPath 1.0.
        -->
        <assign to="$result">concat(substring('result1', 1 div number(boolean($var = 1))), substring('result2', 1 div number(not(boolean($var = 1)))))</assign>
        <!--
            Assignment using JavaScript.
        -->
        <assign to="$result" lang="JavaScript">if ($var == 1) { return 'result1' } else { return 'result2' }</assign>
    </steps>
</testcase>

Needless to say the assignment using JavaScript is much more natural and easy to understand. However use of alternative expression languages, and their definition through the namespaces element, is tricky because we need to know exactly how the target test bed refers to the language (i.e. “JavaScript” in our case) to correctly identify it. Furthermore it must be clear how the test bed will process the expression and how session context variables are looked up. In the above example we assume that context variables (e.g. “$var”) are looked up in exactly the same way as with XPath 1.0 expressions and that the entire expression will be evaluated by first wrapping it in a function, the result of which is returned as the assignment output. Apart from actually supporting JavaScript for expressions, these additional details need to first be defined unambiguously by the test bed and made known to its users. Only then can we use them in a deterministic and portable manner.

Note

GITB software support: Expression languages other than the default XPath 1.0 are not supported. As such the namespaces element is currently ignored.

Imports¶

The imports element allows the use of arbitrary resources that are present in the test suite. This can be very useful when a test case needs to send messages based on a template or load a binary file that is needed as input by a messaging, processing or validation handler (e.g. a certificate). The imports element defines one or more artifact children with the following structure:

Name	Required?	Description
@name	yes	The name with which this artefact will be associated to the test session context for subsequent lookups.
@type	yes	The type as which the artefact needs to be loaded.
@encoding	no	In case the artefact is to be treated as text	this is the character encoding to apply when reading its bytes (default is “UTF-8”).

The text value of the artifact element is the path within the test suite from which the relevant resource will be loaded. This path may be provided as a fixed value or as a variable reference to determine the imported resource dynamically. In case a variable reference is provided this should be one of the following:

A reference to a configuration value (i.e. a domain, organisation, system or actor parameter).
A reference to a variable defined in the test case. In this case the value of the variable can even be adapted during the course of the test session resulting in different resources depending on the point at which the import is referenced.

Regarding the type attribute, this needs to refer to an appropriate type from the GITB type system (see Types). Given that in this case we are referring to a file being loaded, the types that can be used are:

binary: Load the artefact as a set of bytes without additional processing.
object: Load the artefact as a XML Document Object Model. In this case it is best to also explicitly provide the encoding to consider.
schema: Load the artefact as a XML Schema. As in the object case it is best to explicitly provide the encoding to consider.

Regarding the path to the resource this is the resource’s path within the test suite archive (with or without the test suite name as a prefix). As an example consider the following test case fragment where a XML schema is loaded and set in the session context as a variable of type schema that is named “ublSchema”. The path specified suggests that the file is named “UBL-Invoice-2.1.xsd” and exists in a folder within the test suite archive named “resources”. This example also includes another input whose referenced resource is defined dynamically based on an external configuration parameter (at organisation level in this case).

<testcase>
    <imports>
        <!--
            The "ublSchema" is loaded from a fixed resource within the test suite.
        -->
        <artifact type="schema" encoding="UTF-8" name="ublSchema">resources/UBL-Invoice-2.1.xsd</artifact>
        <!--
            The "organisationSpecificSchema" is loaded dynamically based on an organisation-level configuration property named "xsdToUseForOrganisation".
        -->
        <artifact type="schema" encoding="UTF-8" name="organisationSpecificSchema">$ORGANISATION{xsdToUseForOrganisation}</artifact>
    </imports>
    <steps>
        <verify handler="XSDValidator" desc="Validate invoice against UBL 2.1 Invoice Schema">
            <!--
                Variable $fileContent is loaded in another step.
            -->
            <input name="xmldocument">$fileContent</input>
            <input name="xsddocument" source="$ublSchema"/>
        </verify>
    </steps>
</testcase>

Note

Test module import: The GITB TDL schema allows also for module elements to be defined for the import of test modules (validation, messaging and processing handlers). This approach is no longer supported as it required the handler implementations to be bundled within the test bed itself. The preferred and simpler approach now is to simply define the handler in the respective test step (e.g. the verify step’s handler attribute for validators) without previously importing it.

Preliminary¶

The preliminary element allows the test case to interact with the user before the test session begins. The purpose here is to allow the user to provide preliminary input or be informed of certain actions that need to take place before the test session starts. In terms of structure and use, the preliminary element is a UserInteraction element (see interact). The difference is that the interaction takes place before the test session actually starts.

The following example shows a test case that prompts the user before starting to initialise their server and upload a configuration file.

<testcase>
    <preliminary desc="Prepare your system" with="User">
        <instruct desc="Preparation instructions" with="User" type="string">"Make sure your system is up and running"</instruct>
        <request desc="Provide your configuration file" with="User" contentType="BASE64">$sutConfigFile</request>
    </preliminary>
    <variables>
        <var name="sutConfigFile" type="binary"/>
    </variables>
    <actors>
        <gitb:actor id="User" name="User" role="SUT"/>
    </actors>
    <steps>
        <!--
            The provided file can be referenced as $sutConfigFile
        -->
    </steps>
</testcase>

Actors¶

The actors element is where the test case defines the actors involved in its steps and, importantly, their role. It contains one or more actor children with the following structure:

Name	Required?	Description
@id	yes	The actor’s unique (within the specification) ID. This must match an actor ID specified in the test suite.
@name	no	The name to display for the actor. This can differ from the ID to display an actor name specific to the test case. Not specifying this will default to the name for actor provided in the test suite.
@role	yes	The actor’s role in the test case. This is “SUT” if the actor is the focus of the test case, “SIMULATED” if the actor is simulated by the test bed, or “MONITOR” if the actor is present for monitoring purposes.
@displayOrder	no	A number indicating the relative positioning that needs to be respected when displaying the actor in test case’s execution diagram. Setting this here overrides any corresponding setting at test suite level (see Actors for details).
endpoint	no	An optional sequence of configuration endpoints if the actor is simulated.

The endpoint elements require a bit more explanation to understand their use. A specification may foresee actors that are all valid selections for conformance statements. Imagine a specification that defines “sender” and “receiver” actors that can both be the SUTs depending on the actor a system selects to conform to. As such, a test suite focusing on the sender will include test cases with the sender as the SUT and the receiver as being simulated. Similarly, a test suite focusing on the receiver will define the receiver as SUT and the sender as simulated. In terms of configuration properties, the sender might need to define a “replyToAddress” to receive replies, whereas the receiver simply needs to define his “deliveryAddress” which is where messages are expected. In terms of actor configuration in the test suite this would look like this:

<testsuite>
    <gitb:actor id="sender">
        <gitb:name>Sender</gitb:name>
        <gitb:endpoint name="info">
            <gitb:config name="replyToAddress" desc="The address to return replies to" kind="SIMPLE"/>
        </gitb:endpoint>
    </gitb:actor>
    <gitb:actor id="receiver">
        <gitb:name>Receiver</gitb:name>
        <gitb:endpoint name="info">
            <gitb:config name="deliveryAddress" desc="The address to receive messages on" kind="SIMPLE"/>
        </gitb:endpoint>
    </gitb:actor>
</testsuite>

Depending on the test case at hand, the user will be expected to provide the appropriate configuration parameters. For example, in a conformance statement for the sender, the applicable test cases will be those defining the sender actor as the SUT, and the “replyToAddress” parameter will need to be entered before starting the test. How is the “deliveryAddress” then provided for the simulated receiver actor? This can be achieved in two ways:

Dynamically by the simulated actor’s messaging handler. Using this approach, the test bed, while in its initiation phase, will request configuration properties from the handler that will be mapped to the SUT’s corresponding endpoint (see Endpoints to map simulated actor configuration).
Statically by defining the endpoint and one or more of its parameters within the test case itself.

The second option is why we are able to configure endpoint elements as part of the test case. The values configured here will be used only if not already specified by the response of the simulated actor’s handler. The below snippet illustrates this considering the sender as the SUT:

<testcase>
    <gitb:actor id="sender" name="sender" role="SUT"/>
    <gitb:actor id="receiver" name="receiver" role="SIMULATED">
        <gitb:endpoint name="info">
            <gitb:config name="deliveryAddress">SIMULATED_ADDRESS</gitb:config>
        </gitb:endpoint>
    </gitb:actor>
    <steps>
        <!--
            receiver's address can be referenced by $sender{receiver}{deliveryAddress}
        -->
    </steps>
</testcase>

Note

GITB software support: The “MONITOR” value for the actor role is currently not supported.

Variables¶

The variables element can be defined to create one or more variables that will be used during the test case’s execution. It contains one or more var elements, one per variable, with the following structure:

Name	Required?	Description
@name	yes	The name of the variable. It is with this name that the variable can be referenced.
@type	yes	The type of the variable. One of the GITB data types can be used (see Types).
value	no	One or more values for the variable. More than one values are applicable in case of a `map` or `list` type.

Variables can be used to record arbitrary information for the duration of the test session. These can be fixed values defined along with the variable’s definition or dynamically produced values resulting from test steps. The way to reference variables is defined based on the expression language in place. Using the default XPath 1.0 expression language a variable named myVar is referenced as $myVar. More information on expressions to reference variable values is provided in Expressions.

Definition of variables using the variables element is optional given that test steps resulting in output will automatically create variables as needed to store the output in the test session context. Such steps include:

Assign steps that define new values or calculate expressions.

User interaction steps that request data from the user.

Messaging steps to record the output of a send or a receive.

Processing steps to record the output of the process.

The type of the automatically created variables in the above cases is inferred from the type of the relevant data or expression result. For example, when assigning a string to a variable, this will automatically be set with a string type. Considering this, you would use the variables element to predefine variables in the following cases:

To predefine all variables if you prefer this from the perspective of code organisation.

To explicitly set the type of variables in cases where the automatic determination is not suitable (e.g. force a string type for a numeric value).

To cover exceptional cases where automatic type determination is not possible.

To provide initial values to variables.

For examples of automatic variable definition refer to the corresponding steps as well as the documentation on expressions. Coming back to explicitly defined variables, the following example shows two such cases, one to store a user-uploaded file and another to store a part of it, extracted via XPath:

<testcase>
    <imports>
        <artifact type="schema" encoding="UTF-8" name="schemaFile">testSuite/artifacts/UBL/maindoc/UBL-Invoice-2.1.xsd</artifact>
    </imports>
    <variables>
        <var name="fileContent" type="object"/>
        <var name="targetElement" type="object"/>
    </variables>
    <steps>
        <!--
            Store the uploaded result in the fileContent variable.
        -->
        <interact desc="UBL invoice upload" with="User">
            <request desc="Upload the UBL invoice to validate" with="User" contentType="BASE64">$fileContent</request>
        </interact>
        <!--
            Extract a part of it and store in the targetElement variable.
        -->
        <assign to="$targetElement" source="$fileContent">/*[local-name() = 'testcase']/*[local-name() = 'steps']</assign>
        <!--
            Pass the targetElement for validation.
        -->
        <verify handler="XSDValidator" desc="Validate content">
            <input name="xmldocument">$targetElement</input>
            <input name="xsddocument" source="$schemaFile"/>
        </verify>
    </steps>
</testcase>

Setting a variable’s initial value is achieved using the value element, with one or more being used in case of a map or list type. The following example illustrates setting values for different variable types:

<testcase>
    <variables>
        <var name="aList" type="list[string]">
            <value>List value 1</value>
            <value>List value 2</value>
        </var>
        <var name="aMap" type="map">
            <value name="key1" type="string">Map value 1</value>
            <value name="key2" type="string">Map value 2</value>
        </var>
        <var name="aString" type="string">
            <value>A string value</value>
        </var>
    </variables>
</testcase>

Note

List variables: When a list is defined as a variable it also needs to specify its internal element type. To do this you need to specify the type attribute as list[INTERNAL_TYPE]. For example a list of string elements is defined as <var name="myList" type="list[string]"/>.

Steps¶

The steps element is where the test case’s testing logic is implemented. It consists of a sequence of test steps that use a GITB TDL step construct per test step. The available test steps that can be defined are:

Step name	Description
send	Send a message to an actor
receive	Receive a message from an actor
listen	Listen for exchanged messages between actors
btxn	Begin a messaging transaction
etxn	End a messaging transaction
process	Process a set of inputs to get an output
bptxn	Begin a processing transaction
eptxn	End a processing transaction
if	Apply if-else logic to conditionally execute test steps
while	Loop over a set of steps while a condition is true
repuntil	Repeat a set of steps while a condition is true (executing at least once)
foreach	Execute a set of steps a fixed number of times
flow	Execute sets of steps concurrently
exit	Immediately terminate the test session
assign	Process an expression and assign its output to a variable
group	Display a set of steps as a logical group
verify	Validate content
call	Call a scriptlet
interact	Trigger an interaction with the user

Scriptlets¶

The scriptlets element is meant to define reusable blocks of steps that can be called during the test case’s execution. They are similar to function blocks in programming languages considering that:

They receive inputs and produce outputs.
They define a nested context for their own variables and processing that is isolated from the test session context.
They can access variables from the parent test session context.

The main benefit of using scriptlets is really one of test step organisation to avoid copying (and then maintaining) sequences of steps that might be executed multiple times and at different locations in the test case. Each scriptlet is defined in a scriptlet element with the following structure:

Name	Required?	Description
@id	yes	The ID of the scriptlet used to refer to it in `call` steps (see call).
metadata	no	Optional scriptlet metadata. Structurally this matches exactly the `metadata` for the test case (see Metadata).
namespaces	no	Optional namespaces for contained expressions. Structurally this matches exactly the `namespaces` for the test case (see Namespaces).
imports	no	Optional artefacts to import for the scriptlet. Structurally this matches exactly the `imports` for the test case (see Imports).
params	no	An optional set of parameters that the scriptlet will expect from a `call` step (see call).
variables	no	An optional set of local variables. Structurally this matches exactly the `variables` for the test case (see Variables).
steps	yes	The sequence of steps to be executed in this scriptlet. Can contain any supported test steps (see Steps).
output	yes	One or more output values resulting from the scriptlet’s execution.

The params and variables for the scriptlet define their individual elements as var elements. These follow the same structure and logic as the test cases variables (see Variables).

A scriptlet is called using its id in a call step (see call). As part of this call the test case needs to pass as inputs any parameters that the scriptlet expects. Concerning the scriptlet’s output, each defined output value will be evaluated once the scriptlet completes and the overall result will be returned as a map named using the id used in the relevant call step that triggered it’s execution. Regarding output values:

If the call does not specify named output elements then all the scriptlet’s outputs will be returned.
If the call does specify named output elements only these will be returned.

The following example shows a scriptlet that will validate a file passed as an input parameter. Once completed it will also return a string result for a value provided by the user. Calling the scriptlet occurs for two different files and results in the display of their output values.

<testcase>
    <steps>
        <!--
            Request two files to be uploaded.
        -->
        <interact desc="Upload files" with="User">
            <request desc="Upload the first file" with="User" contentType="BASE64">$fileContent1</request>
            <request desc="Upload the second file" with="User" contentType="BASE64">$fileContent2</request>
        </interact>
        <!--
            Call the scriptlet for the first file and store the result under variable "call1".
        -->
        <call id="call1" path="script1">
            <input name="docToValidate">$fileContent1</input>
            <output name="outputMessage"/>
        </call>
        <!--
            Call the scriptlet for the second file and store the result under variable "call2".
        -->
        <call id="call2" path="script1">
            <input name="docToValidate">$fileContent2</input>
            <output name="outputMessage"/>
        </call>
        <!--
            Display the results from both calls to the user.
        -->
        <interact desc="Scriptlet results" with="User">
            <instruct desc="Output one" with="User" type="string">$call1{outputMessage}</instruct>
            <instruct desc="Output two" with="User" type="string">$call2{outputMessage}</instruct>
        </interact>
    </steps>
    <scriptlets>
        <scriptlet id="script1">
            <!--
                This parameter has to be provided when calling the scriptlet.
            -->
            <params>
                <var name="docToValidate" type="object"/>
            </params>
            <!--
                These variables are only locally visible.
            -->
            <variables>
                <var name="userMessage" type="string"/>
                <var name="outputMessage" type="string"/>
                <var name="anotherOutputMessage" type="string">
                    <value>Another value</value>
                </var>
            </variables>
            <steps>
                <!--
                    Ask the user to enter a string value.
                -->
                <interact desc="Scriptlet call" with="User">
                    <request desc="Give me a value" with="User" type="string">$userMessage</request>
                </interact>
                <!--
                    Validate the file passed as a parameter.
                -->
                <verify handler="XSDValidator" desc="Validate content">
                    <input name="xmldocument">$docToValidate</input>
                    <!--
                        The schemaFile variable is retrieved from the global session context
                        (its declaration is ommitted from the example).
                    -->
                    <input name="xsddocument" source="$schemaFile"/>
                </verify>
                <assign to="$outputMessage">$userMessage</assign>
            </steps>
            <!--
                This output is returned as it is specified in the call.
            -->
            <output name="outputMessage" type="string">$outputMessage</output>
            <!--
                This output is not specified in the call and is thus ignored.
            -->
            <output name="anotherOutputMessage" type="string">$anotherOutputMessage</output>
        </scriptlet>
    </scriptlets>
</testcase>

Note

GITB software support: Currently the GITB software requires that scriptlets have at least one variable and one parameter. In addition, declared outputs must also exist as declared scriptlet variables.