Validate your test setup¶
This section details the features you can use to validate your testing setup. The screens addressed here are available to your users to configure, execute and report on their tests and as such, provide insight into the experience they have from using the test bed.
In order to test your overall setup, you are provided with a special organisation named Admin Organisation. You can use this organisation as any other organisation used within the test bed to engage in testing activities. Through this organisation you can:
Manage its systems, as the organisation’s components for which testing is made (see Manage your systems).
Manage its conformance statements, defining the specifications to be tested for (see Manage your conformance statements).
Execute tests, to try out actual test sessions ensuring that your test cases are correctly setup (see Execute tests).
View your test history, to view the tests carried out using your test organisation (see View your test history).
Note
The test organisation: In the documentation sections that follow, this special-purpose organisation is often referred to as “your organisation”. This indeed is the organisation that you and other test bed administrators are linked to. It should however be clear that all actions carried out using this are meant only to validate the testing setup you have configured for your communities.
Manage your systems¶
Before proceeding to test anything on the test bed you need to define one or more systems. Recall that systems represent an organisation’s software components for which your users make conformance statements to test for (see System).
To view your systems click the TESTS button from the screen’s header.
The screen that follows displays your organisation’s currently defined systems.
Your systems are presented in a table that displays for each one:
Its short name, a brief name used to display in search results.
Its full name, the complete system name presented in reports and detail screens.
A description, providing additional context on the specific system.
A version number.
To proceed with a specific system, either to view its past tests, conformance testing status or to execute new tests, click the appropriate row from the table. Once clicked you will change to view the conformance statements for the selected system (see View your conformance statements).
Create a new system¶
To create a new system click on the Create system button displayed in the top right side of the system list header. Doing so presents you with a popup form to input the new system’s information.
The information to enter in the presented form are:
The system’s short name (required). This is used when the system is displayed in lists.
The system’s full name (required). This is included in reports that mention the system.
An optional description to provide more information about the system.
A version number (required). Although required this is not currently used in the test bed apart from display purposes.
If your organisation includes other systems you are also presented here with an option to copy the test setup from one of them as a source. Selecting one will replicate the selected system’s conformance statements for the new system.
Once another system is selected to copy from, you are also presented with additional options to include:
System properties: To also copy any additional system-level properties that the source system defines.
Conformance statement configurations: To also copy any of the source system’s configuration parameters set on its conformance statements.
If additional system properties are foreseen, and as long as you are not copying the properties from another system, you will also see a Show properties checkbox. Checking this you can manage your new system’s properties.
Configured properties can be simple texts, secret values (e.g. passwords) or files for which, if supplied by you, you will also see a help tooltip to understand their meaning. Such properties can be edited as follows:
For texts through an editable text field or by selecting a preset value from a dropdown list.
For files using the Upload button. Once one is selected you can download it by clicking on its link, or delete it by clicking Remove.
For secrets a read-only text field indicates whether a value is currently set. Provide a new value by checking Update which makes the text field editable. While editing you can also toggle the display of typed characters.
Note
Required properties are marked with an asterisk. It is is not mandatory to fill these in when providing the system’s information but as long as required properties are missing you will not be able to launch tests.
Once you have entered the system’s information click the Save button to record it. You can also click the Cancel button to close the popup without making any changes.
Edit an existing system¶
To edit an existing system click the pencil icon displayed on the right end of the system’s row. Doing so results in a popup being displayed with the system’s information, presented in editable input fields.
You can proceed here to modify the short name, full name, description and version of the system. If your organisation defines other systems you can also select to copy the test setup from another system which will reset the system’s conformance statements to match the selected one (upon confirmation).
Once another system is selected to copy from, you are also presented with additional options to include:
System properties: To also copy any additional system-level properties that the source system defines.
Conformance statement configurations: To also copy any of the source system’s configuration parameters set on its conformance statements.
If additional system properties are foreseen, and as long as you are not copying the properties from another system, you will also see a Show properties checkbox. Checking this you can manage the system’s properties.
Configured properties can be simple texts, secret values (e.g. passwords) or files for which, if supplied by you, you will also see a help tooltip to understand their meaning. Such properties can be managed as follows:
For texts the current value is presented in an editable text field or dropdown menu (if the property has preset values).
For files the Upload button is used to select a new file, whereas if one is already set you can download it by clicking on its link, or delete it by clicking Remove.
For secrets a read-only text field indicates whether a value is currently set, whereas to provide a new value you check Update. When providing a new value you can also toggle the display of the typed characters.
Note
Required properties are marked with an asterisk. It is is not mandatory to fill these in when providing the system’s information but as long as required properties are missing you will not be able to launch tests.
Once ready click the Save button to finish. Here you may also click the Delete button which, following confirmation, will proceed to completely delete the system. In this case the tests realised for this system will still be searchable but will be presented as obsolete (see View your test history). Finally, you can also click the Cancel button to close this popup without making any changes.
Manage your conformance statements¶
Conformance statements serve to define an organisation’s testing goals by linking one of its registered systems with a specification’s actor (see Conformance statement). It is a system’s conformance statements that determine the test suites and test cases that will be presented for execution.
View your conformance statements¶
Conformance statements are made at the level of a system and as such, the first step is to select one of the systems configured for your organisation (see Manage your systems).
This table presents for the selected system its list of conformance statements in terms of their domain, specification and actor. Simply put this set of information serves to uniquely identify the specification role that your system aims to play, thus determining the test cases that it should execute. The presented test results also provide you an overview of the latest test results, showing you how many configured tests your system has successfully passed up to this point and, for the ones not passed, their current status (“undefined” or “failed”). Finally, the overall conformance status is also displayed per statement indicating its current result as undefined, failed or successful.
From this table you can click any row to proceed to the conformance statement’s details (see View a conformance statement’s details). You can return to the listing of conformance statements at any time by clicking the Conformance Statement entry in the left side menu.
Create a conformance statement¶
To create a new conformance statement for your selected system click the Create conformance statement button from the top right side of the conformance statement listing’s header. This presents to you a wizard to construct your conformance statement.
The first step is to select the domain relevant to your conformance statement. Note that this screen may be skipped if your test bed defines only a single domain, considering the domain as selected by default.
If you do have the choice over multiple domains select the one you want by clicking its row. With the relevant row highlighted you can now click the Next button to proceed or the Cancel button to abort. The next step presented is the list of specifications for the selected domain.
Similarly if the selected domain defines only a single specification this will be considered as selected by default and this step will be skipped. If you have multiple specifications to choose from select the one you want by clicking its row. You can now click on the Next button to proceed (or, as previously, the Cancel button to abort). The final step presented is the list of actors defined for the specification.
This step is skipped if your selected specification defines a single actor or if an administrator (you or a relevant community administrator) has flagged an actor as the specification’s default. If this is not the case select the actor from the list by clicking its row and then on the Next button (or the Cancel button to abort). As a final step you will be presented with a confirmation screen in which the details of your conformance statement are presented (domain, specification and actor).
To complete the creation of the conformance statement click the Confirm button. Alternatively you can, as in the previous steps, abort by clicking the Cancel button. In either case you will be returned to the listing of your system’s conformance statements (see View your conformance statements).
Note
Automatic conformance statement selection: If your test bed defines a single domain with a single specification and a single actor, clicking the Create conformance statement button will immediately present a confirmation screen without the need to make further selections.
View a conformance statement’s details¶
The conformance statement detail screen provides you the test status summary for a given system of your organisation and a specification’s actor. In addition it is the point from which you can start new tests. The information displayed in this page is organised in three sections to present to you:
The details of the conformance statement.
The configuration for your system, used when it is defined as a test case’s SUT.
The status and controls of the related tests.
Overview¶
The Conformance statement details section provides you the context of what your system is supposed to conform to.
The domain details are presented on the top as the high-level description of the relevant specifications. The specification information follows to define the specification you have chosen for your system to conform to (a domain may have multiple specifications). The actor information defines the specific role your system is expected to fulfil as part of this specification (a specification may have multiple actors). The test results present an overview of the testing progress for the conformance statement’s test cases, whereas the status represents the statement’s current progress. Below this section you are presented with buttons for further actions as follows:
The Download conformance certificate button to generate a conformance certificate for your system (see Export conformance certificate).
The Download report button to export your system’s current conformance statement report (see Export conformance statement report).
The Delete statement button to delete the conformance statement (see Delete conformance statement).
Configuration parameters¶
The next section displayed is the information on the system’s Configuration parameters, the information that you are expected to provide to the test bed for your system pertinent to the current conformance statement. These parameters are typically configured once and for this purpose are displayed by default in collapsed form:
Clicking on the Hidden toggle button on the right-hand side will set it to Visible and in doing so, expand the panel to display the details for all properties:
Configuration properties are presented in rows where for each one the following information is presented:
Whether or not it is set.
Its parameter name, serving as its identifier. This is prefixed with an asterisk if the parameter is mandatory.
Its configured value.
Its description including helpful text to understand its meaning.
An edit icon to change or remove its value.
In case of parameters that are binary files, these are presented with a link to download them. The following example shows such a parameter, along with additional text values.
To edit a configuration parameter click its edit icon on its relevant row. Doing so will open a prompt that presents the parameter’s name, description and current value that is editable.
In case of a parameter that is a file, the popup will be adapted to allow you to download the file and upload a replacement.
A third scenario is that of a parameter being a secret value (e.g. a password). In this case you are prompted to provide and confirm a new value.
Finally, an additional scenario is when preset values are defined for the parameter. In this case you are presented with a dropdown selection list that includes the available options.
To change the parameter’s value click on Save. Clicking on Delete will clear the current value, whereas Cancel will close the popup without making changes.
Finally, note that the complete Configuration parameters section may be missing in case your system is not expected to provide any information before executing its tests.
Tests¶
At the bottom of the page you can find the Conformance tests section.
This is a table displaying the tests you are expected to execute for the given conformance statement as well as the latest test results. The information displayed for each test case is:
Its name, a short text to identify and refer to the test case.
Its description, providing the context you need to understand the purpose of the test case and plan for its execution.
Its latest result which can be either a success (green tick), failure (red cross) or undefined (grey bar) in case the test case has never been executed.
A Play button to start a new test session for this test case (see Execute tests).
Note that the screenshot above indicates a simple conformance statement that contains a single test suite containing in turn a single test case. In this case the test suite is hidden for simplicity. A more elaborate conformance statement would typically include multiple test cases, each addressing a different scenario to be tested. The following screenshot is from a conformance statement for which a single test suite is defined but that contains multiple test cases.
Notice in this case that apart from the individual button to execute each test case, you also have a similar button in the section’s header. Clicking this will proceed to automatically and sequentially execute all listed test cases.
Finally, a further complex conformance statement could define multiple test cases organised in multiple test suites. In this case each test suite becomes important and is presented in the list of tests.
The table in this case displays at a first level the list of test suites, using a grey backdrop to differentiate them from test cases. Similar to test cases, each test suite displays its name, description and overall result, and can be clicked to expand and display its test cases.
Clicking on an expanded test suite collapses it again. Notice in addition that the Play button to execute test cases now displays at two levels:
For each test suite: To automatically and sequentially execute all the test suite’s test cases.
For each test case: To execute the specific test case.
When test sessions are completed for the statement’s different test cases, the result display will be adapted to display them as successful or failed. Moreover, in case a test session also produced a detailed output message, this can be viewed by clicking on the success or failure icon.
In all previous examples you may have noticed a Interactive execution toggle button displayed in the Conformance tests header. This toggle defines the manner in which you will execute the tests, defaulting to sequential interactive sessions. This means that each of the tests will execute in sequence and synchronously, with the current progress displayed to you and any steps requiring your input prompting you as needed. The alternative option, enabled if you click the toggle button, is Background execution that will launch the tests sessions in parallel and run them in the background.
Opting for background execution allows you to launch a potentially large number of test sessions without needing to oversee their progress. Care however needs to be taken here to ensure that all relevant test cases can be carried out without user interaction. If a test session running in the background defines user interaction steps, these are managed as follows:
Instructions are simply skipped, assuming that these are purely of informational value.
Input requests are completed automatically without input. Doing so will most likely cause a test session to fail (e.g. if a user is expected to provide the content of a message to send) but could still result in a successful completion if the test case has been designed to treat user input as optional.
The status of test sessions launched in the background can be monitored by means of the Test Sessions screen.
A further point to mention regarding the selection of test cases to execute is the possibility to view their extended documentation. Test cases and test suites are typically defined to include a brief description that provides context and high-level instructions. This is the description displayed in the test case listing discussed up to this point. The authors of the test cases may have provided in addition to this description, extended documentation that provides further information such as diagrams and reference links. Such documentation may exist at test suite and/or test case level, in which case you will see an additional information button displayed where available.
Clicking this button will result in a popup window containing the extended documentation.
Note that the documentation on test cases is also available to consult during their execution (in case of interactive execution).
Export conformance statement report¶
The conformance statement report (in PDF format) provides the details on the conformance statement and also an overview of its relevant tests. To generate it click the Download report button from the overview section’s panel.
Once the button is clicked you will be prompted for the level of detail you want to include in the report. Two options are available regarding whether or not you want to include each test case’s step results in the report.
Selecting Yes includes the conformance statement details and test overview but also each test case’s step results. Selecting No on the other hand skips the test step results.
The following sample illustrates the information that is included in the report’s overview section that is always included. Specifically:
The information on the domain, specification and actor for the selected system.
The name of the system’s organisation and the system itself.
The date the report was produced and the number of successfully passed test cases versus the total.
A table with the conformance statement’s test cases, displaying a row per test case with its reference number, the name of the the test suite and test case, the test case description and its test result.
In case the option to add each test case’s step results is selected, the report includes a section per test case displaying its summary and the results from each test step. The test case’s title includes its reference number listed in the report’s overview section.
Note
Detailed report size: The detailed conformance statement report presents each test session and individual step in a separate page. If your conformance statement contains numerous test cases, each with multiple test steps, the resulting detailed report could be quite long.
Export conformance certificate¶
The conformance certificate is a report (in PDF format) that attests to the fact that your current system has successfully passed its expected test cases. The option to generate this is only visible if your system has succeeded in all configured tests.
Assuming the option is available for you, clicking the button will generate the certificate and prompt you for its download. The certificate will typically resemble the following sample:
The contents of the certificate are defined by your administrator and are a customisation of the conformance statement report. The certificate may omit certain sections, include a message for you, and potentially be digitally signed.
Delete conformance statement¶
Deleting the conformance statement may be desired if you created it by mistake or if your system is no longer expected to conform to the given specification. Deleting the conformance statement is possible through the Delete statement button from the overview panel.
Clicking this will request confirmation and, if confirmed, will remove the conformance statement. Note that your testing history relevant to this conformance statement still remains and can be consulted through your test history (see View your test history). In addition, if you create the same conformance statement again, your previous tests will be once again counted towards your conformance testing status.
View selected system’s information¶
Once a system is selected from the list of your organisation’s systems (see Manage your systems) you can manage its conformance statements and view its test history. At any given time you can review the information of your selected system by clicking the System Information entry from the left side menu.
In this screen you can see the short and full name of the system, its description and its version number. If you or a community administrator have foreseen additional properties for systems you will also see here the Show properties checkbox. Clicking this will expand to also display the current system’s additional information.
The displayed properties can be simple texts, secret values (e.g. passwords) or files and, if supplied by you, will display a help tooltip to understand their meaning. You can view here the configured values or download linked files. Required properties are marked with an asterisk and will need to be completed before launching any tests for this system.
Note
Editing a system’s information: The information displayed on this screen is read-only. Editing the system’s information is possible through the screen listing your organisation’s systems (see Edit an existing system).
Execute tests¶
Executing conformance tests is the reason your users are using the test bed. Considering that test cases are linked to a system by means of conformance statements, the first step before executing a test is to visit a conformance statement’s detail screen (see View a conformance statement’s details). This screen is the place where you input required configuration and are provided with the controls to execute one or more tests.
Provide your system’s configuration¶
The testing configuration for your selected specification may require that you provide one or more configuration parameters before executing tests. If for example test cases require that the test bed sends messages to your system, it is likely that you need to inform the test bed on how to do so.
Providing and reviewing the configuration for your system is done through the Configuration parameters section of the conformance statement detail page (see Configuration parameters).
Once all required configuration is provided you can choose to execute one or more test cases through the conformance statement detail’s Conformance tests section (see Tests). The test execution process starts by clicking one of the available Play buttons. In short, you can either execute a specific test case or a complete test suite and choose whether the test sessions will be launched in the background or in interactive mode (the default).
Background execution¶
Launching tests in the background is done by toggling the execution mode button to Background execution.
With this set you click the Play button to launch a test suite or a specific test case. Before doing so the test bed will verify that all required configuration properties are defined, and will display a popup notification for those that are missing.
The missing information is presented to you in sections depending on its type:
Organisation properties: Properties at the level of the whole organisation.
System properties: Properties at the level of the system being tested.
Conformance statement parameters: Configuration parameters linked to the specific conformance statement.
In each case you are presented with the following information:
The name of the property or parameter (marked with an asterisk if mandatory).
The information’s description.
From this point you have the following options:
Click the Close button in the bottom right corner to return to the conformance statement detail screen.
Click one of the View buttons on top right corners of the presented tables to access the configuration in question.
Once all required information is correctly defined you can proceed to execute your test(s). Doing so will launch the test sessions in the background where they will proceed to run in parallel. A visual confirmation of the launched test session will briefly appear in the top right area of the screen informing you of this.
The status of test sessions launched in the background can be monitored by means of the Test Sessions screen.
Interactive execution¶
Launching tests interactively is the default option and is enabled by having the execution mode toggle button set to Interactive execution.
Proceeding to execute tests will result in a three-step process:
Verification of the configuration you provide to the test bed.
Communication of any configuration provided by the test bed to you.
Test execution.
Throughout these steps you are presented on top with the list of test cases to execute. Each test case is displayed with its name and description as well as an information button in case the test case in question defines further extended documentation.
The currently active test case is always highlighted in blue, whereas the status column indicates the pending test case as well as the overall result of the ones previously executed. Regarding the information button, if present, clicking it will open a popup displaying the test case’s extended documentation.
Step 1 - Verification of your configuration¶
The first step of executing one or more test cases is the verification on your provided configuration. If you are expected to enter required information that is missing you will be presented with a screen listing the missing properties.
This screen presents on the top the test cases to execute but puts the focus on the missing information. This is presented to you in sections depending on its type:
Missing organisation properties: Properties at the level of the whole organisation.
Missing system properties: Properties at the level of the system being tested.
Missing conformance statement parameters: Configuration parameters linked to the specific conformance statement.
In each case you are presented with the following information:
The name of the property or parameter (marked with an asterisk if mandatory).
The information’s description.
From this point you have the following options:
Click the Back button in the bottom left corner to return to the conformance statement detail screen.
Click one of the View buttons on top right corners of the presented tables to access the configuration in question.
Once all required information is correctly defined you can proceed to execute your test(s).
Note
Valid configuration: If you are not required to provide any information or all required information is correctly provided this screen will be skipped. You will be taken directly to the display of the simulated actors’ configuration (see Step 2 - Simulated actor configuration).
Step 2 - Simulated actor configuration¶
The second step when executing your test(s) is the Preliminary phase screen where you again see the test case(s) to execute but also see the configuration information generated for you by the test bed. The nature of this configuration depends on the tests you are executing but in all cases refers to information that the test bed is communicating to you that you may likely need to configure in your system before starting. If for example your system is expected to send messages to the test bed this step informs you what you need to configure as the test bed’s address.
The configuration properties displayed here are in fact parameters (see Endpoint) that are listed with their names and values under the name of the specification actor that is being simulated. Continuing the example, if the test bed is going to simulate a specification actor named “EU portal” to which you are expected to send messages, the simulated actor name (in the example “EU portal”) is presented before its configuration values.
Apart from the display of such configuration parameters, it could also be the case that this step presents you with additional notification popups to provide you with further information or instructions. The existence or not of such a popup as well as its contents are defined within each test case.
It may be interesting to note that when being presented with this screen, a test session has already been started in the test bed. In case you are executing a complete test suite (see Tests), the information presented to you corresponds to the setup of the session for the first test case. Once this completes, it could be that this screen reappears if any new configuration values have been added or if any previously communicated ones have changed.
At this point you may also click the Back button from the bottom left corner to cancel the execution and return to the conformance statement detail page.
Note
No simulated configuration: This screen is presented to you only if preliminary steps are needed before starting a test session. If the test bed has no configuration to report to you and there are no specific instructions to communicate, this screen will be skipped. You will be taken directly to the test execution screen (see Step 3 - Test execution).
Step 3 - Test execution¶
The third step when executing your test(s) is the Execution screen. This continues the display the test case(s) on top but now also shows you the test steps from the first test case to execute. In addition, a greyed-out cog icon is now presented under the its status indicating that this test case is ready for execution but has not started yet.
The Execution section displays the upcoming test case’s steps in a way similar to a sequence diagram. The elements included in this diagram are:
A lifeline per actor defined in the test case. One of these will be marked as the “SUT” (the System Under Test), whereas the other actor lifelines will be labelled as “SIMULATED”. An additional operator lifeline may also be present in case user interaction is defined in the test case.
Expected messages between actors represented as labelled arrows indicating the type and direction of the communication.
A Test Engine lifeline in case the test case includes validation or processing steps that are carried out by the test bed that don’t relate to a specific actor.
Zero or more cog icons, typically under the “Test Engine” indicating the points where validation or processing will take place.
Visual grouping elements that serve to facilitate the display in case of e.g. conditional steps, parallel steps or loops.
If multiple test cases are being executed (i.e. a complete test suite is selected for execution) the display remains similar but the test case overview section now shows the full list of upcoming tests.
In this case the displayed diagram refers to the test case that is up next for execution, indicated also as such by highlighting in blue the relevant row from the test case overview table.
Starting the test session is achieved by clicking the provided Start button. In case you are executing a single test case this is present in the test case overview header. Otherwise, if multiple test cases are to be executed, the Start button is presented in the display of the specific test case.
Finally, throughout the execution of all test cases you may click the Back button in the bottom left corner to leave the interactive test execution and return to the conformance statement detail page. Doing so will continue to execute the ongoing and pending test cases in the background that you can now follow from the Test Sessions screen.
Monitor and manage test progress¶
Clicking the Start button begins the first selected test case’s session. What follows depends on the definition of the test case as illustrated in the presented diagram but can be summarised in the following types of feedback:
Exchanges of messages between actors (i.e. the displayed arrows) proceed. Messaging initiated by the test bed happens automatically, whereas for messages originating from your system the test session blocks until you trigger them, e.g. through your software component’s user interface.
Popup dialogs relative to interaction steps are presented to either inform you or request input.
Validation or processing steps take place automatically.
During the execution of the test case colours are used to inform on each step’s status:
Blue is used to highlight the currently active or pending step. This could be a blue arrow showing that a message is expected or a spinning blue cog to show active processing.
Grey is used for all elements that haven’t started yet or that have been skipped (e.g. due to conditional logic). Skipped steps are also displayed with a strike-through to enhance the fact they have been skipped.
Green is used for steps that have successfully completed.
Red is used for steps that have failed with a severity level of “error”.
Orange is used for steps that have failed with a severity level of “warning”.
The colour-based feedback is also repeated at the level of the test case overview in the status cog icons. The icon’s colour serves to highlight the currently active test case, versus future ones or completed ones (in case of multiple test cases being up for execution). Once completed the status icon for the test case is replaced by a green tick or red cross to indicate the session’s overall result as a success or failure respectively. Note that a test session is considered as failed if it contains at least one error; warnings are displayed but don’t affect the overall test outcome (i.e. in the presence of warnings and no errors the overall test result will be successful). Regardless of the outcome of individual steps, test execution always continues as even in the presence of errors it could be still interesting to proceed (e.g. if multiple different validation steps take place).
In case multiple test cases are up for execution, testing proceeds automatically, only pausing in case user interaction is needed. Such user interaction can either be a step within a test case or part of the setup for the next test session. In the latter case, automatic test execution is paused and can be restarted by clicking again the test case’s Start button.
Stopping the test(s) execution is achieved through controls that replace the Start button. Specifically:
In case of a single test case a Stop button is displayed in the test case overview header. Clicking this immediately stops the test session.
In case of multiple test cases being executed, The Stop button is presented at the level of the currently active test case and results in stopping the specific test session but proceeding with the subsequent test cases. Stopping the entire test suite execution is achieved through the Stop all button displayed in the test case overview header.
Once all test cases are complete the test case overview displays a Reset button. This serves as shortcut to re-run the same test case(s).
View test step documentation¶
Test steps are presented in the test execution diagram with a limited description label. Test steps can however be defined to also include additional detailed context, documentation, or instructions. Test steps defining such additional documentation are presented with a circled question mark next to their label that can be clicked.
Clicking the presented icon results in a “Step information” popup that displays the further documentation linked to the step. This can range from being a simple text to rich text documentation, including styled content, tables, lists, links and images.
Clicking the Close button or anywhere outside the popup will dismiss it and refocus the test execution diagram.
Note
Test documentation and instructions: Providing extended documentation for key steps is a good way of enriching the feedback provided to
users. This documentation can be used to provide detailed instructions or references to the specifications being tested, complementing the
limited information presented through test step labels, or test case and test suite descriptions. Such documentation is added in the test
cases’ GITB TDL content by means of the test steps’ documentation element.
View test step results¶
During test case execution, additional controls are made available to allow you to inspect the ongoing test(s) results.
First of all, if multiple test cases are selected for execution, completed test case sessions can be inspected by clicking their respective row from the test case overview table. Doing so will replace the currently displayed test session diagram with the one relevant to the clicked test case. Note that if there is a currently running test session that results in a test step update, the displayed diagram will be automatically replaced with the one for the active test session.
Regarding the test steps within a given test session, each completed step displays a clickable control in the form of a document with a green tick or red cross (for success or failure respectively). This applies both for validation steps and messaging steps.
Apart from serving as an additional indication on the success or failure of the test step, these controls provide further details on the step’s results. In case of messaging steps, this triggers a popup that shows the different information elements that can be viewed inline or opened in a separate popup editor. In the case of validation steps, this is extended to also provide the detailed validation results and an overview of the error, warning and information message counts, as illustrated in the following example.
In the test step result popup you are presented with the result and completion time as the step summary. In the sections that follow you can inspect the output information from the step, presented either inline (for short values), as a file you can download, or through a further popup editor. These two latter options are available by clicking the download or view icons respectively at the right of each section. In case you choose to view the content in an editor, a popup is presented that displays the content which, in the case of validation steps, is also highlighted for the recorded validation messages.
The editor popup allows you to copy a specific part of the content or, by means of the Copy to clipboard button, copy its entire contents. The Close button closes this popup and returns you to the test step result display. Note that clicking on a specific error will open the validated content and automatically focus on the selected error.
An alternative to viewing the content in this way is to click the download button which will download the content as a file. The test bed will determine the most appropriate type for the content and name the downloaded file accordingly (if possible).
Note
Viewing binary output: The Download as file option is the best way to inspect information that is binary (e.g. an image). The test bed will nonetheless always present the Open in editor option but given that the content is then assumed to be text, this will likely not be useful.
The errors, warnings and information messages displayed are contained in a details section that also shows the overall counts per violation severity level. This summary title is also clickable, to allow the listed details to be collapsed or expanded if already collapsed. Collapsing the displayed details could be useful in case they are numerous, providing as such easier access to the popup’s additional controls.
The results of the test step can also be exported as a test step report (in PDF format). This is made available through the Export button that triggers the generation and download of the step report.
This report includes:
The test step result overview, including the result, date and, in case of a validation step, the total number of validation findings (classified as errors, warnings and messages).
The report details, included in case of a validation step to list the details of the validation report’s findings.
The step’s context information, to list its output values and returned content.
Note
Test step report size: When exporting a test step report the context information is always included to provide the full information pertinent to its result. In case the context information returned by the step includes large file contents, these would be included resulting in a potentially very large export.
Finally, it is important to point out that the examination of a test session’s result, both in terms of steps and message exchanges, as well as detailed test step results, is possible at any time through your test session history (see View your test history).
View test session log¶
During any point in a test session’s execution you may view its detailed log output. This is done by clicking the View log button in the bottom left corner next to the Back button.
Clicking this will open a popup window that includes the detailed log output (debug statements, warnings and errors) for your test session.
The detailed log output is typically very useful when you receive error messages but for which the description provided is not clear. The log output may be used in such a case to determine the cause of the problem or, for unexpected issues, provide input to the test bed support team (see Contact support).
The test session log popup presents you with three options:
Copy to clipboard, to copy the entire log output to your clipboard (you may also of course selectively copy specific sections).
Download, to download the log output as a text file.
Close, to close the popup.
View your test history¶
All tests you carry out on the test bed are linked to specific conformance statements, each linking one of your organisation’s systems to a specification actor. The first step in viewing your test session history is to select the TESTS button from the screen header.
You can then select the system you are interested in from the listing of your organisation’s systems (see Manage your systems).
Selecting a system from the list brings you to its list of conformance statements (see View your conformance statements). From this point you will be able to click the Test Sessions link from the left side menu.
Clicking this link presents you with the test session history screen. This screen is split into three main parts:
A set of search filters to help locating specific test results.
The list of active test sessions.
The list of completed test sessions.
Active test sessions¶
The currently active sessions are those that are pending completion. These could be sessions that are running in the background or sessions that are interactively being executed by other users.
Each session is presented on a separate table row, with the following information displayed per session:
The specification and actor (defined as the test case’s SUT).
The relevant test case.
The session start time.
The information displayed in the table is sorted using the sessions’ start time in ascending manner (i.e. the oldest sessions are presented first). Sorting can be adapted by clicking on each column’s header to sort by it in ascending manner. The currently active sort column and type are displayed using an arrow icon next to the relevant column’s title.
The set of currently displayed active sessions can be exported in CSV format by clicking the Export CSV button in the table header (see Export test sessions). Finally, each session’s row offers controls to:
Forcibly terminate it, by clicking the cross icon on the relevant session’s row under the Operation column.
View its test step details, by clicking on the row itself (see View a test session’s steps).
Completed test sessions¶
The history of all your completed test sessions is presented in the Completed test sessions table.
Tests are presented in a paged table, offering controls to go to the first, previous, next and last pages as applicable, and are sorted based on their start time in a descending order (i.e. showing the latest tests at the top). Custom sorting can also be made by clicking the title of each column; clicking a column header for the first time will sort by it in ascending manner and clicking it again will switch to descending. The active sort column and type are indicated using an arrow next to the relevant column header.
Test sessions are displayed one per table row, with each row including the following information:
The specification and actor of the test session.
The relevant test case.
The session’s start and end time.
The test result.
Each row provides controls to export the relevant test case report and to view the test’s steps. In addition, you can use the overall Export CSV button to extract a CSV export of the currently displayed sessions (see Export test sessions).
Note
Obsolete test sessions: One or more test sessions may be rendered obsolete in case of a significant change in the test setup (e.g. the relevant specification being deleted) or a test case update that requires relevant test sessions to be re-executed. Such test sessions remain and can be consulted but are displayed greyed-out to indicate that they are no longer considered towards your overall conformance testing.
Apply search filters¶
When you first visit this screen the filter controls are initially ignored, indicated by the relevant toggle button being indicated as Disabled. Clicking on this sets it to Enabled and displays the filtering criteria that you can use to filter the test sessions.
The available filters are:
The domain, specification and actor of the sessions’ corresponding conformance statements.
The sessions’ test suite and test case.
The sessions’ result.
The sessions’ start and end time.
A specific session ID.
All filter controls with the exception of the start time, end time and session ID are multiple selection choices. The start and end time controls are date pickers that allow selection of ranges of dates for both the start and end of the sessions. The session ID is a text field. Selecting multiple values across these controls are applied as follows:
Within a specific filter control using “OR” logic (e.g. selecting multiple specifications).
Across filter controls using “AND” logic (e.g. selecting a specification and a test case).
Note additionally that selecting dependent values serves to limit the filter options that are presented. For example if a given specification is selected, the test suites and test cases available for filtering will be limited to that specification to already exclude impossible combinations.
The presented tests are automatically updated whenever your filter options are modified, or when the filters are removed altogether by setting the filter toggle button to Disabled. It is also possible to collapse the filters without disabling them by clicking on the filter row. Note that displaying the performed tests with no filtering is the default when you first visit the screen. Finally, you may also choose to keep the current filtering but refresh the search results by clicking the Refresh button.
Export a test case report¶
Exporting a test case’s report is possible for completed test sessions, through the file icon control on the far right side of each test’s row.
Clicking this will generate and download the report (in PDF format).
The test case report contains a first Overview section that summarises the purpose and result of the test session. The information included here is:
The name of the system that was tested and the name of its related organisation.
The names of the domain, specification and actor of the relevant conformance statement.
The test case’s name and description.
The session’s result, start and end time.
The overview section is then followed by a section per test case step, each starting on a separate page.
The information displayed for each step is:
Its sequence number.
Its name.
Its result.
Its completion time.
For validation steps, the number of validation report findings classified as errors, warnings and messages.
For validation steps, a Details section listing the details of each validation finding.
Note
Step context values: The information included in the test case report for each step does not include the context information relevant to the step’s output results. This is omitted as the report would in most cases end up being very large.
Export test sessions¶
Apart from exporting an individual test case report you can also export the information for the currently displayed test sessions in CSV format. To do this click the Export CSV button in the right of the active or completed test session table header.
Delete obsolete test results¶
Obsolete test sessions can be deleted by clicking the Delete obsolete sessions button from the search results’ panel.
Doing so will first prompt you for confirmation and then, if confirmed, will proceed to delete the obsolete test results. Note that the results deleted are limited to those specific to the system that is currently selected.
View a test session’s steps¶
Each row from the list of presented test sessions, both active and completed, may also be clicked to view its detailed steps. Doing so expands the row to present the test session’s steps in a manner similar to the live test execution diagram displayed while the test session is active (see Step 3 - Test execution).
In terms of provided controls, a document icon is presented on steps that produced a report that can be clicked to review its details (see View test step details). In addition, the diagram’s header presents the session’s test suite, test case and session identifier. Clicking on the session row will once again collapse the display. Note that once one or more session details are expanded the table’s header will display a Collapse all button that can be clicked to collapse all details.
View test step details¶
Clicking on a step’s document icon triggers a popup that shows the step’s different information elements that can be viewed inline, downloaded or opened in a separate popup editor. In the case of validation steps, this is extended to also provide the detailed results and report counters as illustrated in the following example for a validation failure.
In the test step result popup you are presented with the result and completion time as the step summary. In the sections that follow you can inspect the output information from the step, presented either inline (for short values), as a file you can download, or through a further popup editor. In the latter case this is triggered by clicking the Open in editor link. Clicking to open this, displays its content which, in the case of validation steps, is also highlighted for the recorded validation messages.
The editor popup allows you to copy a specific part of the content or, by means of the Copy to clipboard button, copy its entire contents. The Close button closes this popup and returns you to the test step result display. Note that clicking on a specific error will open the validated content and automatically focus on the selected error.
An alternative to viewing the content in this way is to click the Download as file link which will download the content as a file. The test bed will determine the most appropriate type for the content and name the downloaded file accordingly (if possible).
Note
Viewing binary output: The Download as file option is the best way to inspect information that is binary (e.g. an image). The test bed will nonetheless always present the Open in editor option but given that the content is then assumed to be text, this will likely not be useful.
Export test step report¶
The results of the test step can also be exported as a test step report (in PDF format). This is made available through the Export button that triggers the generation and download of the step report.
This report includes:
The test step result overview, including the result, date and, in case of a validation step, the total number of validation findings (classified as errors, warnings and messages).
The report details, included in case of a validation step to list the details of the validation report’s findings.
The step’s context information, to list its output values and returned content.
Note
Test step report size: When exporting a test step report the context information is always included to provide the full information pertinent to its result. In case the context information returned by the step includes large file contents, these would be included resulting in a potentially very large export.