Manage your conformance statements

Conformance statements serve to define your organisation’s testing goals by linking one of your registered systems with a specification’s actor (see Conformance statement). It is a system’s conformance statements that determine the test suites and test cases that will be presented to you to execute.

As administrator for your organisation you are responsible for configuring its conformance statements. Note however that conformance statements may also be managed for you by your community administrator.

View your conformance statements

Conformance statements are made at the level of a system and as such, the first step is to select one of the systems configured for your organisation (see Manage your systems).

This table presents for the selected system its list of conformance statements in terms of their domain, specification and actor. Simply put this set of information serves to uniquely identify the specification role that your system aims to play, thus determining the test cases that it should execute. The presented test results also provide you an overview of the latest test results, showing you how many configured tests your system has successfully passed up to this point and, for the ones not passed, their current status (“undefined” or “failed”). Finally, the overall conformance status is also displayed per statement indicating its current result as undefined, failed or successful.

From this table you can click any row to proceed to the conformance statement’s details (see View a conformance statement’s details). You can return to the listing of conformance statements at any time by clicking the Conformance Statement entry in the left side menu.

Create a conformance statement

Note

Missing create option: The create conformance statement option may be missing if your community administrator has disabled the management of conformance statements by organisation users.

To create a new conformance statement for your selected system click the Create conformance statement button from the top right side of the conformance statement listing’s header. This presents to you a wizard to construct your conformance statement.

The first step is to select the domain relevant to your conformance statement. Note that this screen will typically be skipped if you are associated to a community linked to a single domain, considering the domain as selected by default.

If you do have the choice over multiple domains select the one you want by clicking its row. With the relevant row highlighted you can now click the Next button to proceed or the Cancel button to abort. The next step presented is the list of specifications for the selected domain.

Similarly if the selected domain defines only a single specification this will be considered as selected by default and this step will be skipped. If you have multiple specifications to choose from select the one you want by clicking its row. You can now click on the Next button to proceed (or, as previously, the Cancel button to abort). The final step presented is the list of actors defined for the specification.

This step is skipped if your selected specification defines a single actor or if your community administrator has flagged an actor as the specification’s default. If this is not the case select the actor from the list by clicking its row and then on the Next button (or the Cancel button to abort). As a final step you will be presented with a confirmation screen in which the details of your conformance statement are presented (domain, specification and actor).

To complete the creation of the conformance statement click the Confirm button. Alternatively you can, as in the previous steps, abort by clicking the Cancel button. In either case you will be returned to the listing of your system’s conformance statements (see View your conformance statements).

Note

Automatic conformance statement selection: It is often the case that organisations are part of a community for which only a single domain, specification and actor are available for selection. If this is your case, clicking the Create conformance statement button will immediately present you with a confirmation screen without the need to make further selections.

View a conformance statement’s details

The conformance statement detail screen provides you the test status summary for a given system of your organisation and a specification’s actor. In addition it is the point from which you can start new tests. The information displayed in this page is organised in three sections to present to you:

• The details of the conformance statement.

• The configuration for your system, used when it is defined as a test case’s SUT.

• The status and controls of the related tests.

Overview

The Conformance statement details section provides you the context of what your system is supposed to conform to.

The domain details are presented on the top as the high-level description of the project you are testing for. The specification information follows to define the specification you have chosen for your system to conform to (a domain may have multiple specifications). The actor information defines the specific role your system is expected to fulfil as part of this specification (a specification may have multiple actors). The test results present an overview of the testing progress for the conformance statement’s test cases, whereas the status represents the statement’s current progress. Below this section you are presented with buttons for further actions as follows:

• The Download conformance certificate button to generate a conformance certificate for your system (see Export conformance certificate).

• The Download report button to export your system’s current conformance statement report (see Export conformance statement report).

• The Delete statement button to delete the conformance statement (see Delete conformance statement).

Configuration parameters

The next section displayed is the information on the system’s Configuration parameters, the information that you are expected to provide to the test bed for your system pertinent to the current conformance statement. These parameters are typically configured once and for this purpose are displayed by default in collapsed form:

Clicking on the Hidden toggle button on the right-hand side will set it to Visible and in doing so, expand the panel to display the details for all properties:

Configuration properties are presented in rows where for each one the following information is presented:

• Whether or not it is set.

• Its parameter name, serving as its identifier. This is prefixed with an asterisk if the parameter is mandatory.

• Its configured value.

• Its description including helpful text to understand its meaning.

• An edit icon to change or remove its value.

In case of parameters that are binary files, these are presented with a link to download them. The following example shows such a parameter, along with additional text values.

To edit a configuration parameter click its edit icon on its relevant row. Doing so will open a prompt that presents the parameter’s name, description and current value that is editable.

In case of a parameter that is a file, the popup will be adapted to allow you to download the file and upload a replacement.

A third scenario is that of a parameter being a secret value (e.g. a password). In this case you are prompted to provide and confirm a new value.

Finally, an additional scenario is when preset values are defined for the parameter. In this case you are presented with a dropdown selection list that includes the available options.

To change the parameter’s value click on Save. Clicking on Delete will clear the current value, whereas Cancel will close the popup without making changes.

Certain parameters may also be non editable, even if displayed as required. The management of such parameters is reserved to your community administrator.

Finally, note that the complete Configuration parameters section may be missing in case your system is not expected to provide any information before executing its tests.

Note

Editing configuration parameters: Editing a conformance statement’s configuration parameters is a prerequisite that you, as organisation administrator, has to complete as your basic users lack the privileges to do so.

Tests

At the bottom of the page you can find the Conformance tests section.

This is a table displaying the tests you are expected to execute for the given conformance statement as well as the latest test results. The information displayed for each test case is:

• Its name, a short text to identify and refer to the test case.

• Its description, providing the context you need to understand the purpose of the test case and plan for its execution.

• Its latest result which can be either a success (green tick), failure (red cross) or undefined (grey bar) in case the test case has never been executed.

• A Play button to start a new test session for this test case (see Execute tests).

Note that the screenshot above indicates a simple conformance statement that contains a single test suite containing in turn a single test case. In this case the test suite is hidden for simplicity. A more elaborate conformance statement would typically include multiple test cases, each addressing a different scenario to be tested. The following screenshot is from a conformance statement for which a single test suite is defined but that contains multiple test cases.

Notice in this case that apart from the individual button to execute each test case, you also have a similar button in the section’s header. Clicking this will proceed to automatically and sequentially execute all listed test cases.

Finally, a further complex conformance statement could define multiple test cases organised in multiple test suites. In this case each test suite becomes important and is presented in the list of tests.

The table in this case displays at a first level the list of test suites, using a grey backdrop to differentiate them from test cases. Similar to test cases, each test suite displays its name, description and overall result, and can be clicked to expand and display its test cases.

Clicking on an expanded test suite collapses it again. Notice in addition that the Play button to execute test cases now displays at two levels:

• For each test suite: To automatically and sequentially execute all the test suite’s test cases.

• For each test case: To execute the specific test case.

When test sessions are completed for the statement’s different test cases, the result display will be adapted to display them as successful or failed. Moreover, in case a test session also produced a detailed output message, this can be viewed by clicking on the success or failure icon.

In all previous examples you may have noticed a Interactive execution toggle button displayed in the Conformance tests header. This toggle defines the manner in which you will execute the tests, defaulting to sequential interactive sessions. This means that each of the tests will execute in sequence and synchronously, with the current progress displayed to you and any steps requiring your input prompting you as needed. The alternative option, enabled if you click the toggle button, is Background execution that will launch the tests sessions in parallel and run them in the background.

Opting for background execution allows you to launch a potentially large number of test sessions without needing to oversee their progress. Care however needs to be taken here to ensure that all relevant test cases can be carried out without user interaction. If a test session running in the background defines user interaction steps, these are managed as follows:

• Instructions are simply skipped, assuming that these are purely of informational value.

• Input requests are completed automatically without input. Doing so will most likely cause a test session to fail (e.g. if a user is expected to provide the content of a message to send) but could still result in a successful completion if the test case has been designed to treat user input as optional.

The status of test sessions launched in the background can be monitored by means of the Test Sessions screen.

A further point to mention regarding the selection of test cases to execute is the possibility to view their extended documentation. Test cases and test suites are typically defined to include a brief description that provides context and high-level instructions. This is the description displayed in the test case listing discussed up to this point. The authors of the test cases may have provided in addition to this description, extended documentation that provides further information such as diagrams and reference links. Such documentation may exist at test suite and/or test case level, in which case you will see an additional information button displayed where available.

Clicking this button will result in a popup window containing the extended documentation.

Note that the documentation on test cases is also available to consult during their execution (in case of interactive execution).

Export conformance statement report

The conformance statement report (in PDF format) provides the details on the conformance statement and also an overview of its relevant tests. To generate it click the Download report button from the overview section’s panel.

Once the button is clicked you will be prompted for the level of detail you want to include in the report. Two options are available regarding whether or not you want to include each test case’s step results in the report.

Selecting Yes includes the conformance statement details and test overview but also each test case’s step results. Selecting No on the other hand skips the test step results.

The following sample illustrates the information that is included in the report’s overview section that is always included. Specifically:

• The information on the domain, specification and actor for the selected system.

• The name of the system’s organisation and the system itself.

• The date the report was produced and the number of successfully passed test cases versus the total.

• A table with the conformance statement’s test cases, displaying a row per test case with its reference number, the name of the the test suite and test case, the test case description and its test result.

In case the option to add each test case’s step results is selected, the report includes a section per test case displaying its summary and the results from each test step. The test case’s title includes its reference number listed in the report’s overview section.

Note

Detailed report size: The detailed conformance statement report presents each test session and individual step in a separate page. If your conformance statement contains numerous test cases, each with multiple test steps, the resulting detailed report could be quite long.

Export conformance certificate

The conformance certificate is a report (in PDF format) that attests to the fact that your current system has successfully passed its expected test cases. The option to generate this is only visible if your system has succeeded in all configured tests. If this is the case and the option is still not visible, this means that your community administrator has disabled this feature. In such a case you will need to contact your administrator to obtain it.

Assuming the option is available for you, clicking the button will generate the certificate and prompt you for its download. The certificate will typically resemble the following sample:

The contents of the certificate are defined by your administrator and are a customisation of the conformance statement report. The certificate may omit certain sections, include a message for you, and potentially be digitally signed.

Delete conformance statement

Deleting the conformance statement may be desired if you created it by mistake or if your system is no longer expected to conform to the given specification. Deleting the conformance statement is possible through the Delete statement button from the overview panel. If this button in missing this means that your community administrator has disabled the management of conformance statements for organisation users.

Clicking this will request confirmation and, if confirmed, will remove the conformance statement. Note that your testing history relevant to this conformance statement still remains and can be consulted through your test history (see View your test history). In addition, if you create the same conformance statement again, your previous tests will be once again counted towards your conformance testing status.

View selected system’s information

Once a system is selected from the list of your organisation’s systems (see Manage your systems) you can manage its conformance statements and view its test history. At any given time you can review the information of your selected system by clicking the System Information entry from the left side menu.

In this screen you can see the short and full name of the system, its description and its version number. If your community administrator has foreseen additional properties for systems you will also see here the Show properties checkbox. Clicking this will expand to also display the current system’s additional information.

The displayed properties can be simple texts, secret values (e.g. passwords) or files and, if supplied by your community administrator, will display a help tooltip to understand their meaning. Required properties are marked with an asterisk and will need to be completed before launching any tests for this system.

Note

Editing a system’s information: The information displayed on this screen is read-only. Editing the system’s information is possible through the screen listing your organisation’s systems (see Edit an existing system).