Manage your conformance statements

Conformance statements serve to define your organisation’s testing goals by linking one of your registered systems with a specification’s actor. It is a system’s conformance statements that determine the test suites and test cases that will be presented to you to execute.

Your organisation’s conformance statements may either be configured by an administrator of your organisation or by the overall community administrator. From your perspective, conformance statements can be viewed but not modified, serving to organise and focus your testing activities for each specification.

View your conformance statements

To view your organisation’s conformance statements click the My conformance statements link from the menu. Doing so presents you with a screen listing each statement and its status.

../_images/conformance_statements_nonadmin.PNG

Conformance statements are made at the level of a system and as such, the first step is to select a system from the presented dropdown. If you have only a single system this appears preselected and its conformance statements are automatically loaded.

Note

Conformance statements need to be created by an administrator before you can review them and execute their tests.

Assuming you have conformance statements defined these will be presented in expandable panels, split and grouped based on their relevant specifications, specification groups and options (if applicable). If you do indeed have such groupings, related statements can be expanded and collapsed by clicking on their relevant titles.

For each statement you can see besides the name of the specification, an overview of your current testing status. This overview consists of:

  • The last update time, corresponding to the last time the status of the conformance statement was updated.

  • The result counts, showing the number of tests in the conformance statement that are completed, failed or incomplete.

  • The result ratios, illustrating the same results but as a percentage of the total tests in the statement.

  • The overall status of the statement which can be successful, failed, or incomplete, based on the latest test results.

In case you have numerous statements, you can use the provided search controls to filter them based on:

  • The specifications’ name.

  • The overall status.

../_images/conformance_statements_filters.png

It could be that certain test cases configured for the specification are optional in nature. Such tests can be consulted but are not counted towards your conformance testing status. If you do have optional tests, the displayed counts and ratios will present a plus button to expand their display allowing you to consult both mandatory and optional tests.

../_images/conformance_statements_optional_counts.png
../_images/conformance_statements_optional_ratios.png

Clicking any conformance statement row will take you to the conformance statement’s details from where you can see further information on the statements’ test cases and execute new test sessions.

View a conformance statement’s details

The conformance statement detail screen provides you the test status summary for a given system of your organisation and a specification’s actor. In addition it is the point from which you can start new tests. The information displayed in this page is organised in three sections to present to you:

  • The details of the conformance statement.

  • The configuration for your system, used when it is defined as a test case’s SUT.

  • The status and controls of the related tests.

Overview

The Conformance statement details section provides you the context of what your system is supposed to conform to.

../_images/conformance_statement_details_overview_nonadmin.PNG

At the top of the detail panel you see the name of the organisation and system for which this conformance statement has been made. Following this you see the name and description of the specification you are claiming conformance for, including any options (e.g. specification versions, profiles or roles) that apply. Finally, at the bottom of the panel you see the current status of the statement, specifically:

  • The overall status based on the latest test results (success, failure or incomplete).

  • The last update time, corresponding to the last time the status of the conformance statement was updated.

  • The result counts, showing the number of tests in the conformance statement that are completed, failed or incomplete.

  • The result ratios, illustrating the same results but as a percentage of the total tests in the statement.

Similar to the conformance statements’ listing, in case your statement includes optional test cases, the counts and ratios will display a plus button that can be clicked to expand and display both mandatory and optional tests. Note that optional and disabled tests do not count towards your conformance status.

../_images/conformance_statement_optional_counts.png
../_images/conformance_statement_optional_ratios.png

At the bottom of the details’ panel you are presented with buttons for further actions as follows:

  • The Download report button to export your system’s current conformance statement report.

  • The Download conformance certificate button to generate a conformance certificate for your system.

  • The Copy badge URL and Preview badge buttons to copy (or preview) a conformance badge for your current status. This is available only if your administrator had configured such badges.

  • View system allows you to navigate to your system or organisation details.

  • The Back button to return to the conformance statement list.

In addition, the overall detail panel can also be collapsed and expanded by clicking its header. Collapsing its display could be useful if you would want to focus on the tests to execute rather than the statement’s details.

Beneath the statement details’ panel you are presented with two tabs that allow you to interact and manage the conformance statement:

Using conformance badges

Conformance badges are an optional feature for specifications that may be set up by the community administrator. Badges are images that indicate a specific live status for a given organisation’s system, with respect to a specific conformance statement. They are meant to be accessible publicly so that they can be embedded in displays such as online dashboards or GitHub README files.

In case conformance badges are configured for the statement’s specification, the detail panel’s controls will also include a Copy badge URL button.

../_images/conformance_statement_details_controls.png

Clicking this will copy to your clipboard a URL that you can then refer to from outside the test bed to display a badge. A typical use case for this would be to add it as the source of an image in a HTML page listing conformant solutions. The same button also includes a secondary option named Preview badge that you can click for a preview.

../_images/conformance_statement_details_badge_preview.png

Note that the displayed badge is dynamically updated to always reflect the latest conformance testing status. For example if new test cases are added to the statement, accessing the same badge (displayed as a “success” badge above) will switch to an “incomplete” badge.

../_images/conformance_statement_details_badge_preview_incomplete.png

Note

By default conformance badges illustrate a “success” and “not success” state. It could be the case however that specific “failure” badges are also configured depending on your community’s setup.

Conformance tests

The Conformance tests tab lists the tests linked to the conformance statement. These are the tests that you need to successfully complete to be considered as conformant. The display includes two parts:

  • A set of controls to filter the displayed test cases and configure test execution.

  • The list of test cases included in the conformance statement.

../_images/conformance_statement_details_tests.png

The statement’s test cases are grouped by their test suite, of which the name is presented in bold, alongside its description. This test suite header can be clicked to expand or collapse the displayed test cases, which could be interesting if there are multiple test suites. Note that if only a single test suite is defined it appears as expanded by default.

Each test suite includes within it the listing of its test cases. The information displayed for each test case includes:

  • An icon indicating of whether it is required, optional, or disabled. Note that if only required tests are defined this icon is hidden.

  • Its name, a short text to identify and refer to the test case.

  • Zero or more tags that highlight specific traits relevant to the test case. These can also be hovered over to view further information.

  • The date and time of the test case’s last execution.

  • The status of the latest test session executed for the test case (displayed on the right).

  • The test case extended description, which is initially hidden but can be displayed if clicking on the test case row.

This information is complemented by the test case controls which depending on the status of the test case include:

  • A shortcut to view the latest test session executed for this test case in the test session history (if such a session exists).

  • An information button to view the test case’s extended documentation (if defined).

  • A play button to start a new test session for this test case (see Execute tests).

At the level of the test suite you can also view the aggregated status of the test suite’s test cases, as well as additional controls:

  • An information button to view the test suite’s extended documentation (if defined).

  • A play button to launch test sessions for all listed test cases.

When test sessions are completed for the statement’s different test cases, the displayed status will be adapted to present them as successful or failed. Moreover, in case a test session also produced a detailed output message, this can be viewed by clicking on the success or failure icon.

../_images/conformance_statement_details_tests_output_message.PNG

Above the display of test suites the Conformance tests tab also includes controls relevant to the displayed test cases.

../_images/conformance_statement_details_tests_controls.png

On the left side you are presented with controls to search test cases. You may use the provided search box to look for a specific test case, with the text you provide being used to match, in a case-insensitive way, test cases based on their name or description. Next to this you are provided with a dropdown menu that defines which tests are displayed based on their status. You may choose to show all tests (the default), or select successful tests, failed tests or incomplete tests. If there are also optional and disabled test cases you may also select (or deselect) their display from here as well. Any change to filtering options will update the display to list the matched test cases. It is important to note that when selecting to run test cases at the level of the displayed test suite, the test cases to be executed will be those currently displayed.

Note

Disabled test cases can be reviewed in case they have previous test results, but they are hidden by default and cannot be executed.

To the right side you may find a further dropdown menu that determines how test cases will be executed. The options presented here are the following:

  • Interactive execution, to launch tests in an interactive manner, presenting their test execution diagram and interacting with you for inputs.

  • Parallel background execution, to launch tests in the background executing them in parallel. Note that test cases that don’t support parallel execution will be ran sequentially.

  • Sequential background execution, to launch tests in the background executing them one by one in sequence.

Opting for background execution allows you to launch a potentially large number of test sessions without needing to oversee their progress. Care however needs to be taken here to ensure that all relevant test cases can be carried out without user interaction. If a test session running in the background defines user interaction steps, these are managed as follows:

  • Instructions are simply skipped, assuming that these are purely of informational value.

  • Input requests are completed automatically without input. Doing so will most likely cause a test session to fail (e.g. if a user is expected to provide the content of a message to send) but could still result in a successful completion if the test case has been designed to treat user input as optional.

The status of test sessions launched in the background can be monitored by means of the Test Sessions screen.

Finally, recall that the listed test suites and test cases may include an information button in case they define extended documentation. This documentation complements the displayed description with further information such as diagrams and reference links.

../_images/conformance_statement_details_tests.png

Clicking this button results in a popup window containing the extended documentation.

../_images/conformance_statement_details_tests_documentation_popup.PNG

Note that the documentation on test cases is also available to consult during their execution (in case of interactive execution).

Configuration parameters

Alongside the Conformance tests tab you are presented with the Configuration parameters tab. This includes any necessary configuration at the level of the specific conformance statement that you are expected to provide.

../_images/conformance_statement_details_endpoints.PNG

Configuration properties are displayed in rows where for each one the following information is presented:

  • Whether or not it is set.

  • Its parameter name, serving as its identifier. This is prefixed with an asterisk if the parameter is mandatory.

  • Its configured value, which in case it is a file will be presented as a download link.

  • Its description to help understand the parameter’s purpose.

Note

Editing configuration parameters: Editing your conformance statement’s configuration parameters is reserved to administrators.

Export conformance statement report

The conformance statement report (in PDF format) provides the details on the conformance statement and also an overview of its relevant tests. To generate it click the Download report button from the overview section’s panel.

../_images/conformance_statement_details_overview_nonadmin.PNG

Once the button is clicked you will be prompted for the level of detail you want to include in the report. Two options are available regarding whether or not you want to include each test case’s step results in the report.

../_images/conformance_statement_report_detail_prompt.PNG

Selecting Yes includes the conformance statement details and test overview but also each test case’s step results. Selecting No on the other hand skips the test step results.

The following sample illustrates the information that is included in the report’s overview section that is always included. Specifically:

  • The information on the domain, specification and actor for the selected system.

  • The name of the system’s organisation and the system itself.

  • The date the report was produced.

  • The overall conformance status, the number of successfully passed test cases versus the total as well as result percentage ratios.

  • A listing of the statement’s test suites, each including its test cases and their result. Test cases are presented as a link to take you directly to the relevant result (if test steps were selected to be included).

../_images/conformance_statement_report_sample.png

If the listed test cases include optional or disabled test cases, or if any of them define tags, the listing of test cases is followed by a legend explaining the meaning of the different status icons and tags.

../_images/conformance_statement_report_legend.png

In case the option to add each test case’s step results is selected, the report includes a page per test case displaying its summary and the result of each test step. The test case’s title includes its reference number listed in the report’s overview section, and provides also a link to return to the listing of test cases.

../_images/conformance_statement_report_sample_test_case.png

Note

Detailed report size: The detailed conformance statement report presents each test session and individual step in a separate page. If your conformance statement contains numerous test cases, each with multiple test steps, the resulting detailed report could be quite long.

Export conformance certificate

The conformance certificate is a report (in PDF format) that attests to the fact that your current system has successfully passed its expected test cases. The option to generate this is only visible if your system has succeeded in all configured tests. If this is the case and the option is still not visible, this means that your community administrator has disabled this feature. In such a case you will need to contact your administrator to obtain it.

Assuming the option is available for you, clicking the button will generate the certificate and prompt you for its download. The certificate will typically resemble the following sample:

../_images/conformance_statement_certificate_sample.png

The contents of the certificate are defined by your administrator and are a customisation of the conformance statement report. The certificate may omit certain sections, include a message for you, and potentially be digitally signed.