Manage your conformance statements

Conformance statements serve to define your organisation’s testing goals by linking one of your registered systems with a specification’s actor. It is a system’s conformance statements that determine the test suites and test cases that will be presented to you to execute.

Your organisation’s conformance statements may either be configured by an administrator of your organisation or by the overall community administrator. From your perspective, conformance statements can be viewed but not modified, serving to organise and focus your testing activities for each specification.

View your conformance statements

To view your organisation’s conformance statements click the My conformance statements link from the menu. Doing so presents you with a screen listing each statement and its status.

Conformance statements are made at the level of a system and as such, the first step is to select a system from the presented dropdown. If you have only a single system this appears preselected and its conformance statements are automatically loaded.

Note

Conformance statements need to be created by an administrator before you can review them and execute their tests.

Assuming you have conformance statements defined these will be presented in expandable panels, split and grouped based on their relevant specifications, specification groups and options (if applicable). If you do indeed have such groupings, related statements can be expanded and collapsed by clicking on their relevant titles.

For each statement you can see besides the name of the specification, an overview of your current testing status. This overview consists of:

• The last update time, corresponding to the last time the status of the conformance statement was updated.

• The result counts, showing the number of tests in the conformance statement that are completed, failed or incomplete.

• The result ratios, illustrating the same results but as a percentage of the total tests in the statement.

• The overall status of the statement which can be successful, failed, or incomplete, based on the latest test results.

In case you have numerous statements, you can use the provided search controls to filter them based on:

• The specifications’ name.

• The overall status.

It could be that certain test cases configured for the specification are optional in nature. Such tests can be consulted but are not counted towards your conformance testing status. If you do have optional tests, the displayed counts and ratios will present a plus button to expand their display allowing you to consult both mandatory and optional tests.

Clicking any conformance statement row will take you to the conformance statement’s details from where you can see further information on the statements’ test cases and execute new test sessions.

Conformance overview report

Besides viewing your conformance statements and their grouping on this screen, you can also generate conformance overview reports. These reports are a complement to the conformance statement reports available when viewing a specific conformance statement, focusing rather on a set of related conformance statements. Such overview reports are available at different levels depending on how specifications are configured, specifically (per decreasing aggregation level):

• Your overall status as an organisation (always available).

• A domain, when your conformance statements can cover more than one domains.

• A specification group, if groups are defined.

• A specification, when multiple actors can be tested for.

A report of your overall conformance status can be produced by clicking on the Download report button presented above the listing of statements. By default this will produce a PDF report, but clicking the presented caret for additional options allows you to also download the report in XML format.

The PDF report includes an overview section summarising the report’s context and status. This includes:

• The organisation and system the report refers to.

• The selected domain, specification group, or specification (skipped if the report refers to the overall status).

• The report date and status.

• The summary of conformance statement results, as counts and also percentage ratios.

Below the overview section, the report includes the listing of conformance statements presented in the same grouping as the on-screen display. Each statement includes its name and test result ratio and status.

Following the summary of conformance statements, the report includes each individual statement report that lists its specific status, test suites and test cases. In fact the name of each presented conformance statement from the summary is a link you may click to access the page of the relevant statement report.

Note

Each conformance statement report included in the overview report, matches the report you can produce separately for the statement’s detail screen. When included in an overview report there are however no extended details for specific test cases.

An alternative to producing the report in PDF is to select the Download report as XML option. The format of this report is defined by the GITB Test Reporting Language (GITB TRL), and allows simpler machine-based processing. The following XML content is a sample of such a report:

<?xml version="1.0" encoding="UTF-8"?>
<ConformanceOverviewReport xmlns="http://www.gitb.com/tr/v1/"
                            xmlns:ns2="http://www.gitb.com/core/v1/"
                            xmlns:ns3="http://www.gitb.com/tbs/v1/">
   <metadata>
      <reportTime>2024-04-02T11:53:07.170+02:00</reportTime>
   </metadata>
   <overview>
      <definition>
         <party>
            <organisation>
               <name>ACME</name>
            </organisation>
            <system>
               <name>Test system</name>
               <version>v1.0</version>
               <description>A system for test purposes.</description>
            </system>
         </party>
      </definition>
      <summary>
         <status>FAILURE</status>
         <succeeded>1</succeeded>
         <failed>1</failed>
         <incomplete>1</incomplete>
      </summary>
      <statementOverview>
         <domain>
            <name>Docs domain</name>
            <description>A demo domain for documentation purposes.</description>
            <result>FAILURE</result>
            <specificationGroup>
               <name>EU Purchase Order Import Format</name>
               <description>Specification to define the data export format for EU Purchase Order Imports.</description>
               <result>SUCCESS</result>
               <specification>
                  <name>Version 2.00</name>
                  <description>Version 2.00 of the EU Purchase Order Import Format.</description>
                  <result>SUCCESS</result>
                  <actor statement="1">
                     <name>Actor</name>
                     <description>Order supplier.</description>
                     <result>SUCCESS</result>
                  </actor>
               </specification>
            </specificationGroup>
         </domain>
      </statementOverview>
      <statementDetails>
         <statement id="1">
            <definition>
               <party>
                  <organisation>
                     <name>ACME</name>
                  </organisation>
                  <system>
                     <name>Test system</name>
                     <version>v1.0</version>
                     <description>A system for test purposes.</description>
                  </system>
               </party>
               <domain>
                  <name>Docs domain</name>
                  <description>A demo domain for documentation purposes.</description>
               </domain>
               <specificationGroup>
                  <name>EU Purchase Order Import Format</name>
                  <description>Specification to define the data export format for EU Purchase Order Imports.</description>
               </specificationGroup>
               <specification>
                  <name>Version 2.00</name>
                  <description>Version 2.00 of the EU Purchase Order Import Format.</description>
               </specification>
               <actor>
                  <name>Actor</name>
                  <description>Order supplier.</description>
               </actor>
            </definition>
            <summary>
               <status>SUCCESS</status>
               <succeeded>1</succeeded>
               <failed>0</failed>
               <incomplete>0</incomplete>
            </summary>
            <lastUpdate>2023-09-28T15:52:16.000+02:00</lastUpdate>
            <testOverview>
               <testSuite>
                  <metadata>
                     <ns2:name>Simple test suite</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>A simple test suite.</ns2:description>
                  </metadata>
                  <result>SUCCESS</result>
                  <testCases>
                     <testCase>
                        <metadata>
                           <ns2:name>Test case 1</ns2:name>
                           <ns2:version>1.0</ns2:version>
                           <ns2:description>A simple test case.</ns2:description>
                        </metadata>
                        <result>SUCCESS</result>
                        <lastUpdate>2023-09-28T15:52:14.000+02:00</lastUpdate>
                     </testCase>
                     <testCase optional="true">
                        <metadata>
                           <ns2:name>Test case 2</ns2:name>
                           <ns2:version>1.0</ns2:version>
                           <ns2:description>Description for the second simple test case.</ns2:description>
                        </metadata>
                        <result>SUCCESS</result>
                        <lastUpdate>2023-09-28T15:52:16.000+02:00</lastUpdate>
                     </testCase>
                  </testCases>
               </testSuite>
            </testOverview>
         </statement>
      </statementDetails>
   </overview>
</ConformanceOverviewReport>

Producing a conformance overview report at the level of a specific domain, specification group or specification, is achieved using the report icon buttons presented at the right side of the statements’ display. The first button is used to produce the report in XML and the second one in PDF.

The reports in either case are structured in the same way as the overall reports discussed above. The only difference is that a report at a specific aggregation level (e.g. a specification group) will also list the relevant level as part of the report’s overview information.

Conformance overview certificate

Depending on your community’s configuration you may also produce a conformance overview certificate. This certificate is a report (in PDF format) that attests to the fact that your current system has successfully completed testing for all relevant conformance statements. Such certificates are available for you to download if enabled by the community administrator, and if you have succeeded all relevant testing.

To produce a certificate you click on the PDF report generation buttons, either at the overall level or for a specific domain, specification group or specification. If generating a certificate is possible you will be prompted with a choice to select the kind of report to produce.

Selecting Conformance overview certificate and clicking Generate report will produce the certificate, which is similar in content to the normal overview report but is further curated by the administrator.

Note

The Conformance overview report option is the default if no certificate can be produced. In such a case clicking a PDF report download button will skip the report type prompt and directly download the overview report.

Select a conformance snapshot

A conformance snapshot represents a milestone in the conformance testing process that can be made by a community administrator. It is called a snapshot as it is a readonly copy of the conformance testing status at a given point in time, that captures the defined specifications, organisations and test results at that moment. If such snapshots are available you will see the header on this screen display an additional control to select which conformance testing status you want to view. By default you always view the latest conformance status.

Selecting a specific snapshot, or switching back to the latest status, will refresh the screen to present the relevant conformance statements. Note that a snapshot could also include elements such as specifications and systems that were defined at the time the snapshot was taken, but that may have since been modified or ever deleted. All controls here otherwise remain the same with the exception of not being able to define new conformance statements while a snapshot is selected.

In addition, when viewing a specific snapshot and selecting to view a statement’s details you will notice that the statement is presented as readonly, meaning that is cannot be deleted or have new test executions. In addition, you will see a message informing you that this statement does not represent the latest testing status.

View a conformance statement’s details

The conformance statement detail screen provides you the test status summary for a given system of your organisation and a specification’s actor. In addition it is the point from which you can start new tests. The information displayed in this page is organised in three sections to present to you:

• The details of the conformance statement.

• The configuration for your system, used when it is defined as a test case’s SUT.

• The status and controls of the related tests.

Overview

The Conformance statement details section provides you the context of what your system is supposed to conform to.

At the top of the detail panel you see the name of the organisation and system for which this conformance statement has been made. Following this you see the name and description of the specification you are claiming conformance for, including any options (e.g. specification versions, profiles or roles) that apply. Finally, at the bottom of the panel you see the current status of the statement, specifically:

• The overall status based on the latest test results (success, failure or incomplete).

• The last update time, corresponding to the last time the status of the conformance statement was updated.

• The result counts, showing the number of tests in the conformance statement that are completed, failed or incomplete.

• The result ratios, illustrating the same results but as a percentage of the total tests in the statement.

Similar to the conformance statements’ listing, in case your statement includes optional test cases, the counts and ratios will display a plus button that can be clicked to expand and display both mandatory and optional tests. Note that optional and disabled tests do not count towards your conformance status.

At the bottom of the details’ panel you are presented with buttons for further actions as follows:

• The Download report button to export your system’s current conformance statement report in PDF or XML, or download a conformance certificate.

• The Copy badge URL and Preview badge buttons to copy (or preview) a conformance badge for your current status. This is available only if your administrator had configured such badges.

• View system allows you to navigate to your system or organisation details.

• The Refresh button allows you to refresh the status of your conformance statement to view its latest results.

• The Back button to return to the conformance statement list.

In addition, the overall detail panel can also be collapsed and expanded by clicking its header. Collapsing its display could be useful if you would want to focus on the tests to execute rather than the statement’s details.

Beneath the statement details’ panel you are presented with two tabs that allow you to interact and manage the conformance statement:

• The Conformance tests tab to view and launch the statement’s tests.

• The Configuration parameters tab to view and edit the statement’s configuration parameters if needed.

Using conformance badges

Conformance badges are an optional feature for specifications that may be set up by the community administrator. Badges are images that indicate a specific live status for a given organisation’s system, with respect to a specific conformance statement. They are meant to be accessible publicly so that they can be embedded in displays such as online dashboards or GitHub README files.

In case conformance badges are configured for the statement’s specification, the detail panel’s controls will also include a Copy badge URL button.

Clicking this will copy to your clipboard a URL that you can then refer to from outside the test bed to display a badge. A typical use case for this would be to add it as the source of an image in a HTML page listing conformant solutions. The same button also includes a secondary option named Preview badge that you can click for a preview.

Note that the displayed badge is dynamically updated to always reflect the latest conformance testing status. For example if new test cases are added to the statement, accessing the same badge (displayed as a “success” badge above) will switch to an “incomplete” badge.

Note

By default conformance badges illustrate a “success” and “not success” state. It could be the case however that specific “failure” badges are also configured depending on your community’s setup.

Conformance tests

The Conformance tests tab lists the tests linked to the conformance statement. These are the tests that you need to successfully complete to be considered as conformant. The display includes two parts:

• A set of controls to filter the displayed test cases and configure test execution.

• The list of test cases included in the conformance statement.

The statement’s test cases are grouped by their test suite, of which the name is presented in bold, alongside its description. This test suite header can be clicked to expand or collapse the displayed test cases, which could be interesting if there are multiple test suites. Note that if only a single test suite is defined it appears as expanded by default.

Each test suite includes within it the listing of its test cases. The information displayed for each test case includes:

• An icon indicating of whether it is required, optional, or disabled. Note that if only required tests are defined this icon is hidden.

• Its name, a short text to identify and refer to the test case.

• Zero or more tags that highlight specific traits relevant to the test case. These can also be hovered over to view further information.

• The date and time of the test case’s last execution.

• The status of the latest test session executed for the test case (displayed on the right).

• The test case extended description, which is initially hidden but can be displayed if clicking on the test case row.

This information is complemented by the test case controls which depending on the status of the test case include:

• A shortcut to view the latest test session executed for this test case in the test session history (if such a session exists).

• An information button to view the test case’s extended documentation (if defined).

• A play button to start a new test session for this test case (see Execute tests).

At the level of the test suite you can also view the aggregated status of the test suite’s test cases, as well as additional controls:

• An information button to view the test suite’s extended documentation (if defined).

• A play button to launch test sessions for all listed test cases.

When test sessions are completed for the statement’s different test cases, the displayed status will be adapted to present them as successful or failed. Moreover, in case a test session also produced a detailed output message, this can be viewed by clicking on the success or failure icon.

Above the display of test suites the Conformance tests tab also includes controls relevant to the displayed test cases.

On the left side you are presented with controls to search test cases. You may use the provided search box to look for a specific test case, with the text you provide being used to match, in a case-insensitive way, test cases based on their name or description. Next to this you are provided with a dropdown menu that defines which tests are displayed based on their status. You may choose to show all tests (the default), or select successful tests, failed tests or incomplete tests. If there are also optional and disabled test cases you may also select (or deselect) their display from here as well. Any change to filtering options will update the display to list the matched test cases. It is important to note that when selecting to run test cases at the level of the displayed test suite, the test cases to be executed will be those currently displayed.

Note

Disabled test cases can be reviewed in case they have previous test results, but they are hidden by default and cannot be executed.

To the right side you may find a further dropdown menu that determines how test cases will be executed. The options presented here are the following:

• Interactive execution, to launch tests in an interactive manner, presenting their test execution diagram and interacting with you for inputs.

• Parallel background execution, to launch tests in the background executing them in parallel. Note that test cases that don’t support parallel execution will be ran sequentially.

• Sequential background execution, to launch tests in the background executing them one by one in sequence.

Opting for background execution allows you to launch a potentially large number of test sessions without needing to oversee their progress. Care however needs to be taken here to ensure that all relevant test cases can be carried out without user interaction. If a test session running in the background defines user interaction steps, these are managed as follows:

• Instructions are simply skipped, assuming that these are purely of informational value.

• Input requests are completed automatically without input. Doing so will most likely cause a test session to fail (e.g. if a user is expected to provide the content of a message to send) but could still result in a successful completion if the test case has been designed to treat user input as optional.

The status of test sessions launched in the background can be monitored by means of the Test Sessions screen.

Finally, recall that the listed test suites and test cases may include an information button in case they define extended documentation. This documentation complements the displayed description with further information such as diagrams and reference links.

Clicking this button results in a popup window containing the extended documentation.

Note that the documentation on test cases is also available to consult during their execution (in case of interactive execution).

Configuration parameters

Alongside the Conformance tests tab you are presented with the Configuration parameters tab. This includes configuration for the specific conformance statement that you are expected to provide, as well as your organisation and selected system.

Configuration properties are displayed in panels grouping together organisation properties, system properties and statement properties. For each property you see its name, value (which is a download link in case of a file) and a tooltip with a description. Properties that are required to be completed before executing tests are marked with an asterisk.

Note

Editing configuration properties: Editing your configuration properties is reserved to administrators.

Export conformance statement report

The conformance statement report provides the details on the conformance statement and also an overview of its relevant tests. To generate it click the Download report button from the overview section’s panel.

Once the button is clicked you will be prompted for the level of detail you want to include in the report. Two options are available regarding whether or not you want to include each test case’s step results in the report.

Selecting Conformance statement report (with test case results) includes the conformance statement details and test overview but also each test case’s step results. Selecting Conformance statement report on the other hand skips the test step results.

The following sample illustrates the information that is included in the report’s overview section that is always included. Specifically:

• The information on the domain, specification and actor for the selected system.

• The name of the system’s organisation and the system itself.

• The date the report was produced.

• The overall conformance status, the number of successfully passed test cases versus the total as well as result percentage ratios.

• A listing of the statement’s test suites, each including its test cases and their result. Test cases are presented as a link to take you directly to the relevant result (if test steps were selected to be included).

If the listed test cases include optional or disabled test cases, or if any of them define tags, the listing of test cases is followed by a legend explaining the meaning of the different status icons and tags.

In case the option to add each test case’s step results is selected, the report includes a page per test case displaying its summary and the result of each test step. The test case’s title includes its reference number listed in the report’s overview section, and provides also a link to return to the listing of test cases.

Note

Detailed report size: The detailed conformance statement report presents each test session and individual step in a separate page. If your conformance statement contains numerous test cases, each with multiple test steps, the resulting detailed report could be quite long.

An alternative to producing the report in PDF is to select the Download report as XML option presented by clicking to expand the download button’s options. The format of this report is defined by the GITB Test Reporting Language (GITB TRL), and allows simpler machine-based processing. As in the case of the PDF report, you are prompted whether or not to include detailed test step results before proceeding with the download. The following XML content is a sample of such a report:

<?xml version="1.0" encoding="UTF-8"?>
<ConformanceStatementReport xmlns="http://www.gitb.com/tr/v1/"
                             xmlns:ns2="http://www.gitb.com/core/v1/"
                             xmlns:ns3="http://www.gitb.com/tbs/v1/">
   <metadata>
      <reportTime>2024-04-02T14:13:18.267+02:00</reportTime>
   </metadata>
   <statement>
      <definition>
         <party>
            <organisation>
               <name>ACME</name>
            </organisation>
            <system>
               <name>Test system</name>
               <version>v1.0</version>
               <description>A system for test purposes.</description>
            </system>
         </party>
         <domain>
            <name>Docs domain</name>
            <description>A demo domain for documentation purposes.</description>
         </domain>
         <specificationGroup>
            <name>EU Purchase Order Export Format</name>
            <description>Specification to define the data export format for EU Purchase Order Exports.</description>
         </specificationGroup>
         <specification>
            <name>Version 1.02</name>
            <description>Version 1.02 of the EU Purchase Order Export Format.</description>
         </specification>
         <actor>
            <name>Actor</name>
            <description>Order supplier.</description>
         </actor>
      </definition>
      <summary>
         <status>FAILURE</status>
         <succeeded>2</succeeded>
         <failed>1</failed>
         <incomplete>0</incomplete>
      </summary>
      <lastUpdate>2023-09-29T19:00:01.000+02:00</lastUpdate>
      <testOverview>
         <testSuite>
            <metadata>
               <ns2:name>Simple TS</ns2:name>
               <ns2:version>0.1</ns2:version>
               <ns2:description>Simple test suite.</ns2:description>
            </metadata>
            <result>FAILURE</result>
            <testCases>
               <testCase>
                  <metadata>
                     <ns2:name>Simple TC</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>A simple test case.</ns2:description>
                     <ns2:tags>
                        <ns2:tag name="security" foreground="#ffffff" background="#d20000">Test cases linked to security issues.</ns2:tag>
                        <ns2:tag name="version 1.0" foreground="#ffffff" background="#000000">Test cases introduced in version 1.02.</ns2:tag>
                     </ns2:tags>
                  </metadata>
                  <result>SUCCESS</result>
                  <lastUpdate>2023-09-29T19:00:01.000+02:00</lastUpdate>
               </testCase>
               <testCase>
                  <metadata>
                     <ns2:name>Simple TC2</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>A second simple test case.</ns2:description>
                  </metadata>
                  <result>FAILURE</result>
                  <lastUpdate>2023-10-02T17:09:32.000+02:00</lastUpdate>
               </testCase>
            </testCases>
         </testSuite>
         <testSuite>
            <metadata>
               <ns2:name>Simple test suite</ns2:name>
               <ns2:version>1.0</ns2:version>
               <ns2:description>A simple test suite.</ns2:description>
            </metadata>
            <result>SUCCESS</result>
            <testCases>
               <testCase>
                  <metadata>
                     <ns2:name>Test case 1</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>A simple test case.</ns2:description>
                  </metadata>
                  <result>SUCCESS</result>
                  <lastUpdate>2023-09-28T15:52:14.000+02:00</lastUpdate>
               </testCase>
               <testCase optional="true">
                  <metadata>
                     <ns2:name>Test case 2</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>Description for the second simple test case.</ns2:description>
                  </metadata>
                  <result>SUCCESS</result>
                  <lastUpdate>2023-09-28T15:52:16.000+02:00</lastUpdate>
               </testCase>
            </testCases>
         </testSuite>
      </testOverview>
   </statement>
</ConformanceStatementReport>

Export conformance certificate

The conformance certificate is a report (in PDF format) that attests to the fact that your current system has successfully passed its expected test cases. The option to generate this is only visible if your system has succeeded in all configured tests. If this is the case and the option is still not visible, this means that your community administrator has disabled this feature. In such a case you will need to contact your administrator to obtain it.

Assuming the option is available for you, you can select the Download conformance certificate option after having expanded the download button’s options. The certificate will typically resemble the following sample:

The contents of the certificate are defined by your administrator and are a customisation of the conformance statement report. The certificate may omit certain sections, include a message for you, and potentially be digitally signed.