Monitor conformance status

The conformance status for your community is the summary of latest test results by all your member organisations for their declared conformance statements. Monitoring this summary is possible by means of the Conformance dashboard, accessible via the relevant menu link.

You can view your conformance dashboard in two ways, depending on your preference:

  • View per organisation, displaying all statements grouped and split according to their specifications for a specific organisation. This approach is best if you are interested in a specific organisation.

  • Detailed tabular view, displaying all statements for all organisations as a flat table. This approach os more suitable if you want to make complex filtering covering both organisations and specifications.

The default dashboard presentation is per organisation, considering that this is the most common dashboard use case. You can switch between both dashboard presentation modes by toggling the View per organisation control from the page’s header.

../_images/admin_conformance_dashboard_banner.png

Note

Conformance dashboard vs session dashboard: A significant benefit of the conformance dashboard is that the focus is placed on the latest result and also that even non-executed test cases are displayed. This allows you to get a clear picture of an organisation’s testing progress without needing to extrapolate information.

View per organisation

The dashboard display per organisation is the presentation mode active by default. It allows you to select a specific organisation and system within the community, to display the complete set of its conformance statements. These statements are grouped and split according to the organisation of your domains, specification groups, specifications and actors.

../_images/admin_conformance_dashboard_organisation_view.png

Assuming there are conformance statements defined these will be presented in expandable panels, split and grouped based on their relevant specifications, specification groups and options (if applicable). If you do indeed see such groupings, related statements can be expanded and collapsed by clicking on their relevant titles.

For each statement you can see besides the name of the specification, an overview of the system’s current testing status. This overview consists of:

  • The last update time, corresponding to the last time the status of the conformance statement was updated.

  • The result counts, showing the number of tests in the conformance statement that are completed, failed or incomplete.

  • The result ratios, illustrating the same results but as a percentage of the total tests in the statement.

  • The overall status of the statement which can be successful, failed, or incomplete, based on the latest test results.

The statement row itself can also be expanded by clicking it to view its details. These details include specific controls at the level of the statement, as well the listing of all its test suites and test cases with their latest test results.

In case numerous statements are defined, you can use the provided search controls to filter statements based on:

  • The specifications’ name.

  • The overall status.

../_images/admin_conformance_dashboard_organisation_view_filters.png

It could be that certain statements include optional test cases. Such tests can be consulted but are not counted towards the conformance testing status. If statements with optional tests exist, the displayed counts and ratios will present a plus button to expand their display allowing you to consult both mandatory and optional test results.

../_images/admin_conformance_dashboard_organisation_view_optional_counts.png
../_images/admin_conformance_dashboard_organisation_view_optional_ratios.png

From the organisation view of the conformance dashboard you can also generate conformance overview reports and conformance overview certificates. These reports capture the aggregate testing status covering multiple related conformance statements.

Conformance overview reports

Conformance overview reports are a complement to the conformance statement reports available when viewing a specific conformance statement, focusing rather on a set of related conformance statements. Such overview reports are available at different levels depending on how specifications are configured, specifically (per decreasing aggregation level):

  • The selected system’s overall status (always available).

  • A domain, when conformance statements can cover more than one domains.

  • A specification group, if groups are defined.

  • A specification, when multiple actors can be tested for.

A report of the selected system’s overall conformance status can be produced by clicking on the Download report button presented above the listing of statements. By default this will produce a PDF report, but clicking the presented caret for additional options allows you to also download the report in XML format.

../_images/admin_conformance_dashboard_organisation_view_overall_report_controls.png

The PDF report includes an overview section summarising the report’s context and status. This includes:

  • The organisation and system the report refers to.

  • The selected domain, specification group, or specification (skipped if the report refers to the overall status).

  • The report date and status.

  • The summary of conformance statement results, as counts and also percentage ratios.

Below the overview section, the report includes the listing of conformance statements presented in the same grouping as the on-screen display. Each statement includes its name and test result ratio and status.

../_images/conformance_overview_pdf.png

Following the summary of conformance statements, the report includes each individual statement report that lists its specific status, test suites and test cases. In fact the name of each presented conformance statement from the summary is a link you may click to access the page of the relevant statement report.

../_images/conformance_overview_pdf_statement.png

Note

Each conformance statement report included in the overview report, matches the report you can produce separately for the statement’s detail screen. When included in an overview report there are however no extended details for specific test cases.

An alternative to producing the report in PDF is to select the Download report as XML option. The format of this report is defined by the GITB Test Reporting Language (GITB TRL), and allows simpler machine-based processing. The following XML content is a sample of such a report:

<?xml version="1.0" encoding="UTF-8"?>
<ConformanceOverviewReport xmlns="http://www.gitb.com/tr/v1/"
                            xmlns:ns2="http://www.gitb.com/core/v1/"
                            xmlns:ns3="http://www.gitb.com/tbs/v1/">
   <metadata>
      <reportTime>2024-04-02T11:53:07.170+02:00</reportTime>
   </metadata>
   <overview>
      <definition>
         <party>
            <organisation>
               <name>ACME</name>
            </organisation>
            <system>
               <name>Test system</name>
               <version>v1.0</version>
               <description>A system for test purposes.</description>
            </system>
         </party>
      </definition>
      <summary>
         <status>FAILURE</status>
         <succeeded>1</succeeded>
         <failed>1</failed>
         <incomplete>1</incomplete>
      </summary>
      <statementOverview>
         <domain>
            <name>Docs domain</name>
            <description>A demo domain for documentation purposes.</description>
            <result>FAILURE</result>
            <specificationGroup>
               <name>EU Purchase Order Import Format</name>
               <description>Specification to define the data export format for EU Purchase Order Imports.</description>
               <result>SUCCESS</result>
               <specification>
                  <name>Version 2.00</name>
                  <description>Version 2.00 of the EU Purchase Order Import Format.</description>
                  <result>SUCCESS</result>
                  <actor statement="1">
                     <name>Actor</name>
                     <description>Order supplier.</description>
                     <result>SUCCESS</result>
                  </actor>
               </specification>
            </specificationGroup>
         </domain>
      </statementOverview>
      <statementDetails>
         <statement id="1">
            <definition>
               <party>
                  <organisation>
                     <name>ACME</name>
                  </organisation>
                  <system>
                     <name>Test system</name>
                     <version>v1.0</version>
                     <description>A system for test purposes.</description>
                  </system>
               </party>
               <domain>
                  <name>Docs domain</name>
                  <description>A demo domain for documentation purposes.</description>
               </domain>
               <specificationGroup>
                  <name>EU Purchase Order Import Format</name>
                  <description>Specification to define the data export format for EU Purchase Order Imports.</description>
               </specificationGroup>
               <specification>
                  <name>Version 2.00</name>
                  <description>Version 2.00 of the EU Purchase Order Import Format.</description>
               </specification>
               <actor>
                  <name>Actor</name>
                  <description>Order supplier.</description>
               </actor>
            </definition>
            <summary>
               <status>SUCCESS</status>
               <succeeded>1</succeeded>
               <failed>0</failed>
               <incomplete>0</incomplete>
            </summary>
            <lastUpdate>2023-09-28T15:52:16.000+02:00</lastUpdate>
            <testOverview>
               <testSuite>
                  <metadata>
                     <ns2:name>Simple test suite</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>A simple test suite.</ns2:description>
                  </metadata>
                  <result>SUCCESS</result>
                  <testCases>
                     <testCase>
                        <metadata>
                           <ns2:name>Test case 1</ns2:name>
                           <ns2:version>1.0</ns2:version>
                           <ns2:description>A simple test case.</ns2:description>
                        </metadata>
                        <result>SUCCESS</result>
                        <lastUpdate>2023-09-28T15:52:14.000+02:00</lastUpdate>
                     </testCase>
                     <testCase optional="true">
                        <metadata>
                           <ns2:name>Test case 2</ns2:name>
                           <ns2:version>1.0</ns2:version>
                           <ns2:description>Description for the second simple test case.</ns2:description>
                        </metadata>
                        <result>SUCCESS</result>
                        <lastUpdate>2023-09-28T15:52:16.000+02:00</lastUpdate>
                     </testCase>
                  </testCases>
               </testSuite>
            </testOverview>
         </statement>
      </statementDetails>
   </overview>
</ConformanceOverviewReport>

Note

You can also define custom formats for any kind of XML report. This is managed as part of the community’s report settings.

Producing a conformance overview report at the level of a specific domain, specification group or specification, is achieved using the report icon buttons presented at the right side of the statements’ display. The first button is used to produce the report in XML and the second one in PDF.

../_images/admin_conformance_dashboard_organisation_view_specific_report_controls.png

The reports in either case are structured in the same way as the overall reports discussed above. The only difference is that a report at a specific aggregation level (e.g. a specification group) will also list the relevant level as part of the report’s overview information.

../_images/conformance_overview_pdf_statement_specific.png

Conformance overview certificates

Depending on the community’s report settings you may also produce a conformance overview certificate for statements at different aggregation levels . This certificate is a report (in PDF format) that attests to the fact that the selected system has successfully completed testing for all relevant conformance statements.

To produce a certificate you click on the PDF report generation buttons, either at the overall level or for a specific domain, specification group or specification. If generating a certificate is enabled you will be prompted with a choice to select the kind of report to produce.

../_images/conformance_report_generation_options_popup.png

Selecting Conformance overview certificate will display an additional section to confirm, and if desired adapt, the certificate’s settings. The presented settings are pre-populated based on the values already configured for the community (see Edit report settings), and can all be adapted except for the signature settings.

../_images/conformance_report_generation_options_popup_overview_certificate.png

Once the settings are reviewed and adapted, clicking Generate report will produce the certificate.

../_images/conformance_overview_pdf_certificate.png

Note

The Conformance overview report option is the default if no certificate is enabled at the selected aggregation level. In such a case clicking a PDF report download button will skip the report type prompt and directly download the overview report.

Detailed view

You can switch to the detailed dashboard view at any time by toggling off the View per organisation control from the page’s header.

../_images/admin_conformance_dashboard_banner.png

Doing so presents the community’s conformance statements in a table, and enables advanced search filters allowing you to inspect specific statements of interest. The table listing statements is paged and includes one row per conformance statement.

../_images/admin_conformance_dashboard.PNG

The information displayed for each conformance statement is:

  • The organisation linked to the statement.

  • The system that is the focus of the testing activities.

  • The specification that the system is selected to conform to.

  • The actor of the specification the system is expected to act as.

  • The date and time when the conformance statement’s status was last updated.

  • The statement’s test results showing how many configured tests are successful, failed, or incomplete. This can also be hovered over to view a text summary of the displayed counts.

  • The statement’s overall status (success, failure or incomplete).

The presented results are by default sorted based on the organisation’s name, but clicking on each header label allows you to apply different sorting, based on the selected column, in either descending or ascending manner. The currently active sorting is indicated by an arrow next to the relevant header’s label.

The statement row can also be expanded by clicking it to view its details. These details include specific controls at the level of the statement, as well the listing of all its test suites and test cases with their latest test results.

Finally, each row also provides on the right side export controls that can be triggered clicking on the provided document icon. Clicking these produce the conformance statement report for the given statement in XML and PDF formats (see Conformance statement report).

Export all conformance statements

It is possible to generate a CSV export including all the conformance statements currently displayed. To do so click the Export CSV button from the conformance statements’ header.

../_images/admin_conformance_dashboard_header_detailed.png

Note that the CSV export will include the conformance statement information as well as the information on the individual related test cases. If you have defined custom properties for the community applicable to organisations or systems (see Edit custom member properties) these may also be included in CSV exports. To include a custom property:

  • It must be a Simple text value (i.e. not a hidden value or a file).

  • It must be configured as Included in exports.

All such properties are included in the export as columns following the “Organisation” or “System”, depending on whether they are organisation of system level properties. Their columns are named using a prefix of “Organisation” or “System” followed by the property’s key value included in parentheses.

../_images/admin_conformance_dashboard_export_csv.PNG

Apply search filters

When the dashboard’s detailed display is active, it offers also a set of filters that can be used to select the displayed conformance statements. These can be enabled by clicking the Search filters button from the table header.

../_images/admin_conformance_dashboard_header_detailed.png

Doing so will expand the table header to present the available filter controls.

../_images/admin_conformance_dashboard_filters_on.PNG

The controls that can be used for filtering are:

  • The relevant organisation and system.

  • The relevant domain (only in case your community is not linked to a specific domain).

  • The relevant specification group, specification and actor.

  • The conformance status.

  • The last update time for the conformance statement’s status.

  • Custom organisation and system properties defined for the community.

Most filter controls are defined as selection choices. Multiple selected values across these controls are applied as follows:

  • Within a specific filter control using “OR” logic (e.g. selecting multiple specifications).

  • Across filter controls using “AND” logic (e.g. selecting a specification and an organisation).

Note additionally that selecting dependent values serves to limit the filter options that are presented. For example if a given organisation is selected, the systems available for filtering will be limited to that organisation to already exclude impossible combinations.

Regarding organisation and system properties, these each present an Add button that, once clicked, will display a list of the available properties, a field or selection list to provide the filter value, and controls to confirm or cancel the filter. Multiple property filters can be added with the following semantics:

  • Values provided for the same property are applied using “OR” logic.

  • Values provided for different properties are applied using “AND” logic.

The presented conformance statements are automatically updated whenever your filter options are modified. The filter panel may also be collapsed and expanded by clicking again the Search filters button. In addition, you may click the caret to the right of the button to select from its options to either refresh the current results, or to clear all applied filters.

../_images/admin_conformance_dashboard_filters_options.png

View statement details

Viewing a statement’s details depends on the type of display you have active for the conformance dashboard. In case you are have enabled the view per organisation (the default) you click the desired statement from the presented statement presentation (statements are the entries displaying test result information):

../_images/admin_conformance_dashboard_statement_organisation_view.png

Alternatively, if you have the detailed dashboard view enabled, you click the desired statement’s row from the statements’ table.

../_images/admin_conformance_dashboard_statement_detailed_view.png

In both cases, the selected conformance statement will expand to present a nested table with its relevant test suites and test cases. Each test suite is presented in a separate panel that indicates the overall result for its contained test cases. These test suite panels can also be clicked to collapse or expand. The content of the panels is a table listing the test suite’s test cases, displaying for each the latest recorded test result. Each test case row includes the following:

  • The test case name and description, the latter visible by clicking the test case’s row.

  • The list of tags, if any are defined for the test case.

  • The time of the test case’s last run.

  • A view button to view the relevant session’s details. Clicking this will open up the test session in the session dashboard.

  • An info button to view the test case’s HTML documentation.

  • Two export buttons, to generate the test case report for the presented, latest test session in XML or PDF format.

  • The latest test result. Note that if the relevant test session resulted in a specific output message, the result icon can be clicked to display it.

../_images/admin_conformance_dashboard_expanded_output_message.PNG

As part of the display of the conformance statement’s details you are provided with additional controls, to navigate to relevant information and view conformance badges. If you have active the dashboard’s view per organisation, you will see the following controls:

../_images/admin_conformance_dashboard_expanded_controls_organisation_view.png

For the detailed dashboard view, the displayed controls differ slightly as follows:

../_images/admin_conformance_dashboard_expanded_controls.png

The presented controls permit the following actions:

Producing a conformance badge preview (if a badge is defined) results in a popup presenting the badge image.

../_images/conformance_statement_details_badge_preview.png

Note that the displayed badge is dynamically updated to always reflect the latest conformance testing status. For example if new test cases are added to the statement, accessing the same badge (displayed as a “success” badge above) will switch to an “incomplete” badge.

../_images/conformance_statement_details_badge_preview_incomplete.png

When the detailed dashboard view is enabled, the controls will be complemented with the overall test result percentages as a horizontal bar chart, showing the ratio of successes, failures and incomplete tests. If the statement includes optional test cases the bar chart includes a plus control that you can click to see separate percentages for mandatory and optional tests (by default mandatory tests are presented).

../_images/conformance_statement_details_ratios.png

Expanded tables can be collapsed by clicking again on the expanded conformance statement’s row. In addition, once one or more rows are expanded the conformance statement header also displays a Collapse all button to collapse all rows with a single click.

../_images/admin_conformance_dashboard_header_expanded.PNG

Export a test case report

Exporting a test case’s report is made possible through the file icon controls included in each test’s row. The two export buttons provided allow you to download the session’s test case report in XML and PDF format.

The following is an example of such a report in XML format, the XML content being defined by the GITB Test Reporting Language (GITB TRL):

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<TestCaseOverviewReport xmlns="http://www.gitb.com/tr/v1/" xmlns:ns2="http://www.gitb.com/core/v1/" xmlns:ns3="http://www.gitb.com/tbs/v1/" id="UBL_invoice_validation_test_3">
    <metadata>
        <ns2:name>TC3: Upload minimal invoice</ns2:name>
        <ns2:description>Test case to verify the correctness of a minimal UBL invoice. The invoice is provided manually through user upload.</ns2:description>
    </metadata>
    <startTime>2022-10-14T15:21:14.000+02:00</startTime>
    <endTime>2022-10-14T15:21:31.000+02:00</endTime>
    <result>FAILURE</result>
    <message>The provided invoice failed validation. Check the failed validation step(s) for further details.</message>
    <steps>
        <step id="1">
            <description>Step 1: UBL invoice upload</description>
            <report xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="TAR" id="1">
                <date>2022-10-14T15:21:25.767+02:00</date>
                <result>SUCCESS</result>
            </report>
        </step>
        <step id="2">
            <description>Step 2: Validate invoice against UBL 2.1 Invoice Schema</description>
            <report xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="TAR" name="XML Schema Validation" id="2">
                <date>2022-10-14T15:21:25.853+02:00</date>
                <result>SUCCESS</result>
                <counters>
                    <nrOfAssertions>0</nrOfAssertions>
                    <nrOfErrors>0</nrOfErrors>
                    <nrOfWarnings>0</nrOfWarnings>
                </counters>
                <reports/>
            </report>
        </step>
        <step id="3">
            <description>Step 3: Validate invoice against BII2 CORE restrictions for Invoice Transaction</description>
            <report xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="TAR" name="Schematron Validation" id="3">
                <date>2022-10-14T15:21:29.756+02:00</date>
                <result>SUCCESS</result>
                <counters>
                    <nrOfAssertions>0</nrOfAssertions>
                    <nrOfErrors>0</nrOfErrors>
                    <nrOfWarnings>1</nrOfWarnings>
                </counters>
                <reports>
                    <warning xsi:type="BAR">
                        <description>Attribute '@listID' marked as not used in the given context.</description>
                        <location>xml:12:0</location>
                    </warning>
                </reports>
            </report>
        </step>
        <step id="4">
            <description>Step 4: Validate invoice against BII RULES</description>
            <report xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="TAR" name="Schematron Validation" id="4">
                <date>2022-10-14T15:21:30.250+02:00</date>
                <result>FAILURE</result>
                <counters>
                    <nrOfAssertions>0</nrOfAssertions>
                    <nrOfErrors>2</nrOfErrors>
                    <nrOfWarnings>0</nrOfWarnings>
                </counters>
                <reports>
                    <error xsi:type="BAR">
                        <description>[BII2-T10-R051]-Sum of line amounts MUST equal the invoice line net amounts</description>
                        <location>xml:172:0</location>
                    </error>
                    <error xsi:type="BAR">
                        <description>[BII2-T10-R052]-An invoice total without VAT MUST equal the sum of line amounts plus the sum of charges on document level minus the sum of allowances on document level</description>
                        <location>xml:172:0</location>
                    </error>
                </reports>
            </report>
        </step>
    </steps>
</TestCaseOverviewReport>

The report includes the following information:

  • The identifier, name and description of the test case.

  • The start and end time.

  • The overall result as well as the output message that may have been produced.

  • The list of step reports that include each step’s identifier, description, timestamp, result and findings (if validations were carried out).

Selecting the second export option produces the report in PDF format which includes similar information to its XML counterpart with certain additional context data. The following sample report illustrates the information included:

../_images/test_case_report.png

The report contains a first Overview section that summarises the purpose and result of the test session. The information included here is:

  • The name of the system that was tested and the name of its related organisation.

  • The names of the domain, specification and actor of the relevant conformance statement.

  • The test case’s name and description.

  • The session’s result, start and end time.

The overview section is then followed by a section per test case step, each starting on a separate page.

../_images/test_case_report_step.png

The information displayed for each step is:

  • Its sequence number.

  • Its name.

  • Its result.

  • Its completion time.

  • For validation steps, the number of validation report findings classified as errors, warnings and messages.

  • For validation steps, a Details section listing the details of each validation finding.

Note

The XML report for a given test session can also be obtained through the test bed’s REST API (if enabled for your test bed instance).

Conformance statement report

The conformance statement report provides an overview of the conformance testing status relevant to a specific conformance statement. It can be generated to include only an overview or include also the results from its individual test cases. It is available in both XML and PDF formats.

Generating this report depends on the active dashboard view. If the view per organisation is active, you use the Download report and Download report as XML controls.

../_images/admin_conformance_dashboard_expanded_controls_organisation_view.png

In case the detailed dashboard view is active, you click the export icons on the right side of the statement’s row.

../_images/admin_conformance_dashboard_statement_detailed_view.png

Selecting to download a PDF report, you will be prompted for the type of report you want to generate:

  • The conformance statement report (the default), for the report including the status overview for the conformance statement.

  • The conformance statement report (with test case results), to also include the detailed test case results.

  • The conformance certificate (discussed in Conformance statement certificate).

When generating an XML report you will see a similar prompt including only the first two options (conformance certificates are not available in XML).

In both cases, once you select the report type and click the Generate report button, the produced report will be downloaded. Clicking on Cancel closes the popup to return you to the previous screen.

The following sample illustrates the information included in the conformance statement PDF report’s overview section. Specifically:

  • The information on the domain, specification and actor for the selected system.

  • The name of the system’s organisation and the system itself.

  • The date the report was produced, the number of successfully passed test cases versus the total, and the percentage of results (successes, failures and incomplete tests).

  • The list of test suites displaying per test suite its name, description and status.

  • For each test suite, the list of test cases, displaying similarly each test case’s name, description and result. The test case name is also prefixed with the test’s overall sequence that, in case test case steps are included, is a link to jump to its detailed report.

../_images/conformance_statement_report_sample.png

In case the option to add each test case’s step results is selected, the report includes a page per test case displaying its summary and the result of each test step. The test case’s title includes its reference number listed in the report’s overview section, and provides also a link to return to the listing of test cases.

../_images/conformance_statement_report_sample_test_case.png

Note

Detailed report size: The detailed conformance statement report presents each test session and individual step in a separate page. If the conformance statement contains numerous test cases, each with multiple test steps, the resulting detailed report could be quite long.

Producing the report in XML is an interesting alternative if you want to use it for machine-based processing or further customisations. The format of this report is defined by the GITB Test Reporting Language (GITB TRL). The following XML content is a sample of such a report:

<?xml version="1.0" encoding="UTF-8"?>
<ConformanceStatementReport xmlns="http://www.gitb.com/tr/v1/"
                             xmlns:ns2="http://www.gitb.com/core/v1/"
                             xmlns:ns3="http://www.gitb.com/tbs/v1/">
   <metadata>
      <reportTime>2024-04-02T14:13:18.267+02:00</reportTime>
   </metadata>
   <statement>
      <definition>
         <party>
            <organisation>
               <name>ACME</name>
            </organisation>
            <system>
               <name>Test system</name>
               <version>v1.0</version>
               <description>A system for test purposes.</description>
            </system>
         </party>
         <domain>
            <name>Docs domain</name>
            <description>A demo domain for documentation purposes.</description>
         </domain>
         <specificationGroup>
            <name>EU Purchase Order Export Format</name>
            <description>Specification to define the data export format for EU Purchase Order Exports.</description>
         </specificationGroup>
         <specification>
            <name>Version 1.02</name>
            <description>Version 1.02 of the EU Purchase Order Export Format.</description>
         </specification>
         <actor>
            <name>Actor</name>
            <description>Order supplier.</description>
         </actor>
      </definition>
      <summary>
         <status>FAILURE</status>
         <succeeded>2</succeeded>
         <failed>1</failed>
         <incomplete>0</incomplete>
      </summary>
      <lastUpdate>2023-09-29T19:00:01.000+02:00</lastUpdate>
      <testOverview>
         <testSuite>
            <metadata>
               <ns2:name>Simple TS</ns2:name>
               <ns2:version>0.1</ns2:version>
               <ns2:description>Simple test suite.</ns2:description>
            </metadata>
            <result>FAILURE</result>
            <testCases>
               <testCase>
                  <metadata>
                     <ns2:name>Simple TC</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>A simple test case.</ns2:description>
                     <ns2:tags>
                        <ns2:tag name="security" foreground="#ffffff" background="#d20000">Test cases linked to security issues.</ns2:tag>
                        <ns2:tag name="version 1.0" foreground="#ffffff" background="#000000">Test cases introduced in version 1.02.</ns2:tag>
                     </ns2:tags>
                  </metadata>
                  <result>SUCCESS</result>
                  <lastUpdate>2023-09-29T19:00:01.000+02:00</lastUpdate>
               </testCase>
               <testCase>
                  <metadata>
                     <ns2:name>Simple TC2</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>A second simple test case.</ns2:description>
                  </metadata>
                  <result>FAILURE</result>
                  <lastUpdate>2023-10-02T17:09:32.000+02:00</lastUpdate>
               </testCase>
            </testCases>
         </testSuite>
         <testSuite>
            <metadata>
               <ns2:name>Simple test suite</ns2:name>
               <ns2:version>1.0</ns2:version>
               <ns2:description>A simple test suite.</ns2:description>
            </metadata>
            <result>SUCCESS</result>
            <testCases>
               <testCase>
                  <metadata>
                     <ns2:name>Test case 1</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>A simple test case.</ns2:description>
                  </metadata>
                  <result>SUCCESS</result>
                  <lastUpdate>2023-09-28T15:52:14.000+02:00</lastUpdate>
               </testCase>
               <testCase optional="true">
                  <metadata>
                     <ns2:name>Test case 2</ns2:name>
                     <ns2:version>1.0</ns2:version>
                     <ns2:description>Description for the second simple test case.</ns2:description>
                  </metadata>
                  <result>SUCCESS</result>
                  <lastUpdate>2023-09-28T15:52:16.000+02:00</lastUpdate>
               </testCase>
            </testCases>
         </testSuite>
      </testOverview>
   </statement>
</ConformanceStatementReport>

Note

You can also define custom formats for any kind of XML report. This is managed as part of the community’s report settings.

Conformance statement certificate

The conformance certificate is a PDF report similar to the conformance statement report, that is meant to be delivered to the organisation linked to the conformance statement as a proof of its test results. It extends the base report by allowing you to selectively include its sections, include a custom text and also add a digital signature for integrity control and non-repudiation. These customisations are done for each generated certificate on the basis of defaults that are configured as part of the community’s report settings.

Generating a conformance certificate depends on the active dashboard view. If the view per organisation is active, you use the Download report control:

../_images/admin_conformance_dashboard_expanded_controls_organisation_view.png

In case the detailed dashboard view is active, you click the export icon on the far right side of the statement’s row:

../_images/admin_conformance_dashboard_statement_detailed_view.png

Selecting to download a PDF report, you will be prompted for the type of report you want to generate:

../_images/admin_conformance_dashboard_export_prompt.PNG

The options available are:

  • The conformance statement report (the default), for the report including the status overview for the conformance statement.

  • The conformance statement report (with test case results), to also include the detailed test case results.

  • The conformance certificate.

Selecting the conformance certificate option will display the customisation options for the certificate, starting from the values already configured for the community (see Edit report settings). You may override all settings, including the custom message that is presented here with the defined placeholders replaced using the information from the selected conformance statement. The only option that cannot be overridden at this point is the digital signature configuration.

../_images/admin_conformance_dashboard_export_prompt_cert.PNG

Once you have selected the report type and adapted your settings you can click the Generate report button to download the produced report. Clicking on Cancel closes the popup to return you to the previous screen.

The following example is a sample conformance certificate. It can significantly resemble the conformance statement report with its overview section drawing from the same information as a normal report. In this example however, a custom message for the recipient organisation is also included:

screenshots/conformance_statement_certificate_sample.PNG

Conformance snapshots

The statements listed in the conformance dashboard correspond to the current status of your community and testing setup. It is possible to take a readonly snapshot of this status that you can later on consult to review the conformance testing progress of your community at previous points in time. You may want to do this to record an overview at specific milestones of your project, or simply to track detailed testing progress over time. You could also find such snapshots useful to provide further versioning for your test configuration over what is normally possible, by defining snapshots as named and readonly version milestones. Regardless of their eventual purpose, these snapshots are referred to in the test bed as conformance snapshots.

You can review and select a given snapshot through the relevant control on the statement listings’ header. This is by default set to Latest conformance status indicating that you are viewing the current status.

../_images/admin_conformance_dashboard_banner.png

To create a new snapshot or manage your existing ones, expand the snapshot button and select option Manage conformance snapshots.

../_images/monitor_conformance_status__snapshots_controls_expanded.png

Doing so presents you with a popup listing your existing snapshots, presenting for each its label and timestamp, the latter used also to sort the entries to display the most recent snapshot first. As a first entry an entry is also included matching the latest conformance status. Above the listed snapshots you also have a simple search filter you can use to filter the displayed entries, as well as a control to toggle between internal and public labels for the snapshots.

../_images/monitor_conformance_status__snapshots_popup.png

Each listed snapshot presents controls to edit and delete it. Editing a snapshot allows you to replace the snapshot’s label, but also to specify whether it is visible to organisation users, and if so, whether it will presented with a different label. This information is also what is requested when you create a new snapshot by clicking the Create snapshot button from the popup’s footer.

../_images/monitor_conformance_status__snapshots_popup_edit.png

The default snapshot entry corresponding to the latest status is always present and visible to organisation users. However, you may still edit it to set a specific public label.

../_images/monitor_conformance_status__snapshots_popup_edit_latest.png

To view a specific snapshot you click its corresponding row. Doing so will close the popup and populate the conformance dashboard with the information from the snapshot. This is highlighted for you by displaying the snapshot’s label in the statement panel’s header.

../_images/monitor_conformance_status__snapshots_selected.png

With the snapshot selected, you may proceed to review its statements, use search filters and carry out all actions as you would on the current conformance status. Given that the snapshot’s data is fully readonly, it is possible that related information is changed or even deleted. The snapshot always displays information reflecting the state at the time of the snapshot. In case of currently deleted information (e.g. a deleted test suite or organisation), references are displayed with the only difference being that navigation controls to view the deleted data’s details will be unavailable. Finally, it is interesting to note that conformance badges are also part of the snapshot’s data so that if a badge is subsequently changed, the snapshot will still refer to the previous badge.

To have the conformance dashboard revert to the current status, select the Latest conformance status option from the snapshot button on the statement panel’s header.