Monitor conformance status

The conformance status for your communities is the summary of the latest test results by all their member organisations for their currently active conformance statements. Monitoring this summary is possible by means of the Conformance Dashboard screen. To access this click on the ADMIN link from the screen’s header.

Doing so presents you with a left side menu containing links to administrative functions, of which you need to click the Conformance Dashboard link.

The screen is split in two sections:

• A set of search filters, initially disabled, to help you focus on specific organisations and specifications.

• The list of conformance statements defined for your communities’ organisations.

The currently defined conformance statements are presented in a table with one conformance statement per row. Recall that a conformance statement represents the link between an organisation’s system and a specification’s actor and defines what the organisation needs to test for (see Conformance statement).

The information displayed for each conformance statement is:

• The community of the organisation linked to the statement.

• The organisation linked to the statement.

• The system that is the focus of the testing activities.

• The domain of the specification.

• The specification that the system is selected to conform to.

• The actor of the specification the system is expected to act as.

• The statement’s test results in terms of successfully passed tests versus the total, including for the ones not passed their current status (“undefined” or “failed”).

• The statement’s overall status (success, failure or undefined).

Each row also provides on the right side an export function that can be triggered clicking on the provided document icon. Clicking this button produces the conformance statement report for the given statement (see Export conformance statement report and certificate).

View a statement’s test results

Each row in the conformance statements table can be expanded to present the test cases that relate to it. To do this click on the desired statement’s row.

Clicking on the row expands to present a nested table with the relevant test cases for this statement. Each row corresponds to a test case and presents its latest test result, executed by the organisation in question. Each row includes the following information:

• The relevant test suite.

• The test case.

• The latest test result. Note that if the relevant test session resulted in a specific output message, the result icon can be clicked to display it.

Each row also presents an export file icon that can be clicked to generate the test case report for the presented, latest test session (see Export a test case report).

Expanded tables can be collapsed by clicking again on the expanded conformance statement’s row. In addition, once one or more rows are expanded the conformance statement header also displays a Collapse all button to collapse all rows with a single click.

Note

Conformance dashboard vs session dashboard: A significant benefit of the conformance dashboard is that the focus is placed on the latest result and also that even non-executed test cases are displayed. This allows you to get a clear picture of an organisation’s testing progress without needing to extrapolate information.

Export a test case report

Exporting a test case’s report is made possible through the file icon control on the far right side of each test’s row.

Clicking this will generate and download the report (in PDF format).

The test case report contains a first Overview section that summarises the purpose and result of the test session. The information included here is:

• The name of the system that was tested and the name of its related organisation.

• The names of the domain, specification and actor of the relevant conformance statement.

• The test case’s name and description.

• The session’s result, start and end time.

The overview section is then followed by a section per test case step, each starting on a separate page.

The information displayed for each step is:

• Its sequence number.

• Its name.

• Its result.

• Its completion time.

• For validation steps, the number of validation report findings classified as errors, warnings and messages.

• For validation steps, a Details section listing the details of each validation finding.

Note

Step context values: The information included in the test case report for each step does not include the context information relevant to the step’s output results. This is omitted as the report would in most cases end up being very large.

Export conformance statement report and certificate

The conformance statement report provides an overview of the conformance testing status relevant to a specific conformance statement. It can be generated to include only an overview or include also the results from its individual test cases.

The conformance certificate is similar to the conformance statement report but is meant to be delivered to the organisation linked to the conformance statement as a proof of its test results. It extends the base report by allowing you to selectively include its sections, include a custom text and also add a digital signature for integrity control and non-repudiation. These customisations are done for each generated certificate on the basis of defaults that are configured as part of the community details’ management (see Edit conformance certificate settings).

To generate these reports for a given statement you start by clicking the export file icon on the right side of the statement’s row.

Once the button is clicked you will be prompted for the type of report you want to generate:

The options available are:

• The Conformance statement report (the default), for the report including the status overview for the conformance statement.

• The Conformance statement report (with test case results), to also include the detailed test case results.

• The Conformance certificate.

Selecting the Conformance certificate option will display the customisation options for the certificate, starting from the values already configured for the community (see Edit conformance certificate settings). You may override all settings, including the custom message that is presented here with the defined placeholders replaced using the information from the selected conformance statement. The only option that cannot be overridden at this point is the digital signature configuration.

Once you have selected the report type and adapted your settings you can click the Generate report button to download the produced report. Clicking on Cancel closes the popup to return you to the previous screen.

The following sample illustrates the information that is included in the conformance statement report’s overview section. Specifically:

• The information on the domain, specification and actor for the selected system.

• The name of the system’s organisation and the system itself.

• The date the report was produced and the number of successfully passed test cases versus the total.

• A table with the conformance statement’s test cases, displaying a row per test case with its reference number, the name of the the test suite and test case, the test case description and its test result.

In case the option to add each test case’s step results is selected, the report includes a section per test case displaying its summary and the results from each test step. The test case’s title includes its reference number listed in the report’s overview section.

Note

Detailed report size: The detailed conformance statement report presents each test session and individual step in a separate page. If the conformance statement contains numerous test cases, each with multiple test steps, the resulting detailed report could be quite long.

Finally, the following example provides a sample of a conformance certificate. It can significantly resemble the conformance statement report but in this case includes a custom message for the recipient organisation.

Export all conformance statements

It is possible to generate a CSV export including all the conformance statements currently displayed. To do so click the Export CSV button from the conformance statements’ header.

Doing so will generate a CSV file taking into account the currently applied filtering settings and include the conformance statement information as well as the information on the individual related test cases. Note that such exports can also include custom properties for communities applicable to organisations or systems (see Edit custom member properties) if these have been defined by you or community administrators. To include such custom properties:

• A single community must be selected from the filtering criteria (otherwise custom properties are skipped).

• It must be a Simple text value (i.e. not a hidden value or a file).

• It must be configured as Included in exports.

All such properties are included in the export as columns following the “Organisation” or “System”, depending on whether they are organisation of system level properties. Their columns are named using a prefix of “Organisation” or “System” followed by the property’s key value included in parentheses.

Note

Exporting custom properties from multiple communities: It is not possible to produce a single export for multiple communities including custom properties. The reason for this is that the resulting CSV file needs to have a single structure in terms of columns. The best workaround is to make individual exports per community selecting one at a time from the filtering criteria.

Apply search filters

The Conformance Dashboard offers a set of filters that can be used to limit the displayed conformance statements.

Filtering is by default switched off as indicated by the toggle button that is indicated as Disabled. Clicking this sets the button to Enabled, displaying the filter controls and switching on filtering.

The controls that can be used for filtering are:

• The relevant community, organisation and system.

• The relevant domain, specification and actor.

• Custom organisation and system properties defined for a given community.

Most filter controls are defined as selection choices. Multiple selected values across these controls are applied as follows:

• Within a specific filter control using “OR” logic (e.g. selecting multiple specifications).

• Across filter controls using “AND” logic (e.g. selecting a specification and an organisation).

Note additionally that selecting dependent values serves to limit the filter options that are presented. For example if a given organisation is selected, the systems available for filtering will be limited to that organisation to already exclude impossible combinations.

Regarding organisation and system properties, these can be selected once a specific community has been selected for filtering. Once enabled, each property type presents an Add button that, once clicked, will display a list of the available properties, a field or selection list to provide the filter value, and controls to confirm or cancel the filter. Multiple property filters can be added with the following semantics:

• Values provided for the same property are applied using “OR” logic.

• Values provided for different properties are applied using “AND” logic.

The presented conformance statements are automatically updated whenever your filter options are modified, or when the filters are removed altogether by clicking the Enabled toggle button. The filter panel may also be collapsed and expanded by clicking the panel’s title while maintaining the defined filters. The Refresh button is used to refresh the display of results based on the current filtering. Finally, note that applying no filtering is the default case when you first visit this screen.