Guide: Value proposition¶
|Track||Test bed setup|
About this guide¶
This guide presents the test bed’s value proposition, use cases and available services.
What you will achieve¶
At the end of this guide you should have a good understanding of what the test bed can do to support conformance testing. To achieve this you will be introduced to the test bed’s value proposition at an overall level but also from the perspective of project leaders and testers.
Having completed this guide you should also have a high-level understanding of the test bed’s main elements and approach to conformance testing. You will gain insight on when to use the test bed through sample use cases and be presented with the services that are available to you.
What you will need¶
- About 15 minutes for the value proposition and description of use cases. An additional 30 minutes would be needed to go through the details of all available services.
- A basic familiarity with IT concepts such as web services and hosting.
- A high-level understanding of the main phases involved in the realisation of IT projects.
How to complete this guide¶
You can approach this guide in two ways:
- You may read through its sections starting from the Value proposition. Each section logically builds upon the previous one, providing altogether a complete narrative.
- You may use individual sections as references. For example, the High-level overview acts as an executive summary of the test bed’s approach and key elements, that can be used as an initial introduction.
In the modern, digital and connected era, information is expected to flow seamlessly between the IT solutions that enable cross-border public services. At the level of the software components involved, such interoperability is achieved through specifications that describe both the information that is exchanged as well as the process to do so. Projects that define or use such specifications typically need to ensure that implementations correctly interpret them and realise them in way that the exchange and processing of information is consistent and unambiguous. This process is addressed by conformance testing and is where the ISA² Interoperability Test Bed comes into play:
The ISA² Interoperability Test Bed is an online, intuitive and self-service platform for conformance testing of IT systems against semantic and technical specifications.
Powered by the European Commission and specifically the Interoperability Test Bed Action of DIGIT ISA², the test bed is offered as a shared platform that can be used for free by any project developing interoperability solutions for the delivery of cross-border public services. Alternatively, the test bed is also offered as software that can be installed on-premise either as a separate instance or for experimentation. Regardless of the selected approach, the test bed’s users can receive support by the test bed’s team on each step of the process, from the initial evaluation of a community’s testing needs to their implementation and use.
A key aspect of the test bed is its flexibility to address varying user needs. It can be used to bring online simple data validation services or to implement complete conformance testing campaigns. It offers rich features to address a wide range of testing needs such as content validation, API simulation and verification of protocols with multi-step message exchanges. These are supported by built-in capabilities for messaging (e.g. AS2, AS4, SOAP, REST) and validation (e.g. XSD, Schematron, SHACL) alongside custom extensions to address domain-specific messaging, validation or processing needs.
Finally, the test bed is standards-based, building upon the results of the GITB project. This is a CEN standardisation initiative funded by the European Commission’s DG GROW to define the specifications and software needed to support a generic interoperability test bed service. The test bed reuses and extends the GITB software and ensures the evolution of the GITB specifications to match the needs of the interoperability testing community.
Value for community leaders¶
The community leader role is responsible for a project’s overall governance and is in a position to make decisions on how it is realised. From the perspective of community leaders the test bed offers:
- Conformance testing for the project’s specifications, ensuring that implementations test as early as possible based on defined scenarios.
- A customisable service that can be tailored to a community’s needs and extended if needed.
- A self-service approach for the community’s users, making use of the service when they need to and reducing support effort from project staff.
- Rich reporting and monitoring, allowing community administrators to follow all aspects of their users’ testing activities.
- Simplified operations by either reusing the shared ISA² test bed instance or, for an on-premise instance, by benefiting from a streamlined installation and update process.
- Comprehensive support and advice by the test bed team for the project’s team, complementing the rich online documentation.
Of special note are the cost reductions that the test bed makes possible. Specifically:
- The ISA² test bed instance is free to use for interoperability solutions and cross-border projects.
- The test bed team manages the operation the ISA² test bed instance and is responsible for the software’s maintenance, addressing both bug fixes and new features.
- By reusing the test bed a project can avoid the time and effort in the design, implementation and support of a project-specific conformance testing solution.
- The test bed team’s support provides valuable experience and best practices to reduce the effort in designing the conformance testing campaign that best suits each project.
For details on the specific services available to the test bed’s user communities, check section Available services.
Value for community members¶
Community members are a project’s participants that are expected to use the specifications it defines, typically through new software implementations. From the perspective of such users the test bed offers:
- An online testing service that is adapted to their needs.
- An approach to testing that is configuration-driven, without the need to make test-specific extensions to the software being tested.
- Rich reporting with test session data that is also visible to project staff in case help is needed.
- A self-service approach allowing users to test when they need to without having to coordinate with project staff.
The test bed is a complete platform consisting of both software and hardware components with the purpose of facilitating testing. The particular focus in this case is conformance and interoperability testing, ensuring that tested systems conform to a specification’s requirements and can interoperate consistently with conformant peer systems.
The following diagram illustrates the high-level operation of the test bed, in which the highlighted elements represent the system being tested and its tester:
Testing is organised in test suites, collections of one or more test cases, that define the scenarios to test for. Such test cases link to specifications and are assigned to systems under test once these are configured to claim conformance to them. The management of specifications and their test cases, organisations and their users, as well as the execution of tests and subsequent monitoring and reporting, take place through the test bed’s web user interface.
Test cases are authored in the GITB Test Description Language (TDL), an XML-based language that allows the definition of the steps to successfully complete and the involved actors, one of which is always realised by the system under test with others either being simulated or realised by other actual systems. Test steps can vary from validation and messaging to arbitrary processing, manipulating and evaluating the test session’s state, either per step or by checking the overall session’s context. A good example of the latter is ensuring conversational consistency by validating that a received message corresponds to an earlier request.
Each step’s operation takes place using the test bed’s built-in capabilities but, if needed can delegate to external components. This is where the test bed shines, in its ability to include custom and independent processing, messaging and validation extensions to address missing capabilities or domain-specific needs, by means of exchanges over a common web service API (the GITB service APIs). In using such extensions the test bed acts as an orchestrator of built-in and externally provided capabilities that make it flexible enough to accommodate most conformance testing needs.
Specifications are key interoperability enablers and their conformance testing allows for an unambiguous verification of their implementations. In this section we will not address the use cases of conformance testing itself, but rather focus on the test bed to highlight specific scenarios where it can bring added value.
Use case: Formal conformance testing¶
The test bed’s full set of capabilities is best leveraged when realising a complete conformance testing campaign. This would typically apply to projects that define a common set of specifications that drive the implementation of third-party systems, ensuring their interoperability once having entered into operation. An example would be a cross-border solution involving the exchange of messages where one would typically find specifications defining the messages’ content as well as the overall messaging protocol.
In cases of formal conformance testing, the specifications’ implementing parties are expected to successfully complete a specific set of test cases that are foreseen by the project’s administrators. Moreover, each party would need to provide proof of such testing before being granted a certification of conformance. This process may be put in place for one or more of the following reasons:
- To ensure the correctness and maturity of implementations before entry into operation.
- To clearly document for each implementation its level of conformance. This could be interesting when numerous types of messages are foreseen that needn’t all be supported, as well as cases where multiple specification versions can be in operation in parallel.
- To substantiate grant claims where eligibility to receive funding is tied to proof of conformance.
The test bed’s rich monitoring and reporting capabilities facilitate such usage scenarios. In addition the test bed typically allows better test coverage than testing against a specification’s reference implementation given its flexibility to address exceptional cases that may be difficult to otherwise trigger and negative tests that could even be impossible.
Use case: Testing tool for solution development¶
Apart from setting up a more formal conformance testing campaign, the test bed’s capabilities can be used to offer a community’s members with a set of sandbox utilities to facilitate development. These could range from simulated API endpoints to data validators that system implementers can use as and when they please without involving project staff. Publishing test services alongside a project’s specifications provides a hands-on way of working with the target specifications that can be used during all steps of development.
It is important to note that such testing tools can be made available early in the project’s lifecycle without requiring a first complete implementation of the target specifications. The test bed can be used to quickly make available validators and test cases based on simulation and its built-in capabilities.
Use case: Testing tool for specification development¶
Use of the test bed doesn’t need to start when a specification is ready to be published. In fact significant benefits can be achieved if the test bed is used as part of the development of the specification itself. This typically takes place by making available validation services for the target specification as a work-in-progress that is continuously updated, allowing the involved content experts to test the specification against their expectations.
Considering conformance testing from the beginning of a specification’s development offers it a powerful quality assurance tool. It ensures that the specification is testable through clear assertions and highlights ambiguous points early on. In addition and if applicable, it may help introduce valuable concepts of conformance profiles and levels, splitting the overall specification into well-defined sections that are easier to claim and test conformance for. Finally, it is worth noting that any validation services set up as part of a specification’s development will also be ready-to-use tools for its target community once implementations begin.
Use case: Public procurement¶
A public administration launching a request for a new IT solution needs to ensure clarity throughout the tendering process. Typical challenges for this are:
- The unambiguous definition of requirements to ensure that tenderers can understand fully what is expected and submit appropriate proposals.
- The evaluation of submissions to ensure their uniform and fair treatment, with deterministic verifications that can potentially be automated to a certain extent.
Referencing specifications as part of the requirements allows solution proposals to become more concrete. Furthermore, the definition of requirements greatly benefits when a common nomenclature is used and ideally when tenderer submissions are also expected to conform to a set of modelling guidelines. A good example of such an approach is use of the European Interoperability Reference Architecture (EIRA) and Solution Architecture Templates (SATs) as the means to express requirements that are expected to be addressed by tenderers through appropriate solution models.
In all cases where requirements and proposals are expressed in a machine-processable manner, the test bed can be used as a quality control instrument, providing validators through which both tenderers and the public administration can validate proposals in an unambiguous and deterministic manner.
Use case: Data quality assurance¶
Given the increasing openness of data and its aggregation through base registries and single access portals, the quality of data submissions needs to be evaluated as early as possible. The test bed can offer data validation services that can be used to measure the quality of data contributions before their aggregation and publication.
Such data validation services could be part of testing campaigns but also be available during production operations to validate data as it is received. This is made possible by the flexible deployment model of its validators that can be adapted based on the availability and capacity needs of the given scenario.
Use case: Training and communication¶
A use case not often considered for the test bed is its use as means of training and communication. Test cases normally focus on testing actual IT systems but can also be used for purely demonstration purposes, potentially also illustrating the operation of a solution that doesn’t yet exist. When launching a test session the user is presented with an easy to follow live diagram of exchanged messages between participating systems, along with validation and processing. This can be used as a live example of a to-be situation to support communication activities (e.g. a webinar), or for interested users to try out as a self-driven demo.
Foreseeing one or more scenarios as such visual demonstrations can increase awareness within a community allowing stakeholders to familiarise themselves early on with the envisioned result. In doing so it can also trigger increased feedback through its interactive nature and active user involvement.
The services that are available to realise a project’s testing needs are based on the conformance testing lifecycle. From a high-level perspective, the steps involved when using the test bed can be summarised as follows:
The first step in the process is the definition of testing requirements in an analysis phase, where the goals of the testing activity are defined as well as the means needed to achieve them. As part of this phase the roles and responsibilities of participating parties are also clarified. The analysis is then followed by the implementation phase where the identified requirements are realised by authoring test cases and developing needed test components. With the test cases in place, the next step is to pass into the operation phase where the test cases and services are made available through the test bed, the test bed itself is operated and used, and a help desk is activated to support its users.
The conformance testing lifecycle is continuous, repeating for each new testing requirement coming either from updated goals to cover new specification needs or from user requests.
The sections that follow present each offered service defining per case its context and description. The table below serves as an overview of all services linked to the lifecycle step they support:
|Analysis||Service: Support definition of envisioned services|
|Analysis||Service: Support definition of requirements and test design|
|Implementation||Service: Support test case authoring|
|Implementation||Service: Support test component development|
|Implementation||Service: Support project team|
|Operation||Service: Support on-premise test bed installation|
|Operation||Service: Support test bed operators|
|Operation||Service: Host test bed services|
|Operation||Service: Operate test bed|
|Operation||Service: Provide help desk services|
|Operation||Service: Support promotion in user community|
|Operation||Service: Train test bed users|
Service: Support definition of envisioned services¶
|Analysis > Define goals and roles|
The starting point for conformance testing is a technical or semantic specification. However, there is often a gap between what is defined in the specification and the means through which its related test services will be offered. Questions to consider include:
- Can test assertions be clearly determined from the specification’s requirements? Are there multiple possible scenarios needed?
- Do validation artefacts (e.g. Schematron rules) and components (e.g. simulators) exist or will they need to be developed? In the latter case, how are these best implemented?
- What are the channels through which the test service should be exposed to users?
- What are the management considerations for the target user community? Should access to the service be anonymous and open or should access control be applied?
The answers to such questions have a significant impact on the resulting services that users will access and also bear significantly on the involved effort to realise them.
The Test Bed team can leverage its experience to help you shape the envisaged service. The approach is to hold one or more workshops with your experts in which the expectations can be clarified. The main points addressed in these are:
- A presentation by you of your key needs and expectations.
- A presentation and possibly demo by the Test Bed team to show what could be achieved.
- An open discussion to answer key questions so that the envisioned service is shaped and the involved effort is estimated.
- A proof of concept implemented by the Test Bed team, based on your specific requirements, illustrating the end result and acting as a starting point for further work.
- Upon successful implementation of the proof of concept, a hands-on workshop with your technical experts to introduce the proposed design and present the elements to consider to continue the implementation.
Combining your goals and domain knowledge with the Test Bed team’s experience allows the clear definition of goals, constraints, and expectations. It is also a key step to help planning, effort estimation and the definition of roles and responsibilities for involved parties.
Service: Support definition of requirements and test design¶
|Analysis > Specify requirements|
Once the envisioned service is clearly defined, the next step is to analyse the specification(s) with the goal of producing clear sets of test assertions. This task involves frequent interaction with domain experts to produce an analysis that is formal enough to act as the starting point in implementing the required test cases and test components. Questions to ask include:
- What is the optimal organisation of test assertions into test suites and test cases?
- What test components (e.g. validators) are needed?
- Are there existing test cases or components that can be reused?
- How do the test bed’s concepts map to your project’s needs?
Answering these points requires analytical skills but also a good understanding of the existing interoperability testing landscape.
The Test Bed team can help at this point through its experience in analysing specifications to determine if they are “ready to implement”. The goal is to support your experts through this process by introducing best practices and points to consider that may not be immediately apparent (e.g. negative test scenarios). In addition, the Test Bed team can determine with you the best way to map its concepts to your testing needs such as the level of granularity of the specifications to test for and support for versioning.
Finally, the team’s experience from monitoring the interoperability testing landscape may already identify reuse opportunities such as existing test cases and services. In addition, existing approaches for conformance testing campaigns that were successfully applied to projects with similar needs or technical architectures can be introduced and evaluated.
Service: Support test component development¶
|Implementation > Develop test components|
All test cases include steps where domain-specific processing is required. Although the test bed software itself includes rich capabilities that can cover numerous cases, in many scenarios the best approach is to delegate to external components accessed via the web service APIs defined in the GITB specifications. The services that could be required include:
- Validation services to validate content or message flows.
- Messaging services to trigger sending and receiving of messages.
- Processing services to perform arbitrary operations required by the test case.
The split of processing between built-in capabilities of the test bed and such service extensions is an important design decision. In addition, implemented services need to follow established design practices and have well defined input and output to facilitate their use in test cases.
The Test Bed team can help you with the implementation of test components as web applications that expose the required GITB web service APIs. In doing so, coordination will be needed with your experts since the bulk of the implementation effort relates to the actual domain-specific processing that is needed rather than the GITB web service API that allows test bed integration. The goal is to provide your test developers with a robust starting point that addresses most GITB-related specifics so that you can focus on the business needs.
This activity results in one or more web applications that realise the defined validation, messaging, or processing needs. A first version of such components is typically implemented by the Test Bed team as part of an initial proof of concept that is subsequently delivered to you. This support complements the rich documentation on test service development and the available executable service templates.
Service: Support project team¶
|Implementation > Develop test components|
The first steps towards realising your conformance testing solution are the definition of its requirements and their initial implementation through test bed concepts, test suites and test service extensions. Following the initial setup, the process rarely completes as you may need to adapt to new testing requirements to match your project’s evolving needs. Moreover, it is often the case that once you familiarise yourself with the test bed’s capabilities you may envisage additional use cases that you did not initially foresee.
The Test Bed team is there to support you throughout your conformance testing process both as a technical expert as well as advisor on your test design. The support offered includes:
- Support to the project’s test developers to troubleshoot existing implementations or address new technical needs, new test suites or test service extensions.
- Support to the project’s management to discuss new requirements, feature requests and to provide strategic advice.
Such support may be offered through simple communication channels (email, phone) or by specific workshops (on-premise or remote). Note that the Test Bed team, based on its experience working with you, will also proactively update you on new developments that would be of interest to you.
Service: Support on-premise test bed installation¶
|Operation > Host services|
The proposed way to implement your interoperability or conformance testing needs is to use the shared ISA² interoperability test bed service. This way your effort will only be focused on the implementation of your domain-specific test cases and services.
An alternative option is to install the test bed software as a separate instance in your environment. You may want to do this if you require complete control over the testing service or if you need to test with data that is sensitive and that cannot be stored on the public cloud. In doing so your operations’ team may desire support from the Test Bed team to ensure a correct setup.
Installing a new instance of the test bed software is largely automated and requires minimal configuration, with steps that are thoroughly documented. If nonetheless your operations’ team would require additional help to setup or verify your environment, you can request assistance from the Test Bed team’s experts. Help can be provided:
- Remotely via email or phone to quickly verify simple points.
- In person, as needed, where the Test Bed team’s experts can visit your premises and meet with your team to verify and tune your setup in a hands-on manner.
Service: Support test bed operators¶
|Operation > Host services|
Once your test infrastructure is in place you will need to still manage your test cases and handle access control for your users. In addition, in case you are running your own test bed instance, you will also need to implement the needed processes to run and monitor your test bed, handle server maintenance tasks and apply updates for new test bed releases.
The Test Bed team can help you manage your environment’s operational aspects by training your operations’ team in their expected tasks. Training can be offered to address:
- The functional aspects of administering the test bed for your user community through the test bed’s user interface. This includes tasks such as test case management, user management, monitoring of test progress and reporting.
- The operational aspects of running your own test bed instance (if applicable) to explain how to start, stop and update the test bed’s components and also how to integrate them with the monitoring tools preferred by your team.
Service: Host test bed services¶
|Operation > Host services|
The core test engine and the developed domain-specific test components need to be hosted on an appropriate environment before they can be used. If you are using the ISA² test bed service then hosting the test engine is already handled for you. Regarding the supporting test components, these are separate applications for the hosting of which you have two options:
- Delegate hosting to the Test Bed team by hosting them alongside the test engine.
- Manage yourself the hosting in an environment that you procure and administer.
The ISA² test bed foresees hosting capacity on the public cloud to run not only the core test bed engine but also additional services required by its users. As part of your on-boarding session with the Test Bed team you may select to use the provided hosting resources or to host yourself the additional components you will implement.
Delegating hosting to the Test Bed team can remove constraints you could otherwise have on your budget, team or time planning if you needed to foresee your own hosting solution. Note however that in doing so there are certain points you would need to consider:
- If hosting requirements are significant they would need to be evaluated by the Test Bed team to ensure that the components in question are optimally configured.
- If capacity needs cannot be addressed with the currently available resources there may be a reasonable delay before these can be increased.
- In case you are implementing the components in question these will need to be packaged and delivered as per the Test Bed team’s requirements.
Service: Operate test bed¶
|Operation > Operate test bed|
Operation of the test bed involves management of your test cases and users, as well as the administration of the test engine itself to ensure its smooth operation. Administering test cases and users is an activity that is typically done by your team as this requires understanding of the domain in question and close communication with your users. This also allows flexibility on your part on the update of test cases and services that you can manage in full.
The Test Bed Team performs two activities while operating the ISA² test bed service:
- System administration of the test bed through its web user interface in support of your domain-specific administrator(s).
- Administration and monitoring of the overall hosting environment and the health of running software components.
Service: Provide help desk services¶
|Operation > Operate test bed|
A non-trivial test service requires the setup of a helpdesk to handle user support requests. This help desk is organised in several levels with each level focusing on requests of a different nature:
- The first level acts as the point of contact where requests are filtered. Those that can be addressed by providing information or using administrative features from the web interface are handled here, otherwise they are passed to the next level.
- The second level deals with issues on artefacts and services that you have developed. Requests that require further knowledge of test engine internals are further escalated to the third level.
- The third level is staffed with experts that can investigate the cause of defects and potentially propose technical solutions and workarounds. Issues at this level need an in-depth technical understanding of the test engine itself.
The first two service desk levels are best handled by your team as they need a good understanding of your business domain, your users and the services and test cases you have developed. You would however likely need support for issues relating to the test engine itself.
In simple cases were a small number of test cases is implemented and usage is limited, the Test Bed team can act as a complete help desk. In most cases however the role of first and second level help desk is best assigned to your support staff, with the Test Bed team acting as the third level that can investigate issues where in-depth knowledge of the test bed software is required.
In terms of follow-up, the Test Bed team can optionally be included in your ticket management system and make use of it to facilitate the tracking of issues it is involved in.
Service: Support promotion in user community¶
|Operation > Promote usage|
In certain cases the reason for setting up a conformance testing service is to allow members of your user community to independently carry out testing. This typically applies when your project defines one or more specifications that you expect software providers or your Member State counterparts to test their implementations against.
In this case apart from setting up your testing service using the test bed, you would need to also create awareness for it. A session to promote the test bed within your user community would stand to benefit from the presence of experts on the test bed’s functionality and its technical details.
The Test Bed team can make itself available to participate in activities promoting your testing services to your users. This participation can take shape as:
- Support in reviewing and possibly contributing to material you produce to ensure it is accurate with respect to the test bed.
- Support in user workshops, in-person or via webinar, in which the Test Bed team’s experts can either actively present the test bed or support you in test bed related discussions.
Service: Train test bed users¶
|Operation > Support users|
One of the main advantages of using the ISA² test bed service is that it offers a self-service environment to your users through which they can manage their own testing. This significantly decreases the effort your team needs to put into user support but does require that your users are able to use the provided web interface. You may in fact decide that it would be preferable to organise one or more training sessions, typically as webinars, where your users can be shown in a hands-on manner how they are expected to work.
The Test Bed team can further support your users by leading one or more training sessions in which the test bed is presented in the context of your specific case. Note that in doing so it is expected that your team undertakes all planning and preparation activities and also that there is participation from your team during the event(s) to address any questions that relate to your specific domain.
Congratulations! Having completed this guide you should have a good understanding of what the test bed can offer you. You should now be familiar with its value proposition, the use cases it can address as well as the various services it makes available. Finally, you should also have a high-level understanding of its approach to testing and its key components.
From this point you can consider going into more details on how you can use the test bed.
For further details on how to get started with implementing conformance testing scenarios in the test bed, several additional guides are available to address each step:
- Guide: Creating a test suite on how to create a simple test suite.
- Guide: Defining your test configuration on how to configure a test suite in the test bed as part of your overall test setup.
- Guide: Executing a test case on how to execute tests and monitor results.
- Guide: Installing the test bed on how to install your own test bed instance to experiment with.
Finally, for more information on the concepts presented in the High-level overview check out: