Guide: Setting up CSV validation

Track

Test bed setup

This guide walks you through the points to consider when setting up CSV validation based on a Table Schema specification and the steps to bring your validation service online.

What you will achieve

At the end of this guide you will have understood what you need to consider when starting to implement validation services for your CSV-based specification. You will also have gone through the steps to bring it online and make it available to your users.

A CSV validation service can be created using multiple approaches depending on your needs. You can have an on-premise (or local to your workstation) service through Docker or use the test bed’s resources and, with minimal configuration, bring online a public service that is automatically kept up-to-date.

For the purpose of this guide you will be presented the options to consider and start with a Docker-based instance that could be replaced (or complemented) by a setup through the test bed. Interestingly, the configuration relevant to the validator is the same regardless of the approach you choose to follow.

What you will need

  • About 30 minutes.

  • A text editor.

  • A web browser.

  • Access to the Internet.

  • Docker installed on your machine (only if you want to run the validator as a Docker container).

  • A basic understanding of CSV and Table Schema. CSV itself is quite simple as a concept but if you want deeper and more formal information you can check its relevant RFC (RFC-4180). Table Schema is a popular means of specifying CSV content and is also simple to get into. The Table Schema specification itself offers a good introduction to its purpose and approach.

How to complete this guide

The steps described in this guide are for the most part hands-on, resulting in you creating a fully operational validation service. For these practical steps there are no prerequisites and the content for all files to be created are provided in each step. In addition, if you choose to try your setup as a Docker container you will also be issuing commands on a command line interface (all commands are provided and explained as you proceed).

Steps

You can complete this guide by following the steps described in this section. Not all steps are required, with certain ones being optional or complementary depending on your needs. The following diagram presents an overview of all steps highlighting the ones that apply in all cases (marked as mandatory):

../_images/step_overview.png

When and why you should skip or consider certain steps depends on your testing needs. Each step’s description covers the options you should consider and the next step(s) to follow depending on your choice.

Step 1: Determine your testing needs

Before proceeding to setup your validator you need to clearly determine your testing needs. A first outline of the approach to follow would be provided by answering the following questions:

  • Will the validator be available to your users as a tool to be used on an ad-hoc basis?

  • Do you plan on measuring the conformance of your community’s members to the CSV-based specification?

  • Is the validator expected to be used in a larger conformance testing context (e.g. during testing of a message exchange protocol)?

  • Should the validator be publicly accessible?

  • Should test data and validation reports be treated as confidential?

The first choice to make is on the type of solution that will be used to power your validation service:

  • Standalone validator: A service allowing validation of CSV content based on a predefined configuration of Table Schemas. The service supports fine-grained customisation and configuration of different validation types (e.g. specification versions) and supported communication channels. Importantly, use of the validator is anonymous and it is fully stateless in that none of the test data or validation reports are maintained once validation completes.

  • Complete test bed: The test bed is used to realise a full conformance testing campaign. It supports the definition of test scenarios as test cases, organised in test suites that are linked to specifications. Access is account-based allowing users to claim conformance to specifications and execute in a self-service manner their defined test cases. All results are recorded to allow detailed reporting, monitoring and eventually certification. Test cases can address CSV validation but are not limited to that, allowing validation of any complex exchange of information.

It is important to note that these two approaches are by no means exclusive. It is often the case that a standalone validator is defined as a first step that is subsequently used from within test cases in the test bed. The former solution offers a community tool to facilitate work towards compliance supporting ad-hoc data validation, whereas the latter allows for rigorous conformance testing to take place where proof of conformance is required. This could apply in cases where conformance is a qualification criterion before receiving funding or before being accepted as a partner in a distributed system. Finally, it is interesting to consider that non-trivial CSV validation may involve multiple validation artefacts (e.g. different schemas for different message types). In such a case, even if ad-hoc data validation is not needed, defining a separate validator simplifies management of the validation artefacts by consolidating them in a single location, as opposed to bundling them within test suites.

Regardless of the choice of solution, the next point to consider will be the type of access. If public access is important then the obvious choice is to allow access over the Internet. An alternative would be an installation that allows access only through a restricted network, be it an organisation’s internal network or a virtual private network accessible only by your community’s members. Finally, an extreme case would be access limited to individual workstations where each community member would be expected to run the service locally (albeit of course without the expectation to test message exchanges with remote parties).

If access to your validation services over the Internet is preferred or at least acceptable, the simplest case is to opt for using the shared ISA² test bed resources, both regarding the standalone validator and the test bed itself. If such access is not acceptable or is technically not possible (e.g. access to private resources is needed), the proposed approach would be to go for a Docker-based on-premise installation of all components.

Summarising the options laid out in this section, you will first want to choose:

  • Whether you will be needing a standalone validator, a complete test bed or both.

  • Whether the validator and/or test bed will be accessible over the Internet or not.

Your choices here can help you better navigate the remaining steps of this guide. Specifically:

Step 2: Prepare validation artefacts

As an example case for CSV validation we will consider a variation of the EU purchase order case first seen in Guide: Creating a test suite. In short, for the purposes of this guide you are considered to be leading an EU cross-border initiative to define a new common specification for the exchange of purchase orders between retailers.

To facilitate bulk imports of purchase orders your experts have created the following Table Schema:

{
    "fields": [
      {
        "name": "shippingAddressName",
        "type": "string",
        "description": "The shipping address recipient name",
        "constraints": {
            "required": true
        }
      },
      {
        "name": "shippingAddressStreet",
        "type": "string",
        "description": "The shipping address street",
        "constraints": {
            "required": true
        }
      },
      {
        "name": "shippingAddressCity",
        "type": "string",
        "description": "The shipping address city",
        "constraints": {
            "required": true
        }
      },
      {
        "name": "shippingAddressZip",
        "type": "string",
        "description": "The shipping address ZIP code",
        "constraints": {
            "required": true
        }
      },
      {
        "name": "billingAddressName",
        "type": "string",
        "description": "The billing address recipient name",
        "constraints": {
            "required": true
        }
      },
      {
        "name": "billingAddressStreet",
        "type": "string",
        "description": "The billing address street",
        "constraints": {
            "required": true
        }
      },
      {
        "name": "billingAddressCity",
        "type": "string",
        "description": "The billing address city",
        "constraints": {
            "required": true
        }
      },
      {
        "name": "billingAddressZip",
        "type": "string",
        "description": "The billing address ZIP code",
        "constraints": {
            "required": true
        }
      },
      {
        "name": "orderDate",
        "type": "date",
        "description": "The date the purchase order was created",
        "format": "%Y-%m-%d",
        "constraints": {
          "required": true
        }
      },
      {
        "name": "comment",
        "type": "string",
        "description": "An optional comment included in the purchase order",
        "constraints": {
            "maxLength": 500
        }
      },
      {
        "name": "totalItemQuantity",
        "type": "integer",
        "description": "The total quality of items included in the purchase order",
        "constraints": {
            "required": true,
            "minimum": 0
        }
      },
      {
        "name": "totalItemCostEUR",
        "type": "number",
        "description": "The total cost (in EUR) of the purchase order's items",
        "decimalChar": ".",
        "constraints": {
            "required": true,
            "minimum": 0.0
        }
      }
    ]
  }

Based on this, a sample CSV file containing purchase orders would be as follows:

shippingAddressName,shippingAddressStreet,shippingAddressCity,shippingAddressZip,billingAddressName,billingAddressStreet,billingAddressCity,billingAddressZip,orderDate,comment,totalItemQuantity,totalItemCostEUR
John Doe,Europa Avenue 123,Brussels,1000,Jane Doe,Europa Avenue 210,Brussels,1000,2020-01-22,Send in one package please,35,40.99
John Doe,Europa Avenue 123,Brussels,1000,Jane Doe,Europa Avenue 210,Brussels,1000,2020-01-23,,12,120.50
John Doe,Europa Avenue 123,Brussels,1000,Jane Doe,Europa Avenue 210,Brussels,1000,2020-01-24,,5,10.30

In addition to this basic specification, your business requirements also define the concept of large purchase orders which are those that include more than 10 items. The same requirements foresee that data imports should be possible for any kind of order or only refer to orders considered as large. To capture this additional requirement we will define a copy of the base Table Schema specification in which we will ensure that the included items for each order are always more than 10. The Table Schema to reflect this would be as follows (the minimum expectation is highlighted below):

{
  "fields": [
    {
      "name": "shippingAddressName",
      "type": "string",
      "description": "The shipping address recipient name",
      "constraints": {
          "required": true
      }
    },
    {
      "name": "shippingAddressStreet",
      "type": "string",
      "description": "The shipping address street",
      "constraints": {
          "required": true
      }
    },
    {
      "name": "shippingAddressCity",
      "type": "string",
      "description": "The shipping address city",
      "constraints": {
          "required": true
      }
    },
    {
      "name": "shippingAddressZip",
      "type": "string",
      "description": "The shipping address ZIP code",
      "constraints": {
          "required": true
      }
    },
    {
      "name": "billingAddressName",
      "type": "string",
      "description": "The billing address recipient name",
      "constraints": {
          "required": true
      }
    },
    {
      "name": "billingAddressStreet",
      "type": "string",
      "description": "The billing address street",
      "constraints": {
          "required": true
      }
    },
    {
      "name": "billingAddressCity",
      "type": "string",
      "description": "The billing address city",
      "constraints": {
          "required": true
      }
    },
    {
      "name": "billingAddressZip",
      "type": "string",
      "description": "The billing address ZIP code",
      "constraints": {
          "required": true
      }
    },
    {
      "name": "orderDate",
      "type": "date",
      "description": "The date the purchase order was created",
      "format": "%Y-%m-%d",
      "constraints": {
        "required": true
      }
    },
    {
      "name": "comment",
      "type": "string",
      "description": "An optional comment included in the purchase order",
      "constraints": {
          "maxLength": 500
      }
    },
    {
      "name": "totalItemQuantity",
      "type": "integer",
      "description": "The total quality of items included in the purchase order",
      "constraints": {
          "required": true,
          "minimum": 10
      }
    },
    {
      "name": "totalItemCostEUR",
      "type": "number",
      "description": "The total cost (in EUR) of the purchase order's items",
      "decimalChar": ".",
      "constraints": {
          "required": true,
          "minimum": 0.0
      }
    }
  ]
}

Given these requirements and validation artefacts we want to support two types of validation (or profiles):

  • basic: For CSV files including any kind of purchase orders. This is realised by validating against PurchaseOrder.schema.json.

  • large: For CSV files including only large purchase orders. This is realised by validating against PurchaseOrder-large.schema.json.

As the first configuration step for the validator we will prepare a folder with the required resources. For this purpose create a root folder named validator with the following subfolders and files:

validator
 |
 +-- resources
      |
      +-- order
           |
           +-- schemas
                |
                +-- PurchaseOrder.schema.json
                +-- PurchaseOrder-large.schema.json

Regarding the PurchaseOrder.schema.json and PurchaseOrder-large.schema.json files you can create them from the above content or download them (here: PurchaseOrder.schema.json and PurchaseOrder-large.schema.json). Finally, note that you are free to use any names for the files and folders; the ones used here will however be the ones considered in this guide’s subsequent steps.

Step 3: Prepare validator configuration

After having defined your testing needs and the validation artefacts for your specific case, the next step will be to configure the validator. The validator is defined by a core engine maintained by the test bed team and a layer of configuration, provided by you, that defines its use for a specific scenario. In terms of features the validator supports the following:

  • Validation channels including a SOAP web service API, a web user interface and a command-line tool.

  • Configuration of Table Schemas to drive the validation that can be local or remote.

  • Definition of different validation types as logically-related sets of validation artefacts.

  • Support per validation type allowing a user-provided schema.

  • Support per validation type of CSV syntax settings that can also be set by users at validation time.

  • Definition of separate validator configurations that are logically split but run as part of a single validator instance. Such configurations are termed “validation domains”.

  • Customisation of all texts presented to users.

Configuration is provided by means of key-value pairs in a property file. This file can be named as you want but needs to end with the .properties extension. In our case we will name this config.properties and place it within the order folder. Recall that the purpose of this folder is to store all resources relevant to purchase order validation. These are the validation artefacts themselves (PurchaseOrder.schema.json and PurchaseOrder-large.schema.json) and the configuration file (config.properties).

Define the content of the config.properties file as follows:

# The different types of validation to support. These values are reflected in other properties.
validator.type = basic, large
# Labels to describe the defined types.
validator.typeLabel.basic = Basic purchase orders
validator.typeLabel.large = Large purchase orders
# Validation artefacts (Table Schema) to consider for the "basic" type.
validator.schemaFile.basic = schemas/PurchaseOrder.schema.json
# Validation artefacts (Table Schema) to consider for the "large" type.
validator.schemaFile.large = schemas/PurchaseOrder-large.schema.json
# The title to display for the validator's user interface.
validator.uploadTitle = Purchase Order Validator

All validator properties share a validator. prefix. The validator.type property is key as it defines one or more types of validation that will be supported (multiple are provided as a comma-separated list of values). The values provided here are important not only because they define the available validation types but also because they drive most other configuration properties. Regarding the validation artefacts themselves, these are provided by means of the validator.schemaFile properties per validation type:

  • validator.schemaFile.TYPE defines one or more (comma-separated) file paths (relative to the configuration file) to lookup schema files.

Using these properties you define the validator’s validation artefacts as local files, where provided paths can be for a file or a folder. If a folder is referenced it will load all contained top-level files (i.e. ignoring subfolders).

Note

Further validation artefact configuration: You may also define validation artefacts as remote resource references and/or as being user-provided.

The purpose of the remaining properties is to customise the text descriptions presented to users:

  • validator.typeLabel defines a label to present to users on the validator’s user interface for the type in question.

  • validator.uploadTitle defines the title label to present to users on the validator’s user interface.

Once you have created the config.properties file, the validator folder should be as follows:

validator
 |
 +-- resources
      |
      +-- order
           |
           +-- config.properties
           +-- schemas
                |
                +-- PurchaseOrder.schema.json
                +-- PurchaseOrder-large.schema.json

As part of the validator’s configuration you may also define further aspects of your CSV syntax that are not addressed by the Table Schema specification. An example of such aspects are whether or not CSV content is expected to include a first header row, the character that is used as a field delimiter, and the character used as a quote (to wrap a field’s value). In our case we didn’t specify these as we have relied on the default settings, i.e. always expected headers, the , character as a field delimiter, and the " character as the quote. Such settings can be adapted per validation type using the validator.hasHeaders, validator.delimiter and validator.quote properties. For example, if we wanted to use the pipe character (|) to separate fields instead of the default comma (,) and never expect a header row we could adapt the configuration as follows:

validator.type = basic, large
...
validator.hasHeaders.basic = false
validator.hasHeaders.large = false
validator.delimiter.basic = |
validator.delimiter.large = |

Furthermore, if you want to offer flexibility to users in defining such syntax settings you can also allow them to specify them as part of their provided input. This is achieved through the validator.input.hasHeader, validator.input.delimiter and validator.input.quote properties that similarly per validation type define whether users may choose their own settings. In each case the option you can specify here is to prevent such input (none, the default), allow it as optional (optional) or require it as mandatory (required). Returning to our earlier example we can further extend the configuration by allowing users to define themselves whether their content includes a header row. This is achieved as follows:

validator.type = basic, large
...
validator.input.hasHeaders.basic = optional
validator.input.hasHeaders.large = optional

If no user input on such settings is requested or provided, the defaults you have specified will apply.

The limited configuration file we have prepared assumes numerous default configuration properties. An important example is that by default, the validator will expose a web user interface and a SOAP web service API. This configuration is driven through the validator.channels property that by default is set to form, soap_api (for a user form and SOAP web service respectively). All configuration properties provided in config.properties relate to the specific domain in question, notably purchase orders, reflected in the validator’s resources as the order folder. Although rarely needed, you may define additional validation domains each with its own set of validation artefacts and configuration file (see Configuring additional validation domains for details on this). Finally, if you are planning to host your own validator instance you can also define configuration at the level of the complete validator (see Additional configuration options regarding application-level configuration options).

For the complete reference of all available configuration properties and their default values refer to section Validator configuration properties.

Remote validation artefacts

Defining the validator’s schemas as local files is not the only option. If these are available online you can also reference them remotely by means of property validator.schemaFile.TYPE.remote.N.url. The N element in the properties’ names is a zero-based positive integer allowing you to define more than one entries to match the number of remote files.

The example that follows illustrates the loading of a schema for a validation type named v2.2.1 from a remote location:

validator.type = v2.2.1
...
validator.schemaFile.v2.2.1.remote.0.url = https://my.server.com/my_schema_1.json

Although rare you may also combine local and remote schema files by defining a validator.schemaFile.TYPE property and one or more validator.schemaFile.TYPE.remote.N.url properties. In all cases, the schemas from all sources will be used for the validation.

Note

Remote schema caching: Caching is used to avoid constant lookups of remote schema files. Once loaded, remote schemas will be automatically refreshed every hour.

User-provided validation artefacts

Another available option on schema file configuration is to allow a given validation type to support a user-provided schema. Such a schema would be considered in addition to any pre-configured local and remote schemas. Enabling user-provided schemas is achieved through the validator.externalSchema property:

...
validator.externalSchema.TYPE = required

These properties allow three possible values:

  • required: The relevant schema(s) must be provided by the user.

  • optional: Providing the relevant schema(s) is allowed but not mandatory.

  • none (the default value): No such schema(s) are requested or considered.

Specifying that for a given validation type you allow users to provide a schema will result in it being combined with your pre-defined one (if present). This could be useful in scenarios where you want to define a common validation base but allow also ad-hoc extensions for e.g. restrictions defined at user-level (e.g. National validation rules to consider in addition to a common set of EU rules).

Note

Generic validator: It is possible to not predefine any schema resulting in a validator that is truly generic, expecting it to be provided by users. Such a generic instance actually exists at https://www.itb.ec.europa.eu/csv/any/upload.

Step 4: Setup validator as Docker container

Note

When to setup a Docker container: The purpose of setting up your validator as a Docker container is to host it yourself or run it locally on workstations. If you prefer or don’t mind the validator being accessible over the Internet it is simpler to delegate hosting to the test bed team by reusing the test bed’s infrastructure. If this is the case skip this section and go directly to Step 5: Setup validator on test bed. Note however that even if you opt for a validator managed by the test bed, it may still be interesting to create a Docker image for development purposes (e.g. to test new validation artefact versions) or to make available to your users as a complementary service (i.e. use online or download and run locally).

Once the validator’s configuration is ready (configuration file and validation artefacts) you can proceed to create a Docker image.

The configuration for your image is driven by means of a Dockerfile. Create this file in the validator folder with the following contents:

FROM isaitb/csv-validator:latest
COPY resources /validator/resources/
ENV validator.resourceRoot /validator/resources/

This Dockerfile represents the most simple Docker configuration you can provide for the validator. Let’s analyse each line:

FROM isaitb/csv-validator:latest

This tells Docker that your image will be built over the latest version of the test bed’s isaitb/csv-validator image. This represents the validator’s core that expects configuration to drive the validation. It is available on the public Docker Hub and as such can be directly used through any Docker installation with Internet access.

COPY resources /validator/resources/

This copies your resources folder to the image under path /validator/resources/.

ENV validator.resourceRoot /validator/resources/

This instructs the validator that it should consider as the root of all its configuration resources the /validator/resources/ folder (which was just copied into it).

The contents of the validator folder should now be as follows:

validator
 |
 +-- Dockerfile
 +-- resources
      |
      +-- order
           |
           +-- config.properties
           +-- shapes
                |
                +-- PurchaseOrder.schema.json
                +-- PurchaseOrder-large.schema.json

That’s it. To build the Docker image open a command prompt to the validator folder and issue:

docker build -t po-validator .

This command will create a new local Docker image named po-validator based on the Dockerfile it finds in the current directory. It will proceed to download missing images (e.g. the isaitb/csv-validator:latest image) and eventually print the following output:

Sending build context to Docker daemon  32.77kB
Step 1/3 : FROM isaitb/csv-validator:latest
---> 39ccf8d64a50
Step 2/3 : COPY resources /validator/resources/
---> 66b718872b8e
Step 3/3 : ENV validator.resourceRoot /validator/resources/
---> Running in d80d38531e11
Removing intermediate container d80d38531e11
---> 175eebf4f59c
Successfully built 175eebf4f59c
Successfully tagged po-validator:latest

The new po-validator:latest image can now be pushed to a local Docker registry or to the Docker Hub. In our case we will proceed directly to run this as follows:

docker run -d --name po-validator -p 8080:8080 po-validator:latest

This command will create a new container named po-validator based on the po-validator:latest image you just built. It is set to run in the background (-d) and expose its internal listen port through the Docker machine (-p 8080:8080). Note that by default the listen port of the container (which you can map to any available host port) is 8080.

Your validator is now online and ready to validate CSV content. If you want to try it out immediately skip to Step 6: Use the validator. Otherwise, read on to see additional configuration options for the image.

Running without a custom Docker image

The discussed approach involved building a custom Docker image for your validator. Doing so allows you to run the validator yourself but also potentially push it to a public registry such as the Docker Hub. This would then allow anyone else to pull it and run a self-contained copy of your validator.

If such a use case is not important for you, or if you want to only use Docker for your local artefact development, you could also skip creating a custom image and use the base isaitb/csv-validator image directly. To do so you would need to:

  • Define a named or unnamed Docker volume pointing to your configuration files.

  • Run your container by configuring it with the volume.

Considering the same file structure of the /validator folder you can launch your validator using an unnamed volume as follows:

docker run -d --name po-validator -p 8080:8080 \
           -v /validator/resources:/validator/resources/ \
           -e validator.resourceRoot=/validator/resources/  \
           isaitb/csv-validator

As you see here we create the validator directly from the base image and pass it as a volume our resource root folder. When doing so you need to also make sure that the validator.resourceRoot environment variable is set to the path within the container.

Using this approach to run your validator has the drawback of being unable to share it as-is with other users. The benefit however is one of simplicity given that there is no need to build intermediate images. As such, updating the validator for configuration changes means that you only need to restart your container.

Configuring additional validation domains

Up to this point you have configured validation for purchase orders which defines one or more validation types (basic and large). This configuration can be extended by providing additional types to reflect:

  • Additional profiles with different business rules (e.g. minimal).

  • Specification versions (e.g. basic_v1.0, large_v1.0, basic_v1.1_beta).

  • Other types of content that are linked to purchase orders (e.g. purchase_order_basic_v1.0 and order_receipt_v1.0).

All such extensions would involve defining potentially additional validation artefacts and updating the config.properties file accordingly.

Apart from extending the validation possibilities linked to purchase orders you may want to configure a completely separate validator to address an unrelated specification that would most likely not be aimed to the same user community. To do so you have two options:

  • Repeat the previous steps to define a separate configuration and a separate Docker image. In this case you would be running two separate containers that are fully independent.

  • Reuse your existing validator instance to define a new validation domain. The result will be two validation services that are logically separate but are running as part of a single validator instance.

The rationale behind the second option is simply one of required resources. If you are part of an organisation that needs to support validation for dozens of different types of CSV content that are unrelated, it would probably be preferable to have a single application to host rather than one per specification.

Note

Sharing artefact files accross domains: Setting the application property validator.restrictResourcesToDomain to false allows to add paths of validation artefacts that are outside of the domain root folder. This enables sharing artefacts between different domains.

In your current single domain setup, the purchase order configuration is reflected through folder order. The name of this folder is also by default assumed to match the name of the domain. A new domain could be named invoice that is linked to CSV invoices. This is represented by an invoice folder next to order that contains similarly its validation artefacts and domain-level configuration property file. Considering this new domain, the contents of the validator folder would be as follows:

validator
 |
 +-- Dockerfile
 +-- resources
      |
      +-- invoice
        (Further contents skipped)
      +-- order
        (Further contents skipped)

If you were now to rebuild the validator’s Docker image this would setup two logically-separate validation domains (invoice and order).

Note

Validation domains vs types: In almost all scenarios you should be able to address your validation needs by having a single validation domain with multiple validation types. Validation types under the same domain will all be presented as options for users. Splitting in domains would make sense if you don’t want the users of one domain to see the supported validation types of other domains.

Important: Support for such configuration is only possible if you are defining your own validator as a Docker image. If you plan to use the test bed’s shared validator instance (see Step 5: Setup validator on test bed), your configuration needs to be limited to a single domain. Note of course that if you need additional domains you can in this case simply repeat the configuration process multiple times.

Additional configuration options

We have seen up to now that configuring how validation takes place is achieved through domain-level configuration properties provided in the domain configuration file (file config.properties in our example). When setting up the validator as a Docker image you may also make use of application-level configuration properties to adapt the overall validator’s operation. Such configuration properties are provided as environment variables through ENV directives in the Dockerfile.

We already saw this when defining the validator.resourceRoot property that is the only mandatory property for which no default exists. Other such properties that you may choose to override are:

  • validator.domain: A comma-separated list of names that are to be loaded as the validator’s domains. By default the validator scans the provided validator.resourceRoot folder and selects as domains all subfolders that contain a configuration property file (folder order in our case). You may want to configure the list of folder names to consider if you want to ensure that other folders get ignored.

  • validator.domainName.DOMAIN: A mapping for a domain (replacing the DOMAIN placeholder) that defines the name that should be presented to users. This would be useful if the folder name itself (order in our example) is not appropriate (e.g. if the folder was named files).

The following example Dockerfile illustrates use of these properties. The values set correspond to the applied defaults so the resulting Docker images from this Dockerfile and the original one (see Step 4: Setup validator as Docker container) are in fact identical:

FROM isaitb/csv-validator:latest
COPY resources /validator/resources/
ENV validator.resourceRoot /validator/resources/
ENV validator.domain order
ENV validator.domainName.order order

See Application-level configuration for the full list of supported application-level properties.

Finally, it may be the case that you need to adapt further configuration properties that relate to how the validator’s application is ran. The validator is built as a Spring Boot application which means that you can override all configuration properties by means of environment variables. This is rarely needed as you can achieve most important configuration through the way you run the Docker container (e.g. defining port mappings). Nonetheless the following adapted Dockerfile shows how you could ensure the validator’s application starts up on another port (9090) and uses a specific context path (/ctx).

FROM isaitb/csv-validator:latest
COPY resources /validator/resources/
ENV validator.resourceRoot /validator/resources/
ENV server.servlet.context-path /ctx
ENV server.port 9090

Note

Custom port: Even if you define the server.port property to a different value other than the default 8080 this remains internal to the Docker container. The port through which you access the validator will be the one you map on your host through the -p flag of the docker run command.

The full list of such application configuration properties, as well as their default values, are listed in the Spring Boot configuration property documentation.

Environment-specific domain configuration

In the previous section you saw how could can configure the validator’s application by means of environment properties. Use of environment properties is also possible for specific validator domains, allowing you to externalise and override their configuration aspects. Typical cases where you may want to do this are:

  • To adapt the configuration for different instances of the same validator.

  • To hide sensitive properties such as internal URLs or passwords.

External configuration properties can be provided through environment variables or system properties (the latter being interesting when using a command line validator). These can contribute to the configuration:

  • Complete properties, by defining a variable or property with the same name as a configuration property.

  • Values, by referring to arbitrary variables or properties within a domain property file. This is done by defining a placeholder ${} and using within it the prefixes env: for environment variables or sys: for system properties (e.g. ${env:MY_VARIABLE}).

As a simple example of this, consider the definition of the title of the validator’s web UI. We want to adapt this title depending on the purpose of the specific validator instance. To begin with we can define the title via the validator.uploadTitle property in the config.properties file as follows:

validator.uploadTitle = Purchase Order Validator

The problem with this approach is that the upload title remains fixed across all instances of the validator. Alternatively we can define the value for the title as an environment variable named VALIDATOR_TITLE. To use this we can adapt config.properties to reference it as follows:

validator.uploadTitle = ${env:VALIDATOR_TITLE}

We can then adapt this for each Docker container we create by defining a specific value for the title:

docker run -e VALIDATOR_TITLE="Purchase Order Validator (Demo)" ...

A further alternative would be to externalise the complete validator.uploadTitle property by removing it from config.properties and specifying it as an environment variable:

docker run -e validator.uploadTitle="Purchase Order Validator (Demo)" ...

Note that such external configuration can also be used as a partial value. If we define an environment variable named VALIDATOR_ENV we could also use it within config.properties as follows:

validator.uploadTitle = Purchase Order Validator (${env:VALIDATOR_ENV})

Finally, you can use environment or system variables to override properties that are already defined in config.properties. This is useful if you use the file’s properties as default values and selectively override certain of them as needed. A property’s value is looked up in sequence as follows:

  1. Look in environment variables.

  2. If not found look in system properties.

  3. If not found look in the domain configuration file.

  4. If not found consider the property’s overall default value (see Domain-level configuration).

Step 5: Setup validator on test bed

Note

When to setup on test bed resources: Setting up your validator on the test bed’s resources removes hosting concerns and allows you to benefit from automatic service reloads for configuration changes. In doing so however you need to keep in mind that the validator will be exposed over the Internet. If this approach is not suitable for you (e.g. you want to expose the validator within a restricted network) you should consider setting up the validator as a Docker container (see Step 4: Setup validator as Docker container) that you can then host as you see fit.

To configure a validator using the test bed’s resources all you need to do is get in touch with the test bed team and provide the validator’s configuration. Specifically:

  1. Send an email to DIGIT-ITB@ec.europa.eu describing your case: This step is needed for two reasons. Firstly you may want to have a further discussion and potentially a demo to better understand the available options. Secondly the test bed’s team would need to ensure that you qualify to use its resources (to e.g. avoid that you are a private company planning to offer commercial validation services).

  2. Share the configuration for the validator: Once contact has been established you need to provide the initial configuration for the validator.

Regarding the second step, the validator’s configuration to be shared is the contents of the validator folder as described in Step 3: Prepare validator configuration. The eventual goal here will be to have the configuration available through an accessible Git repository. This can be done in a number of ways listed below in decreasing order of preference:

  • Create a new Git repository: You can push all resources (the validator folder) to a new Git repository (e.g. on GitHub or the European Commission’s CITNet Bitbucket server). You can of course add any other resources to this repository as you see fit (e.g. a README file). Once done provide the repository’s URL to the test bed team.

  • Provide the resources to the test bed team: You can send the configuration files themselves to the test bed’s team (e.g. make an archive of the validator folder). Ideally you should define the configuration file but if in doubt you can simply describe the resources and the test bed team will prepare the initial configuration for you. When following this approach a new Git repository will be created for you on the European Commission’s CITNet Bitbucket server or on GitHub for which you will be assigned write access (assuming you have a relevant user account).

  • Update an existing Git repository: If you already have a Git repository to maintain the validation artefacts you can reuse this by adding to it the required configuration file (config.properties in our case). When ready you will need to provide the test bed team with the URL to the repository and the location of the configuration file.

Following the initial configuration, the resulting Git repository will be monitored to detect any changes to the validation artefacts or the configuration file. If such a change is detected, the validation service will be automatically updated within a few minutes.

Note

Using a dedicated Git repository for the validator: Whether you define a new Git repository yourself or the test bed team creates one for you, the result is a repository that is dedicated to the validator. This approach is preferable to reusing an existing Git repository to avoid unwanted changes to the validator. whether or not this is done through GitHub, CITNet’s Bitbucket or another service depends on what best suits your needs.

As part of the initial setup for the validator the test bed team will also configure how it is accessed. The name used will match the name of the folder that contains your configuration file (order in the considered example), but this can differ according to your preferences. If this is the case make sure to inform the test bed team of your preferred naming.

Considering our example, for a name of order, the resulting root URL through which the validator will be accessed is https://www.itb.ec.europa.eu/csv/order. The specific paths will depend on the supported validation channels as described in Step 6: Use the validator.

Step 6: Use the validator

Well done! At this step your validator has been successfully configured and is ready to use. Depending on which approach was followed, this may have been done either:

The validation channels that are supported depend on the configuration you have supplied. This is done through the validator.channels property of your configuration file (config.properties) that defaults to form, soap_api. The supported channels are as follows:

  • form: A web user interface allowing a user to provide the CSV content to validate.

  • soap_api: A SOAP API allowing contract-based machine-to-machine integration using SOAP web service calls.

The following sub-sections describe how each channel can be used considering the example EU purchase order specification.

Validation via user interface

The validator’s user interface is available at the /csv/DOMAIN/upload path. The exact path depends on how this is deployed:

The first page that you see is a simple form to provide the CSV content to validate.

../_images/validator_upload.png

This form expects the following input:

  • Content to validate: The CSV content that will be submitted for validation. The preceding dropdown selection determines how this will be provided, specifically as a file input (pre-selected), as a URI to be loaded remotely or as content to be provided using an editor.

  • Validate as: The type of validation to apply.

Note that all displayed labels can be adapted through the config.properties configuration file (see Properties related to UI labels). The available validation types match the ones defined in the validator.type property, displayed using the validator.typeLabel.TYPE labels. Moreover, the text title could be replaced by a configurable HTML banner, and further complemented with a HTML footer (see Domain-level configuration).

../_images/validator_upload_selected.png

It is worth noting also that if your configuration defined only a single validation type, the user interface would be simplified by presenting only the content input controls (i.e. considering the single validation type as pre-selected).

../_images/validator_upload_simple.png

In addition, if your configuration for the selected validation type allows for a user-provided schema, the form also includes the controls to manage this. The user-provided schema can be defined via file upload or remote URI and can be mandatory or optional depending on your configuration.

../_images/validator_upload_external.png

Finally, if you also specified in your configuration user inputs for syntax settings such as the presence of headers, delimiters and quotes, these will also be presented for input (either as optional or mandatory inputs).

../_images/validator_upload_syntax_options.png

Once you have provided your input click the Validate button to trigger the validation. Upon completion you will be presented with the validation results:

../_images/validator_result.png

This screen includes an overview of the result listing:

  • The validation timestamp (in UTC), the name of the validated file and the applied validation type (if more than one are configured).

  • The overall result (SUCCESS or FAILURE).

  • The number of errors, warnings and information messages.

This section is followed by the Details panel, where the details of each report item are listed:

  • It’s type (whether this is an error, warning or information message).

  • It’s description.

Clicking on each item’s details will open a popup that shows within the provided content the specific point that triggered the issue:

../_images/validator_result_errordetail.png

In terms or reporting, apart from the on-screen display, buttons are available allowing you to download the validation report:

  • View annotated input: Open a popup with the provided content, including annotations for the lines with relevant report items.

  • Download XML report: Download as XML (in GITB TRL syntax - sample here).

  • Download PDF report: Download as PDF (sample here).

Note that the download buttons are initially disabled but are enabled as soon as the respective reports become available.

Finally, to trigger a new validation you may either use the form from the top of the result screen or click on the form’s title that will take you back to the previous page.

Validation via minimal user interface

If you are exposing a web user interface (see Validation via user interface) for your validator you also have the option of enabling an alternative minimal interface that could be used as an embedded component in another web page (e.g. via an iframe). This is enabled through the validator.supportMinimalUserInterface property in your domain configuration (file config.properties).

...
validator.supportMinimalUserInterface = true

The result of this is to expose a /uploadm path. The path depends on how this is deployed:

The minimal interface offers largely the same functionality as the complete one but with a more condensed layout and minimal styling. The initial input page you see for the validator is as follows:

../_images/validator_upload_minimal.png

The most significant difference is the result page which provides only an overview and the relevant download controls:

../_images/validator_result_minimal.png

The meaning of the provided controls and displayed information for both input and result pages is identical to the complete user interface (see Validation via user interface).

Validation via SOAP web service API

The validator’s SOAP API is available under the /csv/soap path. The exact path depends on how this is deployed (path to WSDL provided):

The SOAP API used is the GITB validation service API, meaning that the validator is a GITB-compliant validation service. The importance of this is that apart from using it directly, this SOAP API allows integration of the validator in more complex conformance testing scenarios as a validation step in GITB TDL test cases. This potential is covered further in Step 7: Use the validator in GITB TDL test cases.

The operations supported are as follows:

  • getModuleDefinition: Called to return information on how to call the service (i.e. what inputs are expected).

  • validate: Called to trigger validation for provided content.

You can download this SOAP UI project that includes sample calls of these operations (make sure to change the service URL to match your setup).

Regarding the getModuleDefinition operation, a request of:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v1="http://www.gitb.com/vs/v1/">
   <soapenv:Header/>
   <soapenv:Body>
      <v1:GetModuleDefinitionRequest/>
   </soapenv:Body>
</soapenv:Envelope>

Will produce a response as follows:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <ns4:GetModuleDefinitionResponse xmlns:ns2="http://www.gitb.com/core/v1/" xmlns:ns3="http://www.gitb.com/tr/v1/" xmlns:ns4="http://www.gitb.com/vs/v1/">
            <module operation="V" id="ValidatorService">
                <ns2:metadata>
                    <ns2:name>ValidatorService</ns2:name>
                    <ns2:version>1.0.0</ns2:version>
                </ns2:metadata>
                <ns2:inputs>
                    <ns2:param type="binary" name="contentToValidate" use="R" kind="SIMPLE" desc="The content to validate, provided as a string, BASE64 or a URI."/>
                    <ns2:param type="string" name="embeddingMethod" use="O" kind="SIMPLE" desc="The embedding method to consider for the 'contentToValidate' input ('BASE64', 'URL' or 'STRING')."/>
                    <ns2:param type="string" name="validationType" use="O" kind="SIMPLE" desc="The type of validation to perform (if multiple types are supported)."/>
                    <ns2:param type="boolean" name="hasHeaders" use="O" kind="SIMPLE" desc="A boolean flag specifying whether the provided input contains as its first line the header definitions."/>
                    <ns2:param type="string" name="delimiter" use="O" kind="SIMPLE" desc="The character to be used as the field delimiter."/>
                    <ns2:param type="string" name="quote" use="O" kind="SIMPLE" desc="The character to be used as the quote character."/>
                    <ns2:param type="map" name="schema" use="O" kind="SIMPLE" desc="A map that defines the external Table Schema to consider. The map's items define the following keys: 'content' (the schema content to consider, see 'contentToValidate' for its semantics), 'embeddingMethod' (the way to consider the 'content' value)."/>
                </ns2:inputs>
            </module>
        </ns4:GetModuleDefinitionResponse>
    </soap:Body>
</soap:Envelope>

This response can be customised through configuration properties in config.properties to provide descriptions specific to your setup. For example, extending config.properties with the following:

...
validator.webServiceId = PurchaseOrderValidator
validator.webServiceDescription.contentToValidate = The purchase order content to validate.
validator.webServiceDescription.validationType = The type of validation to perform ('basic' or 'large').

Will produce a response as follows:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Body>
      <ns4:GetModuleDefinitionResponse xmlns:ns2="http://www.gitb.com/core/v1/" xmlns:ns3="http://www.gitb.com/tr/v1/" xmlns:ns4="http://www.gitb.com/vs/v1/">
         <module operation="V" id="ValidatorService">
            <ns2:metadata>
               <ns2:name>PurchaseOrderValidator</ns2:name>
               <ns2:version>1.0.0</ns2:version>
            </ns2:metadata>
            <ns2:inputs>
               <ns2:param type="binary" name="contentToValidate" use="R" kind="SIMPLE" desc="The purchase order content to validate."/>
               <ns2:param type="string" name="embeddingMethod" use="O" kind="SIMPLE" desc="The embedding method to consider for the 'contentToValidate' input ('BASE64', 'URL' or 'STRING')."/>
               <ns2:param type="string" name="validationType" use="O" kind="SIMPLE" desc="The type of validation to perform ('basic' or 'large')."/>
               <ns2:param type="boolean" name="hasHeaders" use="O" kind="SIMPLE" desc="A boolean flag specifying whether the provided input contains as its first line the header definitions."/>
               <ns2:param type="string" name="delimiter" use="O" kind="SIMPLE" desc="The character to be used as the field delimiter."/>
               <ns2:param type="string" name="quote" use="O" kind="SIMPLE" desc="The character to be used as the quote character."/>
               <ns2:param type="map" name="schema" use="O" kind="SIMPLE" desc="A map that defines the external Table Schema to consider. The map's items define the following keys: 'content' (the schema content to consider, see 'contentToValidate' for its semantics), 'embeddingMethod' (the way to consider the 'content' value)."/>
            </ns2:inputs>            
         </module>
      </ns4:GetModuleDefinitionResponse>
   </soap:Body>
</soap:Envelope>

Running the validation itself is done through the validate operation. This expects the following inputs:

Input

Description

Required?

Type

Default value

contentToValidate

The CSV content to validate.

Yes

A string that is interpreted based on the embeddingMethod input.

embeddingMethod

The way in which to interpret the contentToValidate value.

Yes, but should be skipped in favour of the embeddingMethod attribute - see below.

One of BASE64 (for content embedded as a BASE64 encoded string), URI (for a reference to remote content) or STRING (for content embedded as-is).

validationType

The type of validation to perform.

Yes, unless a single validation type is defined.

String

The single configured validation type (if defined).

schema

A user-provided Table Schema to be considered with any predefined one(s). This is accepted only if explicitly allowed in the configuration for the validation type in question.

No

A map (see below for content).

hasHeaders

Whether the content to validate contains a header row. This is taken into account only if configured for the validation type in question.

No

Boolean

true

delimiter

The delimiter character used to separate fields in the provided content. This is taken into account only if configured for the validation type in question.

No

String

,

quote

The quote character used to optionally wrap field values. This is taken into account only if configured for the validation type in question.

No

String

"

differentInputFieldCountViolationLevel

The violation level when the number of input fields doesn’t match the number of schema fields. This is taken into account only if configured for the validation type in question.

No

One of error, warning, info or none.

error

differentInputFieldSequenceViolationLevel

The violation level when the sequence of input fields doesn’t match the one defined in the schema. This is taken into account only if configured for the validation type in question.

No

One of error, warning, info or none.

error

duplicateInputFieldsViolationLevel

The violation level when duplicate input fields are found. This is taken into account only if configured for the validation type in question.

No

One of error, warning, info or none.

error

fieldCaseMismatchViolationLevel

The violation level when a field’s name doesn’t match the casing defined in the schema. This is taken into account only if configured for the validation type in question.

No

One of error, warning, info or none.

error

multipleInputFieldsForSchemaFieldViolationLevel

The violation level when multiple input fields map to the same schema field. This is taken into account only if configured for the validation type in question.

No

One of error, warning, info or none.

error

unknownInputFieldViolationLevel

The violation level when an unknown input field is detected. This is taken into account only if configured for the validation type in question.

No

One of error, warning, info or none.

error

unspecifiedSchemaField

The violation level when a schema field does not have a corresponding value in the input. This is taken into account only if configured for the validation type in question.

No

One of error, warning, info or none.

error

addInputToReport

Whether the validated input should be included in the produced response as the report’s context.

No

boolean

true

Note

Configuration for increased throughput: If you expect to be validating large CSV files and/or with high frequency it would be advised to call the validator setting addInputToReport to false to avoid including the validated content (which could be large) in the resulting report’s context. For huge CSV files that can potentially reach several gigabytes in size it is of course advised that validation is not done via web application at all but rather using the command line tool.

Regarding the schema input, this is treated as a map with two properties:

Input

Description

Required?

Type

content

The schema content to consider.

Yes

A string that is interpreted based on the embeddingMethod input.

embeddingMethod

The way in which to interpret the schema input value.

No

One of BASE64 (for content embedded as a BASE64 encoded string), URL (for a reference to remote content) or STRING (the default - for content embedded as-is).

As an alternative to the embeddingMethod inputs, the GITB SOAP API foresees also the embeddingMethod attribute that is defined on each input element. The values it supports are:

Value

Description

STRING

The value is interpreted as-is as an embedded text.

BASE64

The value is interpreted as an embedded BASE64 string that will need to be decoded before processing.

URI

The value is interpreted as a remote URI reference that will be looked up before processing.

Note

embeddingMethod values: The two approaches to provide the embeddingMethod value (as an input or an attribute) is due to a known issue in the GITB software where not all embedding methods can be leveraged within test cases (see Step 7: Use the validator in GITB TDL test cases).

The sample SOAP UI project includes sample requests per case. As an example, validating via URI would be done using the following envelope:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:v1="http://www.gitb.com/vs/v1/" xmlns:v11="http://www.gitb.com/core/v1/">
   <soapenv:Header/>
   <soapenv:Body>
      <v1:ValidateRequest>
         <sessionId>?</sessionId>
         <input name="contentToValidate" embeddingMethod="URI">
            <v11:value>https://www.itb.ec.europa.eu/files/csv/sample.csv</v11:value>
         </input>
         <input name="validationType" embeddingMethod="STRING">
            <v11:value>large</v11:value>
         </input>
      </v1:ValidateRequest>
   </soapenv:Body>
</soapenv:Envelope>

With the resulting report provided as follows:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <ns4:ValidationResponse xmlns:ns2="http://www.gitb.com/core/v1/" xmlns:ns3="http://www.gitb.com/tr/v1/" xmlns:ns4="http://www.gitb.com/vs/v1/">
            <report>
                <ns3:date>2020-09-23T15:58:42.975+02:00</ns3:date>
                <ns3:result>FAILURE</ns3:result>
                <ns3:counters>
                    <ns3:nrOfAssertions>0</ns3:nrOfAssertions>
                    <ns3:nrOfErrors>1</ns3:nrOfErrors>
                    <ns3:nrOfWarnings>0</ns3:nrOfWarnings>
                </ns3:counters>
                <ns3:context>
                    <ns2:item name="contentToValidate" embeddingMethod="STRING" type="string">
                        <ns2:value>shippingAddressName,shippingAddressStreet,shippingAddressCity,shippingAddressZip,billingAddressName,billingAddressStreet,billingAddressCity,billingAddressZip,orderDate,comment,totalItemQuantity,totalItemCostEUR
John Doe,Europa Avenue 123,Brussels,1000,Jane Doe,Europa Avenue 210,Brussels,1000,2020-01-22,Send in one package please,35,40.99
John Doe,Europa Avenue 123,Brussels,1000,Jane Doe,Europa Avenue 210,Brussels,1000,2020-01-23,,12,120.50
John Doe,Europa Avenue 123,Brussels,1000,Jane Doe,Europa Avenue 210,Brussels,1000,2020-01-24,,5,10.30</ns2:value>
                    </ns2:item>
                </ns3:context>
                <ns3:reports>
                    <ns3:error xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns3:BAR">
                        <ns3:description>[4][totalItemQuantity]: Value '5' for field 'totalItemQuantity' is less than the minimum of 10.</ns3:description>
                        <ns3:location>contentToValidate:4:0</ns3:location>
                        <ns3:value>5</ns3:value>
                    </ns3:error>
                </ns3:reports>
            </report>
        </ns4:ValidationResponse>
    </soap:Body>
</soap:Envelope>

The returned report uses the GITB TRL syntax and is the same as the XML report you can download from the user interface (see Validation via user interface). It includes:

  • The validation timestamp (in UTC).

  • The overall result (SUCCESS or FAILURE).

  • The count of errors, warnings and information messages.

  • The context for the validation (i.e. the CSV content that was validated).

  • The list of report items displaying per item its description and location in the validated content.

Validation via command-line tool

Note

Command-line tool availability: Command-line tools are supported only for validators hosted on test bed resources and if generation of such a tool has been requested (see Step 5: Setup validator on test bed).

When a command line tool is set up for your validator it is available as an executable JAR file, packaged alongside a README file in a ZIP archive. This ZIP archive is downloadable from a URL of the form https://www.itb.ec.europa.eu/csv-offline/DOMAIN/validator.zip, where DOMAIN is the name of the validator’s domain. Considering our purchase order example, the command-line validator would be available at https://www.itb.ec.europa.eu/csv-offline/order/validator.zip.

To use it you need to:

  1. Ensure you have Java running on your workstation (minimum version 11).

  2. Download and extract the validator’s ZIP archive.

  3. Open a command prompt and change to the directory in which you extracted the JAR file.

  4. View the validator’s help message by issuing java -jar validator.jar

> Expected usage: java -jar validator.jar -input FILE_OR_URI_1 ... [-input FILE_OR_URI_N] [-noreports] [-schema SCHEMA_FILE_OR_URI]
    Where:
        - FILE_OR_URI_X is the full file path or URI to the content to validate.
        - SCHEMA_FILE_OR_URI is the full file path or URI to a schema for the validation.

The summary of each validation will be printed and the detailed reports produced in the current directory (as "report.X.xml" and "report.X.pdf").

Running the validator will produce a summary output on the command console as well as the detailed validation report(s) (unless flag -noreports has been specified). To resolve potential problems during execution, an output log is also generated with a detailed log trace.

> java -jar validator.jar -input sample.csv

Validating 1 of 1 ... Done.

Validation report summary [sample.csv]:
- Date: 2020-12-07T17:53:03.396+01:00
- Result: FAILURE
- Errors: 1
- Warnings: 0
- Messages: 0
- Detailed reports in [D:\tmp\report.0.xml] and [D:\tmp\report.0.pdf]

Note

Offline validator use and remote files: Depending on the validator’s configuration, one or more of its configured Table Schemas may be defined as URIs (see Step 3: Prepare validator configuration). In addition, you may also provide the content to validate as a reference to a remote file. In these cases the workstation running the validator would need access to the remote resources. Any proxy settings applicable for the workstation will automatically be used for the connections.

Step 7: Use the validator in GITB TDL test cases

As a next step over the standalone CSV validator you may consider using it from within GITB TDL test cases running in the test bed. You would typically do this for the following reasons:

  • You want to control access to the validation service based on user accounts.

  • You prefer to record all data linked to validations (for e.g. subsequent inspection).

  • You want to build complete conformance testing scenarios that are either focused on the validator or that use it as part of validation steps.

As described in Validation via SOAP web service API, the standalone CSV validator offers by default a SOAP API for machine-to-machine integration that realises the GITB validation service specification. In short this means that the service can be easily included in any GITB TDL test case as the handler of a verify step. This is done by supplying as the handler value the full URL to the service’s WSDL, as illustrated in the following example that requests the user to upload the file to validate:

<?xml version="1.0" encoding="UTF-8"?>
<testcase id="testCase1_upload" xmlns="http://www.gitb.com/tdl/v1/" xmlns:gitb="http://www.gitb.com/core/v1/">
    <metadata>
        <gitb:name>testCase1_upload</gitb:name>
        <gitb:version>1.0</gitb:version>
        <gitb:description>Test case that allows the developer of an EU retailer system to upload a purchase order CSV file for validation.</gitb:description>
    </metadata>
    <variables>
        <var name="purchaseOrderFileToValidate" type="binary"/>
    </variables>
    <actors>
        <gitb:actor id="Retailer" name="Retailer" role="SUT"/>
    </actors>
    <steps>
        <interact desc="Upload content">
            <request desc="Purchase order to validate:">$purchaseOrderFileToValidate</request>
        </interact>
        <verify handler="https://www.itb.ec.europa.eu/csv/soap/order/validation?wsdl" desc="Validate purchase order file">
            <input name="contentToValidate">$purchaseOrderFileToValidate</input>
            <input name="validationType">"basic"</input>
        </verify>
    </steps>
</testcase>

Notice in this example how the contentToValidate and validationType inputs are provided as input elements to the verify step. We included the validationType input because we define two validation types (basic and large) but this could be omitted if only a single validation type is supported. In addition, note that although you can define the service’s WSDL URL directly, a better approach to improve portability is to define this in the test bed’s domain configuration as a domain parameter. Defining a validationService parameter in the domain you could thus redefine the verify step as:

...
<verify handler="$DOMAIN{validationService}" desc="Validate purchase order file">
    ...
</verify>
...

Summary

Congratulations! You have just setup a validation service for your CSV data specification. In doing so you considered your needs and defined your service through configuration on the ISA² test bed or as a Docker container. In addition, you used this service via its different APIs and considered how this could be used as part of complete conformance testing scenarios.

See also

In Step 7: Use the validator in GITB TDL test cases we briefly touched upon using the test bed for complete conformance testing scenarios. If this interests you, several additional guides are available that can provide you with further information:

For the full information on GITB TDL test cases check out the GITB TDL documentation, the reference for all test step constructs as well as a source of numerous complete examples.

In case you need to consider validation of further content types, be aware that the test bed provides similar support for:

If you are planning on operating a validator on your own-premises for production use check the validator production installation guide for the steps to follow and guidelines to consider.

Finally, for more information on Docker and the commands used in this guide, check out the Docker online documentation.

References

This section contains additional references linked to this guide.

Validator configuration properties

The following sections list the configuration properties you can use to customise the operation of your validation service.

Domain-level configuration

The properties in this section are to be provided in the configuration property file (one per configured validation domain) you define as part of your validator configuration.

Note

Property placeholders: Numerous configuration properties listed below are presented with placeholders. These are marked in uppercase and have the following meaning:

  • TYPE: The validation type to perform.

  • OPTION: An option for a given validation type (e.g. a specific version).

  • FULL_TYPE: For a validation type with options this is equals to TYPE.OPTION, otherwise it is the TYPE value itself.

  • N: A zero-based integer (used as a counter).

Property

Description

Type

Default value

validator.channels

Comma-separated list of validation channels to have enabled. Possible values are (form, soap_api).

Comma-separated Strings

form, soap_api

validator.type

Comma-separated list of supported validation types. Values need to be reflected in the other properties’ TYPE and FULL_TYPE placeholders. FULL_TYPE equals TYPE.OPTION (in case type options are defined) or simply TYPE if there are no options (see also validator.typeOptions).

Comma-separated Strings

validator.typeOptions.TYPE

Comma-separated list of options defined for a given validation type (added as a postfix). Values need to be reflects in the other properties’ OPTION and FULL_TYPE placeholders. FULL_TYPE equals TYPE.OPTION (in case type options are defined) or simply TYPE if there are no options (see also validator.type).

Comma-separated Strings

validator.typeLabel.TYPE

Label to display in the web form for a given validation type (added as a postfix of validator.typeLabel). Only displayed if there are multiple types.

String

validator.optionLabel.OPTION

Label to display in the web form for an option across all validation types (added as a postfix of validator.optionLabel). Only displayed if there are options defined.

String

validator.typeOptionLabel.TYPE.OPTION

Label to display for an option for a specific validation type.

String

validator.completeTypeOptionLabel.TYPE.OPTION

Label to display for the full validation type and option combination (visible in the validator’s result screen).

String

validator.webServiceId

The ID of the web service.

String

ValidatorService

validator.webServiceDescription.contentToValidate

The description of the SOAP web service for element “contentToValidate”.

String

The content to validate, provided as a string, BASE64 or a URI.

validator.webServiceDescription.validationType

The description of the SOAP web service for element “validationType”. Only displayed if there are multiple types.

String

The type of validation to perform (if multiple types are supported).

validator.webServiceDescription.embeddingMethod

The description of the SOAP web service for element “embeddingMethod”.

String

The embedding method to consider for the 'contentToValidate' input ('BASE64', 'URL' or 'STRING').

validator.webServiceDescription.schema

The description of the SOAP web service for element “schema”.

String

A map that defines the external Table Schema to consider. The map's items define the following keys: 'content' (the schema content to consider, see 'contentToValidate' for its semantics), 'embeddingMethod' (the way to consider the 'content' value).

validator.webServiceDescription.hasHeaders

The description of the SOAP web service for element “hasHeaders”.

String

A boolean flag specifying whether the provided input contains as its first line the header definitions.

validator.webServiceDescription.delimiter

The description of the SOAP web service for element “delimiter”.

String

The character to be used as the field delimiter.

validator.webServiceDescription.quote

The description of the SOAP web service for element “quote”.

String

The character to be used as the quote character.

validator.defaultDifferentInputFieldCountViolationLevelDescription

The description of the SOAP web service for element “differentInputFieldCountViolationLevel”.

The violation level in case the input field count doesn't match the schema field count ('none', 'info', 'optional', 'required').

validator.defaultDifferentInputFieldSequenceViolationLevelDescription

The description of the SOAP web service for element “differentInputFieldSequenceViolationLevel”.

The violation level in case the input fields' sequence doesn't match the schema fields' sequence ('none', 'info', 'optional', 'required').

validator.defaultDuplicateInputFieldsViolationLevelDescription

The description of the SOAP web service for element “duplicateInputFieldsViolationLevel”.

The violation level in case the duplicate input fields are found ('none', 'info', 'optional', 'required').

validator.defaultFieldCaseMismatchViolationLevelDescription

The description of the SOAP web service for element “fieldCaseMismatchViolationLevel”.

The violation level in case input fields match schema fields but with different casing ('none', 'info', 'optional', 'required').

validator.defaultMultipleInputFieldsForSchemaFieldViolationLevelDescription

The description of the SOAP web service for element “multipleInputFieldsForSchemaFieldViolationLevel”.

The violation level in case multiple input fields map to the same schema field ('none', 'info', 'optional', 'required').

validator.defaultUnknownInputFieldViolationLevelDescription

The description of the SOAP web service for element “unknownInputFieldViolationLevel”.

The violation level in case of unknown input fields ('none', 'info', 'optional', 'required').

validator.defaultUnspecifiedSchemaFieldDescription

The description of the SOAP web service for element “unspecifiedSchemaField”.

The violation level in case schema fields are not defined ('none', 'info', 'optional', 'required').

validator.schemaFile.FULL_TYPE

Comma-separated list of schema files loaded for a given validation type (added as a postfix). These can be a files or folders.

Comma-separated Strings

validator.schemaFile.FULL_TYPE.remote.N.url

Reference for a remotely loaded schema file for a given validation type (added as the FULL_TYPE placeholder). One or more such entries can be defined by incrementing the zero-based N counter.

String

validator.externalSchema.FULL_TYPE

Whether or not a user-provided schema is allowed for the given validation type (added as a postfix). Possible values are (required, optional, none).

String

none

validator.hasHeaders.FULL_TYPE

Whether the input is expected to have a header row.

Boolean

true

validator.delimiter.FULL_TYPE

The delimiter character to use.

Character

,

validator.quote.FULL_TYPE

The quote character to use.

Character

"

validator.differentInputFieldCountViolation.FULL_TYPE

The violation level when the number of input fields doesn’t match the number of schema fields. Possible values are error, warning, info, none).

error

validator.differentInputFieldSequenceViolation.FULL_TYPE

The violation level when the sequence of input fields doesn’t match the defined sequence in the schema. Possible values are error, warning, info, none).

error

validator.unknownInputFieldViolation.FULL_TYPE

The violation level when an input field is not found in the schema fields. Possible values are error, warning, info, none).

error

validator.unspecifiedSchemaFieldViolation.FULL_TYPE

The violation level when a schema field is not provided for in the input. Possible values are error, warning, info, none).

error

validator.inputFieldCaseMismatchViolation.FULL_TYPE

The violation level when the name of an input field does not match the casing defined in the schema. Possible values are error, warning, info, none).

error

validator.duplicateInputFieldsViolation.FULL_TYPE

The violation level when duplicate input fields are found. Possible values are error, warning, info, none).

error

validator.multipleInputFieldsForSchemaFieldViolation.FULL_TYPE

The violation level when multiple input fields map to the same schema field. Possible values are error, warning, info, none).

error

validator.input.hasHeaders.FULL_TYPE

Whether users may/must specify if their provided input has headers. Possible values are (required, optional, none).

String

none

validator.input.delimiter.FULL_TYPE

Whether users may/must specify the delimiter character for the provided input. Possible values are (required, optional, none).

String

none

validator.input.quote.FULL_TYPE

Whether users may/must specify the quote character for the provided input. Possible values are (required, optional, none).

String

none

validator.input.differentInputFieldCountViolation.FULL_TYPE

Whether users may/must specify the violation level for a field count mismatch. Possible values are (required, optional, none).

String

none

validator.input.differentInputFieldSequenceViolation.FULL_TYPE

Whether users may/must specify the violation level for a field sequence mismatch. Possible values are (required, optional, none).

String

none

validator.input.unknownInputFieldViolation.FULL_TYPE

Whether users may/must specify the violation level for unknown fields. Possible values are (required, optional, none).

String

none

validator.input.unspecifiedSchemaFieldViolation.FULL_TYPE

Whether users may/must specify the violation level for schema fields not covered in the input. Possible values are (required, optional, none).

String

none

validator.input.inputFieldCaseMismatchViolation.FULL_TYPE

Whether users may/must specify the violation level for a case difference in field names. Possible values are (required, optional, none).

String

none

validator.input.duplicateInputFieldsViolation.FULL_TYPE

Whether users may/must specify the violation level for duplicate input fields. Possible values are (required, optional, none).

String

none

validator.input.multipleInputFieldsForSchemaFieldViolation.FULL_TYPE

Whether users may/must specify the violation level when multiple input fields map to the same schema field. Possible values are (required, optional, none).

String

none

validator.showAbout

Whether or not to show the about panel on the web UI.

Boolean

true

validator.supportMinimalUserInterface

Enable a minimal user interface useful for embedding in other UIs or portals (applies only if the form validation channel is enabled).

Boolean

false

validator.bannerHtml

Configurable HTML banner replacing the text title.

String

validator.footerHtml

Configurable HTML banner for the footer.

String

validator.defaultPlugins.N.jar

Configuration for custom plugins that can be used to extend the validation. This is a default plugin definition that applies to all validation types. This property is the relative path pointing to the all-in-one JAR file that contains the plugin implementation(s).

String

validator.defaultPlugins.N.class

Paired with the its corresponding jar property (see above), this defines the fully qualified names of the classes to be loaded as plugin implementations.

Comma-separated Strings.

validator.plugins.FULL_TYPE.N.jar

Configuration for plugins that can be used to extend the validation specific to a given type (FULL_TYPE) and extending any default plugins (see above). This property is the relative path pointing to the all-in-one JAR file that contains the plugin implementation(s).

String

validator.plugins.FULL_TYPE.N.class

Paired with the its corresponding jar property (see above), this defines the fully qualified names of the classes to be loaded as plugin implementations.

Comma-separated Strings.

validator.maximumReportsForDetailedOutput

The maximum number of report items for which a PDF validation report will be generated (no report will be preoduced if exceeded).

Integer

5000

validator.maximumReportsForXmlOutput

The maximum number of report items to include in the XML validation report.

Integer

50000

validator.importProperties

The path of a file declaring additional domain properties. If the application property validator.restrictResourcesToDomain is set to true or omitted, only relative paths of files under the domain root folder are accepted.

String

Application-level configuration

These properties govern the validator’s application instance itself. They apply only when you are defining your own validator as a Docker image in which case they are supplied as environment variables (ENV directives in a Dockerfile). Note that apart from these properties any Spring Boot configuration property can also be supplied.

Note

Mandatory property: The only mandatory property that needs to be defined is validator.resourceRoot.

Property

Description

Type

Default value

validator.resourceRoot

The root folder under which domain subfolders will be loaded from.

String

validator.domain

The names of the domain subfolders to consider. By default all folders under validator.resourceRoot will be considered.

Comma-separated Strings

validator.domainName.DOMAIN

The name to display for a given domain folder (the folder name replacing the DOMAIN placeholder). This value will also be used in request paths.

String

The folder name is used.

logging.file.path

Path to a folder that will hold the validator’s log output.

String

/validator/logs

validator.tmpFolder

Path to a folder that contains temporary data and reports.

String

/validator/tmp

validator.acceptedSchemaExtensions

Accepted local schema file extensions. All other files found in validator.schemaFile.FULL_TYPE (when folders are defined) are ignored.

Comma-separated Strings

json

validator.cleanupRate

The rate at which the external file cache is refreshed (in milliseconds).

Long

3600000

validator.cleanupWebRate

The rate at which temporary files linked to the web form are cleaned (in milliseconds).

Long

600000

validator.minimumCachedInputFileAge

The minimum time for which input provided through the web interface is cached (in milliseconds).

Long

600000

validator.minimumCachedReportFileAge

The minimum time for which reports generated through use of the web interface are cached (in milliseconds).

Long

600000

validator.acceptedMimeTypes

The accepted mime types for user-provided content through the user interface.

Comma-separated Strings

text/csv, text/plain

validator.restrictResourcesToDomain

Whether local validation artefacts can be loaded from outside the domain root folder. If false, both relative and absolute paths are accepted.

Boolean

true

validator.webhook.statistics

The URL of the backend service that will collect usage statistics. This property is optional and, if (and only if) present, the validator will report usage statistics.

String

validator.webhook.statisticsSecret

The validator client secret to be passed to the backend service for usage statistics reporting.

String

validator.identifier

The validator’s identifier to be sent for usage statistics reporting.

String

csv

validator.webhook.statisticsEnableCountryDetection

Whether the usage statistics reporting service can detect users’ countries from their IP addresses. If false no information about the user’s country or IP address will be reported.

Boolean

false

validator.webhook.statisticsCountryDetectionDbFile

The path to the .mmdb file with the geolocation database that is used to resolve the user’s country from an IP address. This database will only be used when the property validator.webhook.statisticsEnableCountryDetection takes the value true.

String

validator.webhook.ipheader

The HTTP header to use to retrieve the user’s IP address for usage statistics if the validator is behind a proxy. This property is only used to detect the user’s country, when such feature is enabled (see validator.webhook.statisticsEnableCountryDetection).

String

X-Real-IP

Change history