Web Services

Verne supports web services via configuration by automatically leveraging forms and business services.

SOAP

SOAP 1.1 was released in 2000 and leverages XSD/WSDL XML technology to provide well-defined contracts. Verne has always had strong support for web services from the start. Effectively you configure a web service to point at a business service and Verne will crawl the form and dynamically build out a SOAP-based WSDL contract and provide an endpoint for the service to be called at.

The WSDLs created are specific to Verne and exactly reflect the form it is created from. Upon XML arriving at the SOAP endpoint Verne does the following:

1. Parses the incoming SOAP message
2. Determines the web service name based on the SOAP message supplied namespace
3. Looks up the business service for the web service
4. Depending upon the type of service creates a new service transaction or loads existing
5. Marshalls the SOAP XML body supplied on top of the service transaction
6. Executes any business rules such as activation etc.
7. Unmarshalls the Service Transaction back into a SOAP XML body
8. Returns a SOAP response back to the client.

In effect, the SOAP support is relatively simple conceptually and configurators need only to configure one extra configuration item to provide robust and repeatable web services for their clients.

An example configuration item to configure a web service is:

<catcfg:Configuration xmlns:catcfg="http://www.fostermoore.com/schema/catcfg-ng" xmlns="http://www.fostermoore.com/schema/cat-ng">
  <webServices>
    <webService name="professionalSearch" service="professionalSearch">
      <securityPermission permission="PERM_U002_EXTERNAL_REGISTERED_USER"/>
    </webService>
  </webServices>
</catcfg:Configuration>

Rest

REST (Representational state transfer) services is an architectural style used to define an API and interoperability over the web. The REST protocol is stateless and provides a set of loose definitions based on HTTP Operations allowing developers to define their own interfaces more flexibly.

Unlike SOAP (a very formal protocol) REST does not define specific contracts and initially, REST APIs had no mechanism to formally define the contract. This made it very hard to formally define contracts and validate them both on the server and in the client.

Open API

• Swagger was a specification created in 2011 to help document RESTful APIs and help generate client development kits.
• By 2014 interest in Swagger was growing with public endorsement by IBM, Apple, Microsoft and many others.
• In 2015 a new open-source organization was created called the Open API initiative. A variety of prominent software companies – Google, Apple, Microsoft, IBM are founding members.
• On January 2016 – The Swagger specification was renamed the OpenAPI specification.
• In July 2017 – Version 3.0.0 of the OpenAPI specification was released.

The Open API and it’s tooling provide 3 key API use cases:

1. Developing APIs – via a contract first mechanism
2. Interacting with APIs – by building client SDKs based on a specification
3. Documenting APIs – the specification provides placeholders for comprehensive documentation and examples. These specifications can be written in JSON or YAML.

It is important to understand Open APIs as Verne directly leverage the extensibility of the API. The main key parts of the specification from a Verne perspective.

Verne and Open API.

The OpenAPI specification allows extension blocks in each definition. This was a clever design decision and one that Verne leverages extensively. This extension mechanism allows configurators to use standard OpenAPI specifications with x-verne properties to drive the Verne engine.

Verne still leverages business services and forms to provide structure and business rules and as for web services overlays incoming Open API specified JSON onto forms. The engine uses the extensions the configurator has placed in the OpenAPI specification to specify things like the endpoint URI, the service to start, actions etc.

Verne also provides a runtime view of the OpenAPI specification which has the extensions removed and (optionally) a fully generated schema structure of your forms. It also provides a nice capability to test your API’s and experiment with them through the Open API toolset.

By jumping off the back of the OpenAPI specification Verne allows configurators to communicate/collaborate on the design of their APIs early.

One important thing to understand is that Verne does not map between an Open API schema and the form. It expects the schema to match the form. Verne can automatically generate the schema for you based on the form. What the OpenAPI specification does is effectively provide documentation, endpoint mapping and hooks for the configurator to tell the engine which services to execute.

Error Handling

One of the challenges with JSON based APIs is handling errors. With XML based APIs one can locate the error with the offending attribute value whereas with JSON an attribute is typically the actual value. Verne has taken the standard approach of returning a list of errors with the response; each error has a code, text and the JSON path to the offending item, be it a repeater, a record or a simple attribute. This keeps the data clean and separate from the errors but does mean clients have to manage these errors out of context. An example of a failing submission might be:

{
  "businessIdentifier": "ETY-0416e77cdc94a60b",
  "versionIdentifier": "d71fedc0570390c85e9351aa5259ecb0",
  "name": "Sutts LTD",
  "annualFilingMonth": 11,
  "companyAddress": {
    "_type": "postalAddress",
    "identifier": "c5403424c6562c1e",
    "posLine2": "Mt Victoria"
  },
  "bosses": [
    {
      "identifier": "2d0fdadfe1937dc9",
      "lastName": "clausman",
      "education": {
        "degree": "BCOM"
      }
    },
    {
      "identifier": "3737020ecf55a860",
      "firstName": "donald",
      "education": {}
    }
  ],
  "workers": [
    {
      "identifier": "6835ec5d8fdd3795",
      "firstName": "david",
      "lastName": "clausman",
      "position": "Janitor"
    },
    {
      "identifier": "98bec5c47f9357db",
      "firstName": "cheryl",
      "lastName": "hoffman",
      "position": "CEO"
    }
  ],
  "shareholders": [],
  "errors": [
    {
      "path": "companyAddress.posLine1",
      "code": "error.mandatory",
      "message": "Pos Line 1 is mandatory"
    },
    {
      "path": "bosses[0].firstName",
      "code": "error.mandatory",
      "message": "First Name is mandatory"
    },
    {
      "path": "bosses[1].lastName",
      "code": "error.mandatory",
      "message": "Last Name is mandatory"
    },
    {
      "path": "bosses[1].education.degree",
      "code": "error.mandatory",
      "message": "Degree is mandatory"
    }
  ]
}

Example

Let’s dive into it as I think it will be easier to understand. You start with a simple Open API configuration element and you point it at your Open API YAML specification. Note it is beyond the scope of this document to discuss the full Open API and that is left as an exercise for the reader. Suffice to say Open API allows you to configure endpoints and specify schemas and Verne will leverage this API to provide support for your services.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<catcfg:Configuration xmlns="http://www.fostermoore.com/schema/cat-ng" xmlns:catcfg="http://www.fostermoore.com/schema/catcfg-ng">

  <openApiServices>
    <openApiService name="professional-api" specificationResource="/openApis/professionalApi.yaml"/>
  </openApiServices>

</catcfg:Configuration>

And here is the OpenAPI specification for the API. This is a fully Open API compliant YAML file with Verne extensions to inform the engine which services. Note the extensive comments below indicate what Verne is doing with the specified extensions.

openapi: 3.0.1
info:
  title: Professional Open API
  version: 1.0.1
servers:
  - url: http://localhost:8080/professionals/rest/professional-api/v1 # Verne ignores this as it is informational only
    x-verne:  
      # Our very first x-verne extension which tells the engine the endpoint uri.
      # Verne builds this by using the $HOST:$PORT/$APP_CODE/rest/$OPEN_API_NAME$pathPrefix
      # This allows flexibility for the configuration to manage different versions of the API at different endpoints. 
      pathPrefix: /v1 
paths:
  /professionals:
    get:
      summary: Search professionals
      operationId: professionalSearch
      x-verne:
        # Here we tell Verne to start a professionalSearch service and execute the specified rule straight away
        serviceCode: professionalSearch
        ruleExecution: professionals.search.execute
      parameters:
      - name: query
        in: query
        required: false
        description: Search query
        schema:
          type: string
      responses:
        '200':
          description: search results
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ProfessionalSearchResults'
  /professional:
    post:
      summary: Creates professional
      operationId: professionalRegister
      x-verne:
        # Here we tell Verne to start a professionalRegisterWs service and activate it once marshalled
        # thus effectively applying it.
        serviceCode: professionalRegisterWs
        action: activate
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Professional'
      responses:
        '201':
          description: created new professional
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Professional'
  /professional/{identifier}:
    get:
      summary: Retrieves professional
      operationId: professionalView
      x-verne:
        # Here we tell Verne to start a professionalView and check the open api parameter configuration for 
        # 'identifier' which will detail how the identifier is specified (in this case it is part of the URI aka)
        # As an example the full URL of this service would be resolved to 
        # http://localhost:8080/professionals/rest/professional-api/v1/professional/ETY78987   
        serviceCode: professionalView
        identifierParameterName: identifier
      parameters:
        - name: identifier
          in: path
          required: true
          description: Identifier of the professional
          schema:
            type: string
      responses:
        '200':
          description: view of a single professional
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Professional'
  /professional/{identifier}/residentialAddress:
    get:
      summary: Retrieves professional residential address
      operationId: professionalViewResidentialAddress
      x-verne:
        # Here we tell Verne to start a professionalView and check the open api parameter configuration for 
        # 'identifier' which will detail how the identifier is specified (in this case it is part of the URI aka)
        # As an example the full URL of this service would be resolved to 
        # GET http://localhost:8080/professionals/rest/professional-api/v1/professional/ETY78987/residentialAddress
        # The outgoingSubDocumentPath allows us to select just one part of the whole professional document to 
        # return; in this case the residentialAddress   
        serviceCode: professionalView
        identifierParameterName: identifier
        outgoingSubDocumentPath: residentialAddress
      parameters:
        - name: identifier
          in: path
          required: true
          description: Identifier of the professional
          schema:
            type: string
      responses:
        '200':
          description: residential address of professional
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ResidentialAddress'
    put:
      summary: Updates professional residential address
      operationId: professionalUpdateResidentialAddress
      x-verne:
        # Here we tell Verne to start a professionalUpdateResidentialAddressWs and check the open api parameter
        # configuration for 'identifier' which will detail how the identifier is specified (in this case it is 
        # part of the URI aka)
        # As an example the full URL of this service would be resolved to 
        # PUT http://localhost:8080/professionals/rest/professional-api/v1/professional/ETY78987/residentialAddress
        # The incomingTargetPath allows us to select just one part of the whole professional document to overlay the JSON
        # onto; in this case the residentialAddress. Effectively this allows us to update a part of the document.
        # After the JSON is overlaid on the service it is "activated"   
        serviceCode: professionalUpdateResidentialAddressWs
        identifierParameterName: identifier
        incomingTargetPath: .residentialAddress
        action: activate
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ResidentialAddress'
      parameters:
        - name: identifier
          in: path
          required: true
          description: Identifier of the professional
          schema:
            type: string
      responses:
        '200':
          description: updated residential address of professional
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ResidentialAddress'
  /professional/{identifier}/nameChange:
    put:
      summary: Updates professional name
      operationId: professionalNameChange
      x-verne:
        # Here we tell Verne to start a professionalNameChange and check the open api parameter configuration for 
        # 'identifier' which will detail how the identifier is specified (in this case it is part of the URI aka)
        # As an example the full URL of this service would be resolved to 
        # http://localhost:8080/professionals/rest/professional-api/v1/professional/ETY78987/residentialAddress
        # The incomingTargetPath this time is a Vpath thus allowing us to select a specific part of the whole professional 
        # document to overlay the JSON onto; in this case the residentialAddress. Effectively this allows us to update a
        # part of the document.   
        # After the JSON is overlaid on the service it is "activated"   
        serviceCode: professionalNameChange
        identifierParameterName: identifier
        incomingTargetPathType: VPath
        incomingTargetPath: /**/individualNameCurrentRepeater/name[visible=true]/name
        action: activate
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Name'
      parameters:
        - name: identifier
          in: path
          required: true
          description: Identifier of the professional
          schema:
            type: string
      responses:
        '200':
          description: updated name of professional
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Name'
  /professional/{identifier}/postalAddress:
    get:
      summary: Retrieves professional postal address
      operationId: professionalViewPostalAddress
      x-verne:
        serviceCode: professionalView
        identifierParameterName: identifier
        outgoingSubDocumentPath: postalAddress
      parameters:
        - name: identifier
          in: path
          required: true
          description: Identifier of the professional
          schema:
            type: string
      responses:
        '200':
          description: postal address of professional
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PostalAddress'

components:
  x-verne: 
    # triggers generation of schemas and replacing all of them below
    # will create a skeleton of the service professionalRegisterWs and recurse it 
    # generating an open api schema.
    generate: true
    serviceCode: professionalRegisterWs
    rootSchemaName: Professional
  # These schemas can be hand written and designed but MUST represent the actual structure
  # of the form to work correctly.
  schemas:
    ProfessionalSearch:
      type: object
      x-verne:
        # We leave this object definition here else verne would remote it
        remove: false
    ProfessionalSearchResults:
      type: object
      x-verne:
        # We leave this object definition here else verne would remote it
        remove: false
      properties:
        SearchResults:
          $ref: '#/components/schemas/ProfessionalSearchResultsPage'
    ProfessionalSearchResultsPage:
      type: array
      x-verne:
        remove: false
      items: 
        $ref : '#/components/schemas/Professional'
    Professional:
      type: object
    ResidentialAddress:
      type: object
    PostalAddress:
      type: object
    EmailAddress:
      type: object

Versioning

Verne Rest Services support versioning but it is not done automatically but relies on configurators managing the interfaces they expose. This is because APIs are used by external clients and are typically tightly controlled and upgraded at a slow pace supporting older versions whilst new clients migrate to the later versions.
As for labelling releases we would recommend using something like Semantic Versioning.

The simplest approach is to modify the path prefix in your specification and have parallel OpenAPI specifications deployed for each version supported. Thus you could /v1 and /v2 and even have different forms, services to implement different versions of the APIs. Alternatively your business rules could determine the version and trigger different behaviour as the business sees fit.

It is also possible to build versions into your paths and extend the current API specification as you add and retire versions.

It is important to note that maintaining multiple versions of an API requires that there are no breaking changes to the underlying business services. For example adding a mandatory data item with no default to the service would prevent requests against older versions of the API from validating.

Best Practice

Verne Rest Services supports two main approaches to service design.

Contract First

In Contract First an OpenAPI 3.0 Specification can be created, socialised and consulted on, then mapped using the Verne extensions to the forms which back a particular service.

Model First

In Model First the form and service model is defined first in Magellan using an iterative design process, then an OpenAPI specification can be created which uses schema generated from the form(s).

Recommended

Either model is suitable but one recommended best practice is to keep your web service forms separate from your UI based forms. You still share business logic rules but you keep the forms distinct. Verne is agnostic on whether you want to share forms between web services and UI forms but there are multiple reasons for not doing this:

1. UI Forms tend to suffer a lot of churn and have UI specific rules, sometimes complex structures which are just not needed for web services.
2. Web services tend to require to be a lot more stable and can be reflected with way simpler forms. Just repeaters, records and attributes tend to do the trick for a web service.
3. Web services need to be versioned correctly to not break clients. Configurators updating UI forms do not always consider these needs and can forget about the web service view of the form.