Verne is designed to use a document database for persistence which provides great flexibility for storing and retrieving versions of registry data. Each document is a fully inflated view of registry data at a particular point in time. More can be read about the Service Transaction and the life-cycle of registry data here.

One important point to realise is that in Verne, every business service results in a fresh service transaction being created. For example a dashboard request is a transaction, every view is a copy of the transaction we are viewing, a search is a service transaction as well. This is great for audit purposes, business intelligence and registry insights but it can result in 100’s of million of documents in a running application.

Single Collection

By default Verne stores all service transactions in a single collection service_transactions in a document database (our default reference document DB is MongoDB). Storing all transactions in one collection has both pros and cons:

Pros

  • A single collection to query/backup/manage
  • Always know where to look for registry data via id->single collection

Cons

  • Indexes are not targeted and apply to every transaction in collection (for example you never really need to index dashboards, but you do want to selectively apply some indexes to companies for lookups etc.). This also has the performance and storage side affect of large indexes which are not actually required.
  • Only can have 64 indexes (in mongo) – which has to cover all possibilities for all transactions.
  • IDs are not enough to identify the location of a transaction, you must now know the root domain name.

Multi Collection

Verne also provides the capability for configurators to choose where their service transactions are stored based on root domain name and the service mode. This allows very fine grained control of which transactions are stored where. This has more pros and is recommended by the Mongo technical architecture team as well. Going forward Verne will allow collections to be configured to different databases and hosts thus allowing true partitioning to fast/slow disks/network/memory depending upon the collection requirements.

Pros

  • Custom targeted indexes for each collection, smaller indexes -> greater performance.
  • BAU Verne transactions such as dashboards/email/searches/views do not affect the core Business Register collections
  • Keeps business collections smaller as does not have BAU transactions all together.

Configuration

To configure multi collections in your Verne application you leverage the dataset configuration. The dataset describes information about a root domain and you can configure the collections to use inside there. The persistence engine within Verne considers the root domain name of the transaction and uses that and the id to determine where it should be saved/retrieved.

Consider the following xml configuration snippet:

<dataset code="user" rootDomainName="User" authorisedEntity="true">
  <collection name="security_users"/>
  <collection name="security_views" mode="View"/>
  <collection name="security_searches" mode="Search"/>
</dataset>

This means:

  • Any view transactions for User domains go into the security_views
  • Any search transactions for User domains go into the security_searches
  • All other transactions for User domains (e.g register, maintain, correct) go into the security_users.

Here is an example configuration for low level internal Verne transactions. You will note mode is never configured here but all transactions have their own dedicated collections.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<catcfg:Configuration xmlns="http://www.fostermoore.com/schema/cat-ng" xmlns:catcfg="http://www.fostermoore.com/schema/catcfg-ng">

  <datasets>
    <dataset code="serviceAccessError" rootDomainName="ServiceAccessError">
      <collection name="service_errors"/>
    </dataset>
    <dataset code="serviceAccessError" rootDomainName="ServiceTransactionAccessError">
      <collection name="service_errors"/>
    </dataset>
    <dataset code="session" rootDomainName="Session">
      <collection name="sessions"/>
    </dataset>
    <dataset code="schedule" rootDomainName="Schedule">
      <collection name="schedules"/>
    </dataset>
    <dataset code="scheduleJob" rootDomainName="ScheduleJob">
      <collection name="schedule_jobs"/>
    </dataset>
    <dataset code="slowRequest" rootDomainName="SlowRequest">
      <collection name="slow_requests"/>
    </dataset>
    <dataset code="businessCalendar" rootDomainName="BusinessCalendar">
      <collection name="business_calendars"/>
    </dataset>
    <dataset code="emailBlacklist" rootDomainName="EmailBlacklist">
      <collection name="email_blacklists"/>
    </dataset>
    <dataset code="smsDelivery" rootDomainName="SmsDelivery">
      <collection name="sms_deliveries"/>
    </dataset>
    <dataset code="emailDelivery" rootDomainName="EmailDelivery">
      <collection name="email_deliveries"/>
    </dataset>
    <dataset code="home" rootDomainName="Home">
      <collection name="dashboards"/>
    </dataset>
    <dataset code="dashboard" rootDomainName="Dashboard">
      <collection name="dashboards"/>
    </dataset>
    <dataset code="tokenInvalid" rootDomainName="TokenInvalid">
      <collection name="invalid_tokens"/>
    </dataset>
  </datasets>

</catcfg:Configuration>

And here is an example for the security application which puts all views and searches for different domains in the same collection.

<catcfg:Configuration xmlns="http://www.fostermoore.com/schema/cat-ng" xmlns:catcfg="http://www.fostermoore.com/schema/catcfg-ng">

  <datasets>
    <dataset code="userLogon" rootDomainName="UserLogon" authorisedEntity="true">
      <collection name="security_users"/>
      <collection name="security_views" mode="View"/>
      <collection name="security_searches" mode="Search"/>
    </dataset>
    <dataset code="user" rootDomainName="User" authorisedEntity="true">
      <collection name="security_users"/>
      <collection name="security_views" mode="View"/>
      <collection name="security_searches" mode="Search"/>
    </dataset>
    <dataset code="group" rootDomainName="Group" authorisedEntity="true">
      <collection name="security_groups"/>
      <collection name="security_views" mode="View"/>
      <collection name="security_searches" mode="Search"/>
    </dataset>
    <dataset code="team" rootDomainName="Team" authorisedEntity="true">
      <collection name="security_teams"/>
      <collection name="security_views" mode="View"/>
      <collection name="security_searches" mode="Search"/>
    </dataset>
    <dataset code="securityToken" rootDomainName="SecurityToken">
      <collection name="security_tokens"/>
      <collection name="security_views" mode="View"/>
      <collection name="security_searches" mode="Search"/>
    </dataset>
    <dataset code="organisations" rootDomainName="Organisation">
      <collection name="security_organisations"/>
      <collection name="security_views" mode="View"/>
      <collection name="security_searches" mode="Search"/>
    </dataset>
    <dataset code="twoFactorAuthentication" rootDomainName="TwoFactorAuthentication">
      <collection name="security_two_factor_authentications"/>
      <collection name="security_views" mode="View"/>
      <collection name="security_searches" mode="Search"/>
    </dataset>
  </datasets>
</catcfg:Configuration>

Recommendations

Our recommendations for configuring your collection storage strategy is to go with separate collections for most if not all of your service transactions by root domain name. You should also separate out all views and searches into distinct collections.
0
0
Previous
Data Persistence
Next
User Interface Architecture

Jump to Section