Lesson 6: Search

Goal

To create a search form for searching for our super heroes so they can be filtered and viewed.

Starting code

Start tag: lesson/06

Requirements

The person who registered the super hero can now view the details via the link on the dashboard, but we also want other users to be able to search for and view the super hero data.

The "Search Super Heroes" search should have a single search bar at the top with a field that searches the "Name" value of super heroes. There should also be an expandable "advanced" search criteria section that includes the following search options:

• A “Super powers” field that searches against their list of super powers
• A “Workflow status” field for the service transaction status with values “Pending” and “Completed” for internal users only, and be limited to only “Completed” service transactions for external and guest users
• A “Caped crusaders” radio button that matches against super heroes who wear capes

Matched results should display the super hero’s name as a link to the super hero where you can view their details. We also only want the uniform related questions to display if you have authority over the super hero registration (i.e. you were the one who registered it in our case so far).

Steps

Add a menu item

Because a search business service isn’t a maintenance of existing registry data (even though they search existing data) and also because we want it to be available to everyone we’ll go ahead and add it to the main menu. We’ll do this by adding a new entry in the existing menuItems.xml file in the "common" module within the heroes structure. Go ahead and try adding it yourself, underneath the "Register a Super Hero" menu item before checking it against the example configuration below. Don’t forget to add the "Search Super Heroes" entry in the en.xml file within the "common" module too.

<menuitem code="heroSearch" type="service" service="heroSearch" target="heroes" textKeyPrefix="menu.heroes.search"/>

Now check it appears as a new menu item shown in Fig 6.1. Clicking on it will give an error, but we’ll fix that up by adding the relevant business service and form in the next step.

For extra credit, play around with the menu item configuration moving the search item to a different group or adding some additional nested structure and see the impact this has on how the menus are displayed in the application.

Create a search business service

We’ve already done this a few times adding our registration and view business services, so you should feel quite comfortable doing this now, so go ahead and add a business service and form both called "heroSearch". The key difference is the mode attribute value in the business service which should be… you guessed it: "Search". Also add the localised text key entry for its title with the text "Search Super Heroes". Also give it a formName attribute value of "heroSearch" which we’ll get to in the next step.

Create a search form

Firstly, create a new form resource called "heroSearch" as you’ve done before for the registration and view business service forms. This gives the familiar root <record> element, and the shortCode attribute value should be set to "heroSearch". The "domain" attribute value is an interesting one though. What do you think it should be? You might think "Hero" is a suitable value, but this service isn’t actually loading in a single existing hero, or creating a new one, but rather searching for them. So "SearchRequest" or something feels a little more appropriate. Since we might have multiple searches within an application we’ll also include some context regarding what’s being searched, so in this case we’ll use "HeroSearch".

Because this is a new root domain we should also add a new data set for this new root domain and configure a well-named collection (say, "heroes_search") so our srevice transactions relating to these searches are stored within their own separate MongoDB collection. This should give you the data set configuration shown below.

<dataset code="heroesSearch" rootDomainName="HeroSearch">
  <collection name="heroes_search"/>
</dataset>

Now check it correctly loads our new, empty search service form as per Fig 6.2 below.

Add search index configuration

This part is new. We need to add search index configuration so our hero data gets indexed as Elasticsearch data we can then search. The search documentation outlines the indexing process and configuration as well as how to perform searches, so make sure you read through that before continuing and feel free to keep going back to it throughout this lesson.

First add the search index configuration file by right-clicking on the "heroes/searchIndexes" directory and selecting a type of "Xml File" (unfortunately Magellan doesn’t have a template for search index configuration resources at the time of writing) and use the "File name" value of "heroSearch". Then fill out the basic structure of a search index including a root "Hero" <domain> element so you have something similar to the following configuration.

<catcfg:Configuration xmlns="http://www.fostermoore.com/schema/cat-ng"
                      xmlns:catcfg="http://www.fostermoore.com/schema/catcfg-ng">

  <searchIndexes>
    <searchIndex name="heroSearch" serviceTransactionDomainTree="Hero">
      <domain name="Hero">
      </domain>
    </searchIndex>
  </searchIndexes>
</catcfg:Configuration>

Search index data

We now have some search index configuration with nothing in it. Go to the System Administration console again and select the Search Index tab which should give you a view similar to the one in Fig 6.3 below.

Nice! ? As you can see there’s an entry for "heroSearch" although there’s no data in it (as indicated by the lack of data within the cells from the "Index Alias" column onwards). Also notice the other search indexes we haven’t explicitly configured, but included either as part of the included registry-quick-start modules configuration (e.g. "complianceSearch"), or by the core product (e.g. "sessionSearch").

Re-indexing

Although the "heroSearch" search index configuration we just added is still rather minimal, let’s be daring and run the batch index to reindex our MongoDB data anyway and see what happens. So click the Batch Index tab within the System Administration console and behold the beauty… when you’ve recovered go ahead and click the "Run batch index" button at the top and confirm you want to go ahead when asked. Then faster than a speeding bullet (well, a bullet shot quite far ?), or about 10s later, your Batch index tab should now resemble something like the one in Fig 6.4 below.

Inspect the search index data

Select the Search Index tab again to return to it and see what’s changed. You’ll notice now there are details and statistics relating to our "heroSearch" index. Click on the row for the "heroSearch" and direct your attention to the area below the details table and you’ll notice a number of tabs starting with "Query", "Query (Examples)", "Query (Raw)", etc. Take time to inspect them all because they all have their uses, but for now we’ll concentrate on the first two. Select the first tab, "Query" and notice there’s a pre-loaded query, which has been executed with the matching results shown in the right hand panel as per Fig 6.5 below.

Take time to look at the returned results… where did all that data come from? We’ve only configured the root domain element so far, so what gives? As outlined in the search index <serviceTransaction> element documentation, certain properties are included by default, even if not configured explicitly. You’ll notice the details returned in the results panel closely mimics these default properties for the most part.

Now also look at the second tab, "Query (Examples)", with lots of different examples of how to query data, including "starts with" matches against string data values, "greater than" for index data such as numbers or dates, etc. Have a play around for a bit and get comfortable with querying the data using this simplified syntax. You can separate different queries by line spaces and run them using the Ctrl + Enter shortcut keys.

Spend some time querying the data using the “Query” tab and using the “Query (Examples)” tab as a reference. Also check out the “Query (Raw)” tab after each query to see what the full, raw Elasticsearch JSON query would look like by comparison.

Remember, you can specify multiple different queries within the query tab, separated by new lines as illustrated in Fig 6.6 below so you don’t have to keep re-writing common queries. You can even exit and come back later and your previous queries will still be lovingly waiting for you. Magic 🙂

Add index data

Now that we’ve got a good handle on what data has already been included in the "heroSearch" index we need to decide what other data we want to include. As mentioned in the search documentation the general rule of thumb is anything you want to either query on within the search criteria or display in the results should go into the index. So looking at our requirements that means we’ll need the following data in the index:

• Name – used as the main search criteria field and main results link text (note: as per the index data we’ve already seen above we know the super hero name is already stored as the service transaction businessName property so we can use that… right? See the special section on “Business name data” below)
• Super powers – used in the advanced search criteria
• Workflow status – used in the advanced search criteria (note: this is already stored as the service transaction status property and this we can use)
• Includes a cape – used in the advanced search criteria

Business name data

The business name is an interesting one. As mentioned above we already have the super hero name indexed under the businessName property. And we know the "Name" attribute and "businessName" property are kept in sync during registration since we configured it that way in lesson 4. So on the face of it there’s no downside to only indexing the implicit "businessName" property data.

Applying a bit of registry experience however, we’ll see having both details will be beneficial, particularly when it comes to adding functionality where you can change the super hero’s name. When performing a maintain service you don’t really want to change the official business name of the registry data until the service is officially applied and live on the register. Up to that point, you want the "Name" to be the new name (so it can be searched when searches include in-flight services) but the official business name to remain that of the live registry data. This is when indexing them both would become required, so given we’re working towards being efficient registry configurators we’ll go with this approach for now. We’ll focus on the "businessName" property more as being present for the display value in the results, and the "Name" attribute more as the searchable field (although it might also be displayed in results). So we’ll still include the "Name" attribute for the super hero name data.

Add domain data configuration

So that means we need to add the "Name", "SuperPowers" and "CapeYn" domain attributes to the search index configuration. Again, note that the status is still implicitly added as part of the search index <serviceTransaction> element for service transaction data, and doesn’t need to be configured explicitly since the default configuration is all that’s needed, so we’ll leave it out. That results in the search index configuration now having it’s <domain> element as follows.

<domain name="hero">
  <attribute name="Name"/>
  <attribute name="SuperPowers"/>
  <attribute name="CapeYn" dataType="boolean"/>
</domain>

Now perform a re-index again and query the data in the System Administration console to confirm can can now see these additional fields appearing in the “heroSearch” index data.

Configure the search form

Now we have data in the search index we just need to configure our form so we can query it and display the results. Exciting times! 🙂 So back to the "heroSearch" form, let’s start by adding a <search:search-form> tag to specify some high-level search options for our search form. The main bit of information is the index-name option to link this search to our "heroSearch" search index, but we also have to specify sort-options and sort-text-prefix options so we’ll add those in too. Now the requirements didn’t say what order we should return our results in, so for now let’s just use the "id" field as the value for that which will really mean no order at all other than it being a known order (i.e. the results won’t change if you do multiple searches with the same criteria). This should result in configuration similar to the following.

<search:search-form index-name="heroSearch" sort-options="id" sort-text-prefix="heroSearch.sortOptions"/>

Next we’ll look for a configuration component to help us out with the rest of the form, so type "<component:search" then hit Ctrl+Space and see what comes up. Quite a bit! The SearchForm components sounds promising so let’s select that. Wow, now we’re talking! Look at all those beautifully named slots for us to fill in. Lovely! 😉

Search form component

Set the required "resultsDomain" attribute (hopefully the value to use springs to mind?). One thing that’s important to note though is that since the search will only involve data from the search index, the domain in this case is the document name from the index data. This can be the same as the domain name in the database data, but may not be if an "alias" was used for the root <domain> element in the search index configuration. You can also check the value you’re after by inspecting the "rootDomainName" field in the index data, which you can see in Fig 6.3 above.

The requirements refer to a search bar relating to a single "Name" criteria, and three advanced criteria attributes, so we’ll start by adding these to the relevant slots. We’ll use the <component:SearchCriteriaAttribute> component for each of the criteria attributes. Have a go at adding these with an appropriate "textKeyPrefix" value and using the following attributes for each:

• Name – “Name” attribute
• Super powers – “SuperPowers” attribute plus the relevant data constraint
• Workflow status – “status” attribute plus a new “heroSearchWorkflowStatus” dataconstraint that mimics the “svcTxnStatusSearch” data constraint but with only the two relevant “Pending” and “Completed” entries we require (tip: they’re the resolved textual values, they actually relate to the “pending” and “activated” service transaction status, hence data constraint code, values)
• Includes a cape – “CapeYn” attribute plus the relevant data constraint

In terms of the results it only mentions a link with the hero’s name (the business name in our case) which we can add using the <component:SearchHeader> component and since we want it to navigate to our view service we’ll use the "targetServiceMode" parameter with a value of "View" so our default view service from our data set is used.

Once all this is done you should end up with your search form looking something like the configuration shown below, along with appropriate supporting localised text and data constraint entries.

<record shortCode="heroSearch" domain="HeroSearch">
  <search:search-form index-name="heroSearch" sort-options="id" sort-text-prefix="heroSearch.sortOptions"/>
  <component:SearchForm resultDomain="Hero">
    <slot:searchBar>
      <component:SearchCriteriaAttribute attribute="Name" textKeyPrefix="heroSearch.criteria.name"/>
    </slot:searchBar>
    <slot:advancedCriteria>
      <component:SearchCriteriaAttribute attribute="SuperPowers" dataConstraint="heroPowers" textKeyPrefix="heroSearch.criteria.superPowers"/>
      <component:SearchCriteriaAttribute attribute="status" dataConstraint="heroSearchWorkflowStatus" textKeyPrefix="heroSearch.criteria.workflowStatus"/>
      <component:SearchCriteriaAttribute attribute="CapeYn" dataConstraint="yesNo" textKeyPrefix="heroSearch.criteria.capeYn"/>
    </slot:advancedCriteria>
    <slot:searchResult>
      <component:SearchHeader targetServiceMode="View"/>
    </slot:searchResult>
  </component:SearchForm>
</record>

OK, now let’s load up our sparkling new Search Super Heroes service via the main menu and see what we get. Wow, would you look at that, a pretty much fully functional search – impressive! You should see something similar to Fig 6.4 below. Click the Search button and see if you get some results.

Searching

Assuming you get some search results, you’d probably be feeling pretty good about things at this point. If you’re one of the ones who didn’t get any search results however, you might think otherwise. Why would some of you get results and others not? Well, in this case it depends on the values you selected when you registered your super heroes, specifically the super powers. Super heroes who have just one super power will be returned in the search results, but those gifted multi-powered types will not. (Go on, try it and see). Why? Well, it all comes back to our second lesson where we opted for the super easy option of making the "heroPowers" data constraint support multiple values. Remember? No, probably not 😉

Multiple super powers

• Fix up the search to include heroes with any number of super powers when no specific super powers are specified in the advanced search criteria

Take a look at the search index data again and look at the format of the "SuperPowers" fields in the results. Specifically for super heroes who have multiple powers and notice how they’re stored as a CSV e.g. "fighting,senses". To support attributes that are backed by data constraints that can support multiple values within the index we simply need to add a corresponding "supportsMultipleValues" value on the attribute within the search index configuration. Update the "SuperPowers" attribute element in the search index configuration to include this value, resulting in the configuration shown below.

<attribute name="SuperPowers" supportsMultipleValues="true"/>

Re-index again and notice how the values are now split into an array against the "SuperPowers" field in the search index results data? Also do a search using the "Super Hero Search" via the application and notice that all the super heroes are being returned now, even those with more than one power? Great stuff.

Implicit Workflow Status

• Hide the Workflow Status field for external users, defaulting it to only return registered super heroes

For non-internal users the Workflow Status field should not be shown, and it should be nailed to "Completed" (remember, this relates to a value of "activated" in terms of the raw data value). To do this we’ll use a <set-property> tag and a <set-attribute-value> tag within the "status" advanced search criteria attribute, both with a condition of when the user does not have a security permission role of "PERM_U003_INTERNAL_USER". Have a go at adding these yourself before comparing to the configuration shown below (no peeking ?).

<component:SearchCriteriaAttribute attribute="status" dataConstraint="heroSearchWorkflowStatus" textKeyPrefix="heroSearch.criteria.workflowStatus">
  <platform:set-attribute-value value="activated" when-user-does-not-have-permission="PERM_U003_INTERNAL_USER"/>
  <platform:set-property visible="false" when-user-does-not-have-permission="PERM_U003_INTERNAL_USER"/>
</component:SearchCriteriaAttribute>

Caped crusaders

• Ensure selecting the “No” option still returns super heroes who have selected “No” for not having a costume/uniform at all, thus the cape field doesn’t apply, versus those that do have a costume/uniform but have explicitly selected “No” for including a cape

Try this out for yourself using the application first to be sure you understand the problem. Again, look at the search index data via the System Administration console Search Index "Query" tab to see the difference between a super hero who doesn’t even have a uniform versus one that just doesn’t include a cape. first. The "CapeYn" field doesn’t even exist in the first case, versus it having a value of "false" in the latter. Now compare the resulting raw search query when you select "No" for the "Caped crusaders" advanced search criteria attribute on the form.

A trick for finding the raw query used in a search is to select the System Log tab in the System Administration console, then perform the search in the application (in this case with the "No" option selected for just the "Caped Crusader" field in the advanced criteria. Now go back to the System Administration console and select the entry relating to the search, as shown in Fig 6.5 below.

Copy the raw query and paste it into the query section of the "Query (Raw)" tab within the Search Index section in the System Administration console and execute to see what is returned. Change the "CapeYn" value and remove it from the query and see how this changes what results are returned.

Really what we want is either no "CapeYn" field at all or one with a value of "false" returned when we select "No" for the "Caped Crusaders" field. Test this theory using a "CapeYn = false or CapeYn is null" query back on the simpler "Query" tab. Looks good? OK, so how do we configure that condition when the "No" option is selected? Why we simply use the "equalsOrIsNull" search operator of course ?

Go ahead and add some <set-key-value> tag to the appropriate option on the SearchCriteriaAttribute component to set the operator to "equalsOrIsNull" and "equals" when "Yes" is selected. While we’re at it we should also make it a drop-down select field rather than a radio button so the value can be cleared if not longer wanted. Note that a checkbox would be another option, but that would only support a "yes" and "any value" options. After making these changes the advanced criteria configuration for this field should then look like the one below.

<component:SearchCriteriaAttribute attribute="CapeYn" dataConstraint="yesNo" textKeyPrefix="heroSearch.criteria.capeYn" widget="attribute-select">
  <platform:set-key-value scope="search-execute" key="search-operator" value="equalsOrIsNull"/>
  <platform:set-key-value scope="search-execute" key="search-operator" value="equals" when-attribute-value-equals="Y"/>
</component:SearchCriteriaAttribute>

Check the application now gives the expected results when selecting any value for the "Caped Crusaders" advanced criteria field.

Name text matching

We’re really getting there now! All our advanced criteria are done, but you might’ve noticed that if you enter a value in the main Name search field it only returns heroes whos name exactly matches the criteria value. No leeway for incorrect character case or anything i.e. a search for "super configurator" won’t return the "Super Configurator" super hero. This seems pretty poor so should be addressed.

We can do this by simply changing the data type of the "Name" field within the search index data to a "text" type. How do we do this? Using the… wait for it… dataType attribute within the search configuration item ? The default data type for strings is "keyword" which requires exact matches, which is often what we want when they relate to data constraint -backed values where the raw value is fixed by the data constraint, despite the displayed value looking textual. Go ahead and do that so you end up with the configuration shown below.

<attribute name="Name" dataType="text"/>

Index mappings

Again, since we’ve updated the search index configuration we’ll have to perform a re-index to ensure the changes are picked up. Before we do that however, let’s take a look at the Elasticsearch mappings for our heroSearch index to see how this update might change them. For this we will once again jump into our trusty System Administration console, this time the "Mapping" tab in the Search Index area. Select it then look for the entry for the "businessName" property and note the "keyword" data type as highlighted in Fig 6.6 below.

OK, now do the re-index then re-check to see the"Name" now has a "text" data type within the search index mapping.

Also now that we’ve re-indexed check you now get results when searching for hero names without the correct character case.

More complex transformations

So the "text" data type allows for more flexible matching i.e. case-insensitive. But we’ll round out this lesson touching on another approach that can achieve this as well as providing even more flexibility. Whilst the "text" data type worked for case insensitive matches, what if we wanted more complicated matches? Like individual word matches, or stripping out certain characters and things like that?

Our example will focus on word matches. Before we proceed, confirm the issue by searching for one word of a super hero with a two or more word name doesn’t return them… go on, try it and see)

In these cases we have the <index-text> and <search-text> tags that can be used to transform the indexed and searched values in the same arbitrary ways. The beauty of these tags is they still add the original value by default (so it can be used if displayed in the results) but also add additional fields for different versions of the value(s) with names hidden behind these tags. In other words, you don’t need to know what these additional index fields are even called, as you’d only see them if looking at the Elasticsearch data or the raw queries searches using the <search-text> tag produce.

So let’s try this approach now. We’ll update the "Name" configuration in the search index to use an <index-text> tag, and similarly with the "Name" search criteria attribute in the search form with a <search-text> tag. Use the "case-insensitive" and inline "tokenise" options to achieve successful case-insensitive and single worded queries. You should have a go at doing this yourself to see if it you can manage it using the documentation available before checking against the relevant configuration snippets below.

Search index configuration:

<search:index-text name="Name" case-insensitive="true" tokenise="true"/>

Search form configuration:

<component:SearchCriteriaAttribute attribute="Name" textKeyPrefix="heroSearch.criteria.name">
  <search:search-text tokenise="true" case-insensitive="true"/>
</component:SearchCriteriaAttribute>

Use the Search Index “Resolved config” tab in the System Administration console to see what the resolved search index configuration looks like (Fig 6.7 below) and Service Form for the same with the search form.

Do a final re-index before checking you can now search using these more lenient name query values in the Search Super Hero form.

Wrap up

Wow, we made it! Search is surprisingly complex for a concept we’re all very comfortable with and take for granted given our regular usage of search engines and search functionality for accessing content on myriad websites and platforms. Here’s what we covered in this lesson:

• Adding a search business service
• The dual notion of searching as a concept: indexing the data, and searching it
• Special consideration of business name data
• Search components to help build search forms efficiently
• The numerous and impressive System Administration console tools to help re-index and configure search indexes and search forms
• Delved a little into the workings of Elasticsearch in terms of raw queries, data types and mappings
• Looked at transforming both indexed and searched values in more complicated ways to ensure positive matches