Sitecore Cognitive Services: Intelligent Search

September 16, 2019

For a few years now I've been pushing the idea of using machine learning to rethink how we build websites. Today I've got more good news. In yet another example, we can now get a lot closer to building a google like search experience for all the Sitecore sites we build. The sample project I've put together to demonstrate this is called "Intelligent Search" and is built on top of the Sitecore Cognitive Services "Core" and uses Microsoft's LUIS to parse input questions for a search form to answer common domain specific questions as well as engage in conversations with the user. 

For the amount of value that this provides, the work required comparatively is very light. Just starting with answering the top three questions for any given business should immediately mean higher conversion rates on your goals. There's so many occurrences where asking any question to a site search offers you nothing. Not a product link, not any business information. Nada. But today is a new day. We have powerful new tools that can improve the type of experiences we can provide our audiences. 

Here's some videos to get you started:

Language Parsing

This all hinges on the ability to parse language but what does that even mean. Well when someone says "I'm looking for two tickets to the Jay-Z concert in Chicago" we have a lot of information that we can use. The hardest part was always breaking up those statements in a general way to extract what words matter. This is where LUIS comes in (but you could do something simliar with IBM, Google, Amazon among others). What it does is allows you to provide sample (training) phrases and then mark sections of those phrases as terms (variables). This allows you it to learn what parts of phrases should be extracted and provided to you as a developer. The Intelligent Search module has some sample training data that breaks down the phrases into parts of speech (verb, noun, adjective etc). Some of the sample intents and form conversations that it handles are:

Search Intents

  • Events
  • Information
  • Locations
  • Purchases

Single Field Forms

  • Contact Us
  • Registration

Installation

The installation goes like this:

  • Install a local stock Sitecore installation
  • download the source code (Core and IntelligentSearch)
  • Setup the solution
  • publish the solution to your local Sitecore environment
  • run a unicorn sync
  • log into Sitecore and go to /sitecore/system/modules/Sitecore Cognitive Services/Intelligent Search/Search Profiles/Intelligent Search open the content tab and save the Application Backup contents to intelligentsearch.json on your desktop
  • Create an Azure Portal account, login and then create a Luis resource and a Bing Spell Check v7 resource (copy a key for each for use later) 
  • Using your Azure portal account login into LUIS and go to your account settings (click on top right user icon -> settings) and copy the Starter_Key (authoring key)
  • At the top click 'My apps', click 'Import new app' and select the intelligentsearch.json file you created. Provide an optional name: 'Intelligent Search' and click 'Done'
  • Click on the newly created app and at the top click on 'Manage', then on the left navigation select 'Application Information' and copy out the application id (to be saved in Sitecore)
  • Now in the left nav go to 'Azure Resources' and click 'Add prediction resource' (this will ask you to select the Luis resource from your Azure account)
  • At the top right click on 'Train' and when it's finished click on 'Publish' and set the Sentiment analysis and Bing Spell check to 'On' for both staging and production
  • go back to Sitecore and browse to /sitecore/system/Modules/Sitecore Cognitive Services/APIs/MSSDK and under the Luis and Bing Spell Check sections fill in the keys and endpoint fields for your Azure resources and save
  • browse to /sitecore/system/modules/Sitecore Cognitive Services/Intelligent Search/Search Profiles/Intelligent Search open the content tab and save the Application Id 
  • you can now switch to the 'Query App' tab and check that it works
  • add a hosts file entry for intelligentsearch.local
  • browse to intelligentsearch.local and ask a question

Most of the setup is actually within Luis and Azure portal but once you've gone through that process and are familiar using the endpoints becomes almost trivial. The most valuable part of all of this is really the training data. Once you start developing your own applications, the training data is what you'll spend the most time curating for your unique business needs or client needs.