Sitecore Cognitive Services: Modules

I'm excited to announce the release of the cognitive services I've been working on for the last year as modules. I present to you the Cognitive Core, the Image Search and Ole Chat. Of all the ideas it generated, being able to polish up two into pretty practical modules was really satisftying. They'll eventually be moved to a space on the Marketplace but I'm still working on that. 

To be able to use these modules you'll need to be using Sitecore 8.2. It could be backported with sufficient effort but, with minor changes to the search config, it will work on 9. To get started you'll need to do a few non-sitecorey things. 

Getting Azure Resources

This steps assumes you have an Azure Portal account. If you don't it's free to do so and the resources required have free versions. As the name implies, these modules rely on Microsoft Cognitive Services from Azure. Each module requires a different set of services so I'll explain what you'll need to do generally. 

Go to Azure Portal and click on 'Create a resource'.

Click on the 'AI Cognitive Services' link in the nav.

The next nav will have a link for some of the services but you can click to see all or search for the right one. In this case I'll show you how to create a Face service by clicking on the 'Face API' link.

Next you'll want to name the resource something similar to the API's name itself. You'll also need to select a pricing tier (free or paid) and resource group. If you don't have a resource group then create one and name it something like 'CognitiveServices'.

After it's been created click on the resource in the dashboard and go to the 'Overview' section. There you'll see the 'Endpoint' link and a link to 'Manage keys'. You'll need both values for each resource when setting up a module.

If you click in to see your keys, you can click to copy it.

Installing the Core Framework

The modules rely on a core package that provides the framework (binaries, shared views, routing, sitecore modules items for storing api keys). This is so you can focus more on building interesting features and less on plumbing.

To install the framework, install the Cognitive Core package. Once it's installed... that's it!  

If you want create you're own features, you could get the source code from Github (mostly to see how the others do it) or you can just inject or create any of the services from the SitecoreCognitiveServices.Foundation.SCSDK.Services namespace into your own projects (sorry haven't got a nuget feed up but feel free to contribute that if you can). The services use the keys that are saved on the item at /sitecore/system/Modules/Sitecore Cognitive Services/APIs/MSSDK. 

Cognitive Image Search

What It Can Do

The image search indexes all the media library items with a special set of computed fields that store information after it's been analyzed . With it, you can do a faceted search on images based on color, size, faces, gender, emotion, glasses and mature content.

You can access the search through a new 'Cognitive Image' field:

or through a button on Rich Text Editor with a 'Default' or 'Full' profile:

Or since it's a regular index, you could build your own interface.

Aside from the image search, there is another feature worth mentioning; Set Alt Tags (on one or many). When you select a folder or image in the media library, you'll see another button in the 'Cognitive' tab in the ribbon that will allow you to set the alt tags based on its analysis. On folders there is a threshold setting to allow you to say how confident the analysis must be to use the generated description. 

One last note is that on the image search module item there is a checkbox field that allows you to say whether any newly uploaded images should automatically be analyzed or not. Just one of those nice features that comes in handy.

How To Set It Up

In Azure Portal, you'll need to create a Face API resource and a Computer Vision API resource (as detailed earlier). Once they've been created and you have the endpoint and a key handy, then install the Image Search module. 

After it's been installed, there's a setup page on the new sitecore modules folder item that's been created. If you select it, in the right tab you'll see a form to add your Azure Cognitive Services API Keys and save them. The search comes with search config files which are enabled on setup so you need to choose between Lucene, Solr or skipping it to deal with it yourself later. The files will end up under the \<webroot>\Website\App_Config\Include\SitecoreCognitiveServices folder.

Now go to your media library and click on any folder (including the root) where you want to analyze the images. In the top ribbon go to the 'Cognitive' tab and click on 'Analyze All Descendents'. When the modal pops up, click 'Analyze Descendents'. The analysis for each image is stored under the bucket here: /sitecore/system/Modules/Sitecore Cognitive Services/Image Search/Image Analysis. What you've just done is analyzed the images, stored that value and made it's analysis searchable from the installed image search.

Ole Chat

What It Can Do

Ole Chat is a chatbot built on LUIS and allows you to manage tasks in Sitecore through text. It's uses it's ability to differentiate phrases and understand context. Its features are limited but it's just a start and it allows you to create your own business related tasks. A chatbot is a fairly narrow ability but if you consider how the underlying APIs work, you can create a lot of powerful tools that can automatically trigger actions based on what people are saying such as responding or alerting based on social media posts, incoming emails or form submissions, even being able to just understand the deeper meaning of what's a page but I digress. 

Ole is available either through a button on the LaunchPad:

Or you can find it in the Content Editor Ribbon:

How To Set It Up

In Azure Portal, you'll need to create a LUIS API resource and a Text Analytics API resource (as detailed earlier). Once they've been created and you have the endpoint and a key handy, then install the Ole Chat module. Now browse to the Ole module item, enter your keys into the setup form and click submit. This will connect to luis, see if it needs to restore Ole from a backup or just sync the Application Id.

 Once it's successfully been setup, you can open it and start chatting with Ole.

Extending Ole

If you want to add more intents to Ole, you'll need to log into LUIS with your Azure Portal account and add new intents to the Ole Application. Then you'll need to create a class in your own project (you can use the built in intents as an example) and register it using dependency injection (you can use the built in intent registrations as an example). Depending on how much you want to extend the interface, you'll need to update the javascript, conversation service, or even the conversation parameter.

Cognitive Core

To provide you with some insight into the core, here's a high level diagram of how the it works and how I'm expecting to expand it to work with more API sources. Each API source is broken up into it's own project and follows the pattern of implementing repository classes for any of it's APIs. Each has it's own client to handle passing data back and forth but that's all it does. It should be purely a .NET project with no sense of Sitecore or anything else. This also has the benefit of making it portable. The SCSDK is where all the repository classes are wrapped in service classes and logging, ioc registrations and retries are bound together into a cohesive surface for anyone to build features against. 

I've also got one for the Image Search but not for Ole yet (sorry, these things do take time). 

Wrap Up

I hope what you take away from this is not that it just a great tool but a foundation to continue building an underlying intelligence system for Sitecore by fundamentally rethinking how users interact with it. We are living in a time where intelligence services are cheap and plentiful.

Technology moves pretty fast, if you don't stop and look around once in a while, you could miss it.