Sitecore Cognitive Services

February 22, 2017

For a few years now I've been working to raise awareness how machine learning could be used with Sitecore to make web applications a much more powerful experience. Microsoft has recently made my work a whole lot easier.


In the not too distant past, Microsoft released a number of web services using machine learning to the general public under the banner of Cognitive Services. There's quite a few and some even have implemented C# client libraries that are available on Nuget under the Project Oxford. Most don't have client libraries yet (it looks like they're still under active development) but there's plenty of value with what's already available.

Each API appears to be managed separately but here's a list of what Microsoft is putting out there:

  1. Computer Vision
  2. Content Moderator
  3. Emotion Recognition
  4. Facial Analysis
  5. Video Analysis
  6. Speech Analysis
  7. Custom Speech Service (Custom Recognition Intelligence Service - CRIS)
  8. Speaker Recognition
  9. Language Understanding Intelligent Service (LUIS)
  10. Linguistic Analysis
  11. Text Analytics
  12. Text Translator
  13. Web Language Model (WebLM)
  14. Academic Knowledge
  15. Entity Linking
  16. Knowledge Exploration
  17. Q and A Maker
  18. Recommendations
  19. Bing Autosuggest
  20. Bing Image Search
  21. Bing News Search
  22. Bing Spell Check
  23. Bing Video Search
  24. Bing Web Search


I want to give a shout out to some other Sitecore developers who's dabbling motivated me to get started. Tom Dudfield, Pavel Veller and last but certainly not least Goran Halvarsson. Long live the community.

Sitecore Cognitive Services

Now without further adieu, I'd like to introduce you to Sitecore Cognitive Services. An integration between Sitecore and Microsoft's Cognitive Services.


I would like to mention that I have not yet implemented every API, only about half. So if there's something specific you want/need and it's not yet working, you may need to build it in (sorry). If you do, feel free to fork my repo and send me a pull request.

Also it's not a full fledged module with Nuget releases for every version. I've been working against a specific version of Sitecore (8.2 rev. 160729) so if you want to try to get it to work against another version there may be a little bit (but not a lot) of configuration you'll need to undertake.

The good news

In order to get further adoption of the Cognitive Services I've developed a foundation that uses Sitecore's Content Search to analyze text and images with a number of cognitive API's. I'm running the analyses as computed fields so content is indexed as it's created and will store the analysis in the index. Sitecore then moves from being a stodgy content management system (CMS) to being an intelligent content learning system (CLS). Wink wink.

Since the backbone of this is a search index, you can at any time, go to the developer tab and reindex everything. It may take a while (images are actually being uploaded) but thereafter, everything will be reindexed as it is created/uploaded. You can alternatively spot reindex each individual item and it's children from the top nav.

scs reindex

Accessing the Analysis

To be able to access the analysis, there is a new top level 'Cognitive' navigation element which are slightly different if you're on a content item, media item or media folder

Content Item

scs main nav

Media Item

scs main nav media

The ribbon buttons will pop open modals that allow you to manage the content.

Images, for example, allow you to set the alt tag based on the cognitive analysis description

scs set alt

or you can try to set the alt tag on all descendents. Since you'll be unable to check all the descriptions before you do so, there's a slider that allows you to require a certain level of confidence from the analysis before the alt tag can be set.

scs set alt folder

On media folders and content items, you can recursively reanlyze a certain section of the tree.

scs reanalyze

Text Analysis

There's a number of ways that text can be analyzed. I've broken up each API into it's own tab and each field into it's own section.

Link Analysis

This will find parts of a body of text that are linked.

scs link analysis

Key Phrase Analysis

This finds the important phrases in a body of text.

scs key phrase analysis

Linguistic Analysis

This will break down the text into it's component parts and tell you what the nouns, pronouns, adjectives etc are.

csc linguistic analysis

Sentiment Analysis

This will determine whether the text is pleasant or hostile.

csc sentiment analysis

Language Analysis

This will determine the language of the current text. This may be somewhat less useful but still.

scs language analysis

Image Analysis

Image analysis also breaks up the API's into tabs but each tab may have additional hover details associated with it.

Visual Analysis

This will determine what's in an image and tag it. This includes whether an image is adult/racy and color elements.

scs visual 1

a visual






Emotional Analysis

The emotional anlysis will determine the emotion of the people in the image.

a emotion

Facial Analysis

The facial analysis will provide information about facial features.

a facial

Text Analysis

Text analysis will tell you about the words in an image. This is actually very powerful in that it enables searching on text in an image.

scs textual 2

scs textual 3scs textual 1

Image Search

It may not be immediately obvious the implications of having all your content analyzed and indexed (it wasn't for me either) but after you start chewing on the possibilities, you start to realize that you're unlocking the content in a whole new way. The media library was always just a kludgy block of binary data but not anymore. With this, you can now see into the images and what they actually represent. This now makes searching for an image a whole lot more pleasant.

To demonstrate the power, I've added an image search into the Rich Text Editor:

scs rte button

What you get is a fairly intuitive search interface that populates with all the images that have been indexed. I'm using the Launch Sitecore site to provide me with images and content.

scs image search 1

filter by 'sitecore'

scs image search 3

filter by 'smiling'

scs image search 2

How to Git started (yuck, yuck, yuck)

So as I mentioned, this isn't a module quite yet. I did try my best to follow the Helix guidlines and design the project in as modular a way as possible. There's three projects, the core library, test library and launch demo examples. The setup steps are as follows:

  • Environment Prep
    • Get a set of API Keys from Microsoft
    • SIM Install a version of Sitecore 8.2 rev. 160729 and the Launch Sitecore package (or use your own site if you're comfortable with it)
    • Download the source code from GitHub
    • Enter your API Keys into the SitecoreCognitiveServices.config file. 
    • Update the publishing profile on the core library project with the path to your Sitecore webroot (so the publish goes to your local web root)
    • Build the projects (to ensure all the Nuget packages get downloaded because this doesn't always work right away)
    • Publish the core and launch demo projects onto your website
  • Go into the webroot and modify the web.config (I couldn't automate this part)
    • add a node to the compilation assembles: <add assembly="System.Runtime, Version=, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
    • change Newtonsoft.Json version from to (the project oxford clients are using the new version)
  • Unicorn sync
    • manually (I know, I know) copy the contents of <sourceroot>/SitecoreCognitiveServices/serialization folder into the <webroot>/data/Unicorn folder
    • browse to http://<domain>/unicorn.aspx and sync the folders
  • additionally you may want to browse to these files and force refresh them (browsers cache them)
    • http://<domain>/sitecore/shell/Themes/Standard/Default/Content%20Manager.css
    • http://<domain>/sitecore/shell/Controls/Rich%20Text%20Editor/RichText%20Commands.js
  • log into Sitecore and reindex the cognitive_master_index
  • start viewing the analysis and building out more functionality!

 It may seem daunting and it definitely pushed my skillset to a higher level but it was totally worth every minute. Hopefully once you start to see what's possible with the limited set of API's I've implemented, you'll be motivated to contribute and help build a really great module.

As an easter egg, anyone who gets this started running on the launch demo site can get a sample of a giphy moderator. For anyone who's used it, you've probably been burned by it when it posted something wholly inappropriate. To check it out browse to http://<domain>/CognitiveLaunchDemo/CognitiveLaunch/Moderator. You can either just click the 'Get Image' button or add some text for the random search. It will then put the image in one of three columns: Acceptable, Racy or Adult. It's a simple example of how to handle user generated image moderation and actually pretty funny at times.

Future Features

I'm hoping to get all the API's connected and working and then begin developing more high level features with content selecting renderings and custom rules.

Closing Remarks

Hopefully these types of applications won't seem too creepy and more importantly, hopefully you won't create anything creepy. These types of tools are powerful but nothing to be feared. You should find that these services are of great benefit to you and your clients. They're less like HAL 900 and more like Ziggy!

 quantum leap 2