Sitecore Cognitive Services
Recent Posts
Recent Comments
-Chris That’s some amazing stuff Mark! Hope I can clear up some time to play around with it and contribute!
Archives
Tags
For a few years now I've been working to raise awareness how machine learning could be used with Sitecore to make web applications a much more powerful experience. Microsoft has recently made my work a whole lot easier.
Background
In the not too distant past, Microsoft released a number of web services using machine learning to the general public under the banner of Cognitive Services. There's quite a few and some even have implemented C# client libraries that are available on Nuget under the Project Oxford. Most don't have client libraries yet (it looks like they're still under active development) but there's plenty of value with what's already available.
Each API appears to be managed separately but here's a list of what Microsoft is putting out there:
- Computer Vision
- Content Moderator
- Emotion Recognition
- Facial Analysis
- Video Analysis
- Speech Analysis
- Custom Speech Service (Custom Recognition Intelligence Service - CRIS)
- Speaker Recognition
- Language Understanding Intelligent Service (LUIS)
- Linguistic Analysis
- Text Analytics
- Text Translator
- Web Language Model (WebLM)
- Academic Knowledge
- Entity Linking
- Knowledge Exploration
- Q and A Maker
- Recommendations
- Bing Autosuggest
- Bing Image Search
- Bing News Search
- Bing Spell Check
- Bing Video Search
- Bing Web Search
RAHSPEK
I want to give a shout out to some other Sitecore developers who's dabbling motivated me to get started. Tom Dudfield, Pavel Veller and last but certainly not least Goran Halvarsson. Long live the community.
Sitecore Cognitive Services
Now without further adieu, I'd like to introduce you to Sitecore Cognitive Services. An integration between Sitecore and Microsoft's Cognitive Services.
Caveats
I would like to mention that I have not yet implemented every API, only about half. So if there's something specific you want/need and it's not yet working, you may need to build it in (sorry). If you do, feel free to fork my repo and send me a pull request.
Also it's not a full fledged module with Nuget releases for every version. I've been working against a specific version of Sitecore (8.2 rev. 160729) so if you want to try to get it to work against another version there may be a little bit (but not a lot) of configuration you'll need to undertake.
The good news
In order to get further adoption of the Cognitive Services I've developed a foundation that uses Sitecore's Content Search to analyze text and images with a number of cognitive API's. I'm running the analyses as computed fields so content is indexed as it's created and will store the analysis in the index. Sitecore then moves from being a stodgy content management system (CMS) to being an intelligent content learning system (CLS). Wink wink.
Since the backbone of this is a search index, you can at any time, go to the developer tab and reindex everything. It may take a while (images are actually being uploaded) but thereafter, everything will be reindexed as it is created/uploaded. You can alternatively spot reindex each individual item and it's children from the top nav.
Accessing the Analysis
To be able to access the analysis, there is a new top level 'Cognitive' navigation element which are slightly different if you're on a content item, media item or media folder
Content Item
Media Item
The ribbon buttons will pop open modals that allow you to manage the content.
Images, for example, allow you to set the alt tag based on the cognitive analysis description
or you can try to set the alt tag on all descendents. Since you'll be unable to check all the descriptions before you do so, there's a slider that allows you to require a certain level of confidence from the analysis before the alt tag can be set.
On media folders and content items, you can recursively reanlyze a certain section of the tree.
Text Analysis
There's a number of ways that text can be analyzed. I've broken up each API into it's own tab and each field into it's own section.
Link Analysis
This will find parts of a body of text that are linked.
Key Phrase Analysis
This finds the important phrases in a body of text.
Linguistic Analysis
This will break down the text into it's component parts and tell you what the nouns, pronouns, adjectives etc are.
Sentiment Analysis
This will determine whether the text is pleasant or hostile.
Language Analysis
This will determine the language of the current text. This may be somewhat less useful but still.
Image Analysis
Image analysis also breaks up the API's into tabs but each tab may have additional hover details associated with it.
Visual Analysis
This will determine what's in an image and tag it. This includes whether an image is adult/racy and color elements.
Emotional Analysis
The emotional anlysis will determine the emotion of the people in the image.
Facial Analysis
The facial analysis will provide information about facial features.
Text Analysis
Text analysis will tell you about the words in an image. This is actually very powerful in that it enables searching on text in an image.
Image Search
It may not be immediately obvious the implications of having all your content analyzed and indexed (it wasn't for me either) but after you start chewing on the possibilities, you start to realize that you're unlocking the content in a whole new way. The media library was always just a kludgy block of binary data but not anymore. With this, you can now see into the images and what they actually represent. This now makes searching for an image a whole lot more pleasant.
To demonstrate the power, I've added an image search into the Rich Text Editor:
What you get is a fairly intuitive search interface that populates with all the images that have been indexed. I'm using the Launch Sitecore site to provide me with images and content.
filter by 'sitecore'
filter by 'smiling'
How to Git started (yuck, yuck, yuck)
So as I mentioned, this isn't a module quite yet. I did try my best to follow the Helix guidlines and design the project in as modular a way as possible. There's three projects, the core library, test library and launch demo examples. The setup steps are as follows:
- Environment Prep
- Get a set of API Keys from Microsoft
- SIM Install a version of Sitecore 8.2 rev. 160729 and the Launch Sitecore package (or use your own site if you're comfortable with it)
- Download the source code from GitHub
- Enter your API Keys into the SitecoreCognitiveServices.config file.
- Update the publishing profile on the core library project with the path to your Sitecore webroot (so the publish goes to your local web root)
- Build the projects (to ensure all the Nuget packages get downloaded because this doesn't always work right away)
- Publish the core and launch demo projects onto your website
- Go into the webroot and modify the web.config (I couldn't automate this part)
- add a node to the compilation assembles: <add assembly="System.Runtime, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
- change Newtonsoft.Json version from 6.0.0.0 to 9.0.0.0 (the project oxford clients are using the new version)
- Unicorn sync
- manually (I know, I know) copy the contents of <sourceroot>/SitecoreCognitiveServices/serialization folder into the <webroot>/data/Unicorn folder
- browse to http://<domain>/unicorn.aspx and sync the folders
- additionally you may want to browse to these files and force refresh them (browsers cache them)
- http://<domain>/sitecore/shell/Themes/Standard/Default/Content%20Manager.css
- http://<domain>/sitecore/shell/Controls/Rich%20Text%20Editor/RichText%20Commands.js
- log into Sitecore and reindex the cognitive_master_index
- start viewing the analysis and building out more functionality!
It may seem daunting and it definitely pushed my skillset to a higher level but it was totally worth every minute. Hopefully once you start to see what's possible with the limited set of API's I've implemented, you'll be motivated to contribute and help build a really great module.
As an easter egg, anyone who gets this started running on the launch demo site can get a sample of a giphy moderator. For anyone who's used it, you've probably been burned by it when it posted something wholly inappropriate. To check it out browse to http://<domain>/CognitiveLaunchDemo/CognitiveLaunch/Moderator. You can either just click the 'Get Image' button or add some text for the random search. It will then put the image in one of three columns: Acceptable, Racy or Adult. It's a simple example of how to handle user generated image moderation and actually pretty funny at times.
Future Features
I'm hoping to get all the API's connected and working and then begin developing more high level features with content selecting renderings and custom rules.
Closing Remarks
Hopefully these types of applications won't seem too creepy and more importantly, hopefully you won't create anything creepy. These types of tools are powerful but nothing to be feared. You should find that these services are of great benefit to you and your clients. They're less like HAL 900 and more like Ziggy!
