I began working with web testing years ago with the MS Test framework when I got a version of Visual Studio that allowed it's use. I started small and began building simple ping tests. Over time it developed into a tool would run sets of tests daily and fed the results into a blog that I could read every morning. It tought me a lot but eventually I hit too many limitations. License cost being the most obvious but also I couldn't create a custom interface for it. I had to work within Visual Studio and the UI was too blunt. This forced me to finally bite the bullet and start converting my system to NUnit and Watin.
I originally only expected to build a simple web app to run my web tests but it quickly became something much more comprehensive. I decided to incorporate both web(integration) tests and unit tests. Up until now, unit tests were something I've avoided because they were too time/code intensive. For any given piece of code you'll likely need several tests. On the other hand, you can get quite of a bit of mileage from a single web test. Now web testing is not a replacement for unit tests. Really they're complementary. Unit tests will catch all types of things like logical errors, regression errors etc. but web testing will catch context errors. A unit of code may purposefully return an exception and that will pass a unit tests but as useful as a stack trace is, users of your website won't ever think so. That being said, just because you stop an exception from being thrown doesn't mean there isn't an error. So both are needed to support a reliable system.
I want to give credit where it is due. I spent a lot of time reading about unit testing and testing Sitecore in particular. I found a helpful kickstart article about unit testing Sitecore specifically by Matt Kenny. Another by Alistair Deneys that provides a tool to run your tests. Another article by Dan Solovay detailing a solution to some context issues. A really insightful Wikipedia article on Code Coverage and a great article about the philosophy of unit tests. Apparently there was some wellspring article by Mike Edwards but the page appears to have vanished.
So I named the tool "TestStar". I'm clearly a fan of Star Wars. In my mind building this tool was an enormous task that wields a great amount of power and it focuses a lot of scrutiny on any given site so.... yeah, TestStar. You can get it on GitHub and I assure you, it is fully operational. I've also put up a video on YouTube to explain it a little more visually.
At it's core it's a web application with separate projects for unit and web tests. The intention is to make a companion application to a continuous integration server so that you can copy the binaries and run unit tests, then copy the binaries to your web application and run your web tests. The web app will run these tests through the webpages and when you've got a test configuration you like, there's a form to create a batch file with those settings. This can then be tied up to scheduled tasks and be run hands-free. My end use is to test Sitecore and my associated libraries with it, so I'll go into that towards the end.
Alright now that we got that out of the way let's get down to the nitty gritty.
I've packaged the NUnit and Watin libraries in with the project. I realize this creates dependencies and will not automatically update NUnit and Watin but I wanted to have the application be functional out of the box. You should be aware of that though. You can update the version on your own and/or fork it and do whatever you like to it.
The NUnitTesting.Core library has set of tools to retrieve test fixtures, test methods and categories from an assembly. The core of the code handles running tests and delegating events and corresponding interfaces which define how to handle test events. The web testing requires a bit of structure to store domain names, domain prefixes, environments etc. so there's entity classes and retrieval methods. The data for web testing is stored in JSON in local files and is reconstituted through deserialization. There's also some configuration information stored in the config files so there's a utility that will retrieve those values as well.
The NUnitTesting.WebTests library is where the web tests should be written. This is also a place where you'll likely want to add a fair amount of customization. In my implementation I created a few other base test classes and entities specific for my systems and Sitecore. I've left a few samples to show what's involved in creating a test and how they are displayed on the web application and I'll also explain it in more detail later in this post.
The NUnitTesting.UnitTests library is where unit tests should be written. I could've put all the tests into a single project but I found that separating them made more sense. You'll need to include references to any library you intend to test but I'm sure that's standard fare. Aside from that, you should use categories on either the test fixture or test method to help group the tests. If you intend to test Sitecore, you should read the Matt Kenny article detailing the setup which will help you debug your tests but when you build the library. The binaries (including the ones your testing) will end up in the NUnitTesting.WebApp project. You will still need to add your config files to the NUnitTesting.WebApp project but you won't need to manipulate the App_Config path since it will run just like Sitecore does.
The NUnitTesting.TestLauncher is a console application that your batch scripts will call to run your tests. You probably won't have to modify it at all but you can debug it if you need to understand why something isn't working or if you're curious about how the chain of events works. There are separate test handlers for unit and web tests because they require different parameters and may output messages differently.
The NUnitTesting.WebApp is where everything converges. There are really only three pages: Unit Test page, Web Test page and Web Test Data Manager page. The data manager page is how you'll start creating the information you'll need to run web tests against. This information is stored in JSON locally under the /data folder. Once you have the information, you're then able to run the web tests you've built against that data. The unit test will run against the binaries you have locally in the project.
This is also the intended root of where all the batch scripts will run from and by extension the console app and web/unit tests. There's a /scripts folder where batch scripts are stored and there's even a sample to show how to use it.
To get started with web testing you'll first need to understand a few concepts that the system is built on. These have developed organically over time.
An environment is largely defined by the domain prefix. In my sphere of influence, many of my sites are duplicated across several environments so that I can test various branches of code independently. I tend to create similar subdomains for each site environment. IE: my local environment has the "http://test." prefix and my integration environment has the "http://integration." prefix. I have around six different environments not all site live support each environment so the site itself stores the site-to-environments relationship information. For unique situations, the environment values, including the domain prefix, can be overridden on the local value stored on the site.
A system is really a collection of sites. I use it to be able to select multiple sites at one time. It really only is just a name and any given site only belongs to one system. This evolved because I wanted to be able to target a certain test against a group of sites. So instead of having to select every one individually, I can select by the group.
A site stores information such as root domain, environments, system but also has a key/value store called properties to allow extensibility. I use this to store Sitecore specific information such as site ID and language code. I can then write a subclass of the site class which exposes a property that will retrieve the values from the key/value store.
Tests are the modular components and should be customized to your specific environment. The project comes with two samples: PingTest and SitemapTest. To build a custom web test, you should extend the BaseWebTest abstract class and decorate the class with the [TestFixture, RequiresSTA] attribute. The BaseWebTest class provides contextual information that allows you to decide what you want the test to do for a given site and environment. The RunTest method is called when the test is run. It is also up to you to update the RequestURL and ResponseStatus properties to tell the event handlers what happened during the test.
Unit tests are standard NUnit tests and only require that the class has the [TestFixture] attribute and the test method have the [Test] attribute. You should also set a Category on either the [TestFixture] or [Test] attribute so that you can select groups of tests to run from the web app. For examples of what's available to you when setting up an NUnit test class refer to NUnit.org's documentation.
There's a handful of settings that are stored in the web.config and app.config files. This will allow you to configure and customize the system to so that the web pages and batch script creators will use a different library to load and launch tests from and the web test data can be stored in different files. It also helps when debugging the applications to have the values stored locally in the project app.config files. Here's the breakdown of those settings.
This is the file name where the web testing environments will be stored. The file lives under the NUnitTesting.WebApp/data folder.
Similar to the environments file, this is the file that stores the web testing systems.
Like the two previous files, this lives in the same folder and stores the web testing site information
This defines which assembly contains the web tests
This defines which assembly contains the web test launcher console application.
This defines which assembly contains the unit tests
This defines which assembly contains the unit test launcher console application.
For me the end game was to have the tests that I designed be run from scheduled tasks and continuous integration build events on my server. To that end, both web and unit testing pages have a form to create a batch script from the test configuration selected on the page. This can help you get up and running more quickly especially if you're not familiar with the syntax of batch scripts. There's some caveats with the web testing batch creator. The systems are essentially a grouping of sites but when the script is created the system value won't be used in the startup params. This is because you could select a system and then deselect a single site and the system will still be selected. So to prevent any issues, it just creates a list of the site id's. If you've got the saavy you can replace the list of id's with a single system id manually though.
The syntax of the batch script is specific to what the console application is expecting for input parameters. The unit and web test runner each has a different set of parameters. Here's the breakdown:
Web Tests: [0 - test type] [1 - test assembly] [2 - test name] [3 - environments] [4 - systems (optional)] [5 - sites (optional)]
The environments, systems and sites are all comma separated lists of the id's of their respective objects. The id's can be determined from the manager page where the id is displayed next to each item in the drop down lists. The id's are ordered in a zero-based index. Also although systems and sites are optional, at least one of them should be set.
example:
@echo off set TestLauncherPath=C:PathToNUnitTestingWebsiteinNUnitTesting.TestLauncher.exe @echo on "%TestLauncherPath%" "-w" "NUnitTesting.WebTests" "SitemapTest" "3,4" "0" ""
Unit: [0 - test type] [1 - test assembly] [2 - test categories (optional)] [3 - test methods (optional)]
The test categories and test methods are comma separated lists of strings that match the category name or method name from the unit test classes. Both are optional but at least one should be set.
example:
@echo off set TestLauncherPath=C:PathToNUnitTestingWebsiteinNUnitTesting.TestLauncher.exe @echo on "%TestLauncherPath%" "-u" "NUnitTesting.UnitTests" "Some Test Category,Another Test Category" "SomeNullTest"
It is possible to test a Sitecore system and it's associated class libraries from this system. It was actually it's intended purpose but I built it to work as a standalone utility for any web/unit tests. There are a handful of configurations you will have to make locally which are well explained by Mathew Kenny but I'll reiterate. Fair warning; this will test your skill level in working with application contexts.
For the web application to run the unit tests, you'll need to copy in some information from your existing Sitecore site.
1. You'll need to copy the following sections into your web.config file: Sitecore, appSettings, connectionStrings and configSections.
2. You'll also need to copy in the App_Config folder so there's a local reference to all the Sitecore configurations.
3. Next you'll need to add project references to Sitecore libraries and your site libraries.
4. You'll either need to create a data folder and copy in the license file then update your config to that folder, or update the data file path to an existing folder.
To enable debugging the class library projects you'll have to setup the project similar to a web application.
1. You'll need to create an App.config file and add the following sections from your web.config: Sitecore, appSettings, connectionStrings and configSections.
2. You'll then need to update all references to "/App_Config" in your App.config file to ".App_Config" because it needs a relative path to these files.
3. You'll also need to copy in the App_Config folder so there's a local reference to all the Sitecore configurations. You should know that these files all get copied into the bin folder and run from there in case you're having trouble determing where it's context is.
4. Next you'll need to add project references to Sitecore libraries and your site libraries from the WebApp project.
5. You'll either need to create a data folder and copy in the license file then update your config to that folder, or update the data file path to an existing folder.
6. If you're working on WebTests in the TestLauncher you'll need to create a data folder and copy the sites, systems and environments files from the WebApp project
It has been suggested that you can create a patch include config to update all the modified settings. This is true except for one; the connection strings path.
Since the TestLauncher requires a config file to help it run these libraries you'll see in the TestLauncher project's bin folder a NUnitTesting.TestLauncher.exe.config file which copies the app.config settings. You'll need this file in the WebApp project for it to run but, of course, there's an exception; the ".App_Config" path anywhere within the "Sitecore" section in the config file needs to be set to "..App_Config". This is because the context of the TestLauncher is partially in the scripts folder where the exe is called from and partially in the bin folder where the exe lives. I've kept a separate copy of the App.config and I copy it into the WebApp on the Post Build Event. Here's the code for that:
copy "$(ProjectDir)App.config.WebApp" "$(SolutionDir)WebsiteinNUnitTesting.TestLauncher.exe.config"
When you're creating unit tests for Sitecore you'll want to be able to set context data such as the site, language, database, device and whether or not the system is unit testing. I've handled this by setting these manually in the Setup method decorated by the [Setup] attribute. For some tests I needed to have a live site content to test against so I setup a test website in my system. Here's the code I'm using to create that context:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Sitecore.Collections;
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Globalization;
using Sitecore.SecurityModel;
using Sitecore.Sites;
using Sitecore.Web;
namespace NUnitTesting.UnitTests.Tests {
public class GlobalStatics {
public static StringDictionary SiteDic = new StringDictionary(){
{"allowDebug","true"},
{"browserTitle",""},
{"cacheHtml",""},
{"cacheMedia",""},
{"contentLanguage",""},
{"contentStartItem",""},
{"database","master"},
{"defaultDevice",""},
{"device",""},
{"disableClientData",""},
{"disableXmlControls",""},
{"domain","extranet"},
{"enableDebugger","true"},
{"enableLinkedItems",""},
{"enablePreview","true"},
{"enableWebEdit","true"},
{"enableWorkflow",""},
{"enableAnalytics",""},
{"filterItems",""},
{"hostName","unitesting.local"},
{"htmlCacheClearLatency",""},
{"language","en"},
{"loginPage",""},
{"masterDatabase",""},
{"mediaCachePath",""},
{"mode","true"},
{"name","UnitTesting"},
{"physicalFolder","/sites/UnitTesting"},
{"port",""},
{"requireLogin",""},
{"rootPath","/Sitecore/content"},
{"scheme",""},
{"startItem","/UnitTesting/Home"},
{"targetHostName",""},
{"virtualFolder","/"},
{"xmlControlPage",""}
};
public static SiteContext UTSiteContext {
get {
SiteInfo si = new SiteInfo(SiteDic);
return new SiteContext(si);
}
}
public static string DefaultDeviceID = "{FE5D7FDF-89C0-4D99-9AA3-B5FBD009C9F3}";
public static Database MasterDB {
get {
return Sitecore.Configuration.Factory.GetDatabase("master");
}
}
public static DeviceItem DefaultDevice {
get {
using (new SecurityDisabler()) {
return MasterDB.Resources.Devices[DefaultDeviceID];
}
}
}
public static Language EnglishLanguage {
get {
return Language.Parse("en");
}
}
public static void SetupContext(){
Sitecore.Context.Site = UTSiteContext;
Sitecore.Context.Language = EnglishLanguage;
Sitecore.Context.Database = MasterDB;
Sitecore.Context.Device = DefaultDevice;
Sitecore.Context.IsUnitTesting = true;
}
}
}
and here's how I'm calling it:
[SetUp]
public void SetUp() {
GlobalStatics.SetupContext();
}
The use of the .IsUnitTesting property is optional and will modify the path where Sitecore looks for the config files. Dan discusses this in more detail.
While working with web tests I found it useful to convert the TestSite object to a more specific subclass. This is because I added key/value properties to each of my Sitecore site entries to identify the site ID and content language. To be able to do this I created a SitecoreSite subclass from TestSite. Here's the definition for that:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using NUnitTesting.Core.Entities;
namespace NUnitTesting.WebTests.Entities {
public class SitecoreSite : TestSite {
public string LanguageCode {
get {
return GetProperty<string>("LanguageCode", string.Empty);
}
}
public string SiteNodeID {
get {
return GetProperty<string>("SiteNodeID", string.Empty);
}
}
private T GetProperty<T>(string key, T defaultValue) {
return (Properties.ContainsKey(key)) ? (T)Properties[key] : defaultValue;
}
public string SCBaseURL(TestEnvironment env) {
return (string.IsNullOrEmpty(LanguageCode))
? BaseURL(env)
: string.Format("{0}/{1}", BaseURL(env), LanguageCode);
}
}
}
The TestSite class has a ConvertTo method that basically just serialized the class the JSON and then back into the specified class. Here's an example of that from within a web test class:
SitecoreSite scs = ContextSite.ConvertTo<SitecoreSite>();
After the system is setup and running the long term goal is to get the binaries/configs updated regularly. I haven't made it to this point yet, so I don't have anything to provide as a sample but it will require some sort of batch/build script to do it.
So that's TestStar. The web app / console app that can run both unit and web tests. Development is ongoing but at the current time of this writing it is stable so please download and fork away.