Writing Acceptance Tests for an Orchard / ASP.NET MVC Application – using SpecFlow, Coypu (Selenium) and the MvcIntegrationTestFramework

When we started development on Marinas.info, we decided to write acceptance tests for all important features of our application. This decision was even more justified by the fact that a bunch of similar platforms are to follow using the same codebase. We wanted an application with less bugs and easier maintenance.

Writing good, automated acceptance tests is not easy and it’s not fast, either. For some time now, we’ve been trying to get the first set of our tests run green, which proved especially tricky on our TeamCity continuous integration server. This post investigates a working solution.

The ingredients: SpecFlow, Coypu (Selenium), Browser, Web Server, and MvcIntegrationTestFramework

SpecFlow

In .NET world, using SpecFlow to write acceptance tests is nothing new and has recently become, yet again, more appealing after its update to version 1.9. One of our scenarios for verifying image upload functionality looks like this:

It’s simple to write, easy to read and great living documentation.

For Browser based tests you need:

Coypu (Selenium)

Everyone who has written tests for Selenium for even a mildly ajax-y site knows how painful it can be to create reliably working tests. Coypu alleviates the pain and makes test creation as straight-forward as it should be in the first place. Coypu is:

• A robust wrapper for browser automation tools on .Net, such as Selenium WebDriver that eases automating ajax-heavy websites and reduces coupling to the HTML, CSS & JS
• A more intuitive DSL for interacting with the browser in the way a human being would

A few examples of Coypu’s clean API can be seen here in one of the step definitions for the above scenario (Browser is an instance of the BrowserSession class from Coypu):

A web browser

To run browser based tests you, of course, need … a browser! Coypu offers support for quite a bunch of them, including the usual suspects Internet Explorer, Chrome, and Firefox.

A web server

You need to host your application in some web server or another to process requests. Well, this statement turns out to be only partially true, as you will see with the MvcIntegrationTestFramework. But at least for browser based test you need a web server, and you basically have the choice between IIS and IIS Express (if you don’t want to write your own or use someone else’s implementation). We chose IIS Express as it is manageable through a non-administrator account, but it needs to be installed on all machines that will execute the tests.

For non-browser based tests:

MvcIntegrationTestFramework

Introduced by Steven Sanderson in 2009, this small framework allows to write integration tests for ASP.NET MVC applications and execute them without a browser! It empowers you to make assertions on your controllers’ actions’ results rather than on the rendered html output by injecting some clever hooks into your MVC application under test. An example of how a test would look can be found in the above mentioned post.

The “magic” of this framework lies in the use of ApplicationHost.CreateApplicationHost() which creates an application domain for hosting your ASP.NET application. Check out this screenshot of part of the source code:

How to put the pieces together

After a quite radical evolution of our test code (which you can read up on in my follow-up post The Long Road to Browser Based Acceptance Testing), we finally settled for the following:

1. Before the first test starts, setup an instance of the AUT (application under test). This includes:
• deploying the AUT as we do for our staging environment, but to a temp folder
• initialize an AppHost instance à la MvcIntegrationTestFramework, i.e. an ASP.NET enabled application domain that hosts the AUT
• execute the Orchard setup command via the AppHost instance (instead of running the setup through a browser, which we used to do but was a lot slower)
2. Before each test run (SpecFlow scenario) we then execute various commands to setup the environment for the concrete test, e.g.:
• clean the database simply by overwriting it with a copy we saved after the initial setup
• create Marina entries that will be displayed and searchable on the site, again, using the AppHost instance
3. Once we want to execute steps in the browser, we do the following:
• start an instance of IIS Express pointing to the deployed application (we used the wrapper code from Spinning up IISExpress for integration testing)
• initiate a Coypu BrowserSession which under the hood creates an instance of the browser you choose
• after battling with Internet Explorer, Chrome, and Firefox Portable, we now use Firefox 10.0.6 ESR (Extended Support Release) because version 10 is of now the highest version supported by Selenium (2.1.25) and the ESR doesn’t ask to be updated all the time
4. After each test run (SpecFlow scenario) we do this:
• close the browser
• shut down the IIS Express instance (we slightly modified the above mentioned wrapper code calling Kill() on the process instance after the call to CloseMainWindow() so that it reliably terminates even on TeamCity)

Conclusion

Setting up a reliable environment for automatically executing acceptance tests has not been a walk through the park but we finally have a solution that basically “just works”. Hopefully, our experience will help you save a couple of hours and also some headache along the way

Happy coding!