11. March 2013 15:02
This is a technical post about how we automatically deploy new portals that will be run by our own portal software discoverize.
After filling in and submitting our portal creation form we do the following:
Save the entered information into /App_Data/<PortalName>/portal.xml
Trigger the TeamCity build through a web request to a specific URL
Send an email with the request status or any errors that occurred
Display a Thank You page.
Now, the TeamCity build configuration that was triggered under 2. does the following:
Execute the Powershell script create-portal.ps1 in c:\projects\discoverize
create the AppPool Discoverize-Portals if it doesn't exist yet (all our portals run within it)
iterate over all folders inside wwwroot\discoverize.com\App_Data
if a folder with the same name exists under wwwroot\discoverize-portals, skip it
else move the current folder from wwwroot\discoverize.com\App_Data to wwwroot\discoverize-portals
add a line with the full path to the new portal folder to newportals.txt (e.g. C:\inetpub\wwwroot\discoverize-portals\<PortalName>)
add a new site to IIS and set its AppPool to Discoverize-Portals
start the new site in IIS (needs Admin privileges, that's why we use a parameterized task scheduler task for that)
Build the source code from the repository
Execute the Deploy-NewPortals target from deploy.proj using MSBuild in c:\projects\discoverize
Iterate over the entries in newportals.txt, for each entry do:
Copy the build output from step 2. to wwwroot\discoverize-portals\<PortalName>
Run the site setup (this uses Orchard's setup method) with our own recipe (using Orchard.exe)
Call an action (SetupComplete) on the new site to mark deployment as done
Create a default entry so there's something to look at (using Orchard.exe)
The SetupComplete action then finishes doing this:
save the information of the portal.xml into an application specific configuration file
create new user "portal-owner" for content management
send an email about the successful installation of the new portal
Quite complex, if I look at it now, but it works!
22. August 2012 13:48
When we started development on Marinas.info, we decided to write acceptance tests for all important features of our application. This decision was even more justified by the fact that a bunch of similar platforms are to follow using the same codebase. We wanted an application with less bugs and easier maintenance. Writing good, automated acceptance tests is not easy and it’s not fast, either. For some time now, we’ve been trying to get the first set of our tests run green, which proved especially tricky on our TeamCity continuous integration server. This post investigates a working solution. The ingredients: SpecFlow, Coypu (Selenium), Browser, Web Server, and MvcIntegrationTestFramework SpecFlow In .NET world, using SpecFlow to write acceptance tests is nothing new and has recently become, yet again, more appealing after its update to version 1.9. One of our scenarios for verifying image upload functionality looks like this: It’s simple to write, easy to read and great living documentation. For Browser based tests you need: Coypu (Selenium) Everyone who has written tests for Selenium for even a mildly ajax-y site knows how painful it can be to create reliably working tests. Coypu alleviates the pain and makes test creation as straight-forward as it should be in the first place. Coypu is: A robust wrapper for browser automation tools on .Net, such as Selenium WebDriver that eases automating ajax-heavy websites and reduces coupling to the HTML, CSS & JS A more intuitive DSL for interacting with the browser in the way a human being would A few examples of Coypu’s clean API can be seen here in one of the step definitions for the above scenario (Browser is an instance of the BrowserSession class from Coypu): A web browser To run browser based tests you, of course, need … a browser! Coypu offers support for quite a bunch of them, including the usual suspects Internet Explorer, Chrome, and Firefox. A web server You need to host your application in some web server or another to process requests. Well, this statement turns out to be only partially true, as you will see with the MvcIntegrationTestFramework. But at least for browser based test you need a web server, and you basically have the choice between IIS and IIS Express (if you don’t want to write your own or use someone else’s implementation). We chose IIS Express as it is manageable through a non-administrator account, but it needs to be installed on all machines that will execute the tests. For non-browser based tests: MvcIntegrationTestFramework Introduced by Steven Sanderson in 2009, this small framework allows to write integration tests for ASP.NET MVC applications and execute them without a browser! It empowers you to make assertions on your controllers’ actions’ results rather than on the rendered html output by injecting some clever hooks into your MVC application under test. An example of how a test would look can be found in the above mentioned post. The “magic” of this framework lies in the use of ApplicationHost.CreateApplicationHost() which creates an application domain for hosting your ASP.NET application. Check out this screenshot of part of the source code: How to put the pieces together After a quite radical evolution of our test code (which you can read up on in my follow-up post The Long Road to Browser Based Acceptance Testing), we finally settled for the following: Before the first test starts, setup an instance of the AUT (application under test). This includes: deploying the AUT as we do for our staging environment, but to a temp folder initialize an AppHost instance à la MvcIntegrationTestFramework, i.e. an ASP.NET enabled application domain that hosts the AUT execute the Orchard setup command via the AppHost instance (instead of running the setup through a browser, which we used to do but was a lot slower) Before each test run (SpecFlow scenario) we then execute various commands to setup the environment for the concrete test, e.g.: clean the database simply by overwriting it with a copy we saved after the initial setup create Marina entries that will be displayed and searchable on the site, again, using the AppHost instance Once we want to execute steps in the browser, we do the following: start an instance of IIS Express pointing to the deployed application (we used the wrapper code from Spinning up IISExpress for integration testing) initiate a Coypu BrowserSession which under the hood creates an instance of the browser you choose after battling with Internet Explorer, Chrome, and Firefox Portable, we now use Firefox 10.0.6 ESR (Extended Support Release) because version 10 is of now the highest version supported by Selenium (2.1.25) and the ESR doesn’t ask to be updated all the time After each test run (SpecFlow scenario) we do this: close the browser shut down the IIS Express instance (we slightly modified the above mentioned wrapper code calling Kill() on the process instance after the call to CloseMainWindow() so that it reliably terminates even on TeamCity) Conclusion Setting up a reliable environment for automatically executing acceptance tests has not been a walk through the park but we finally have a solution that basically “just works”. Hopefully, our experience will help you save a couple of hours and also some headache along the way Happy coding!
26. January 2011 20:30
Today: a simple hosts file editor Today I set up a new project on GitHub: the Teamaton tool belt! It shall serve us as a central store for small tools, probably mostly command-line, built for a single purpose. The first tool in our new tool belt is: HostsEditor. I wrote this small command line utility after reading this blog post: http://apdubey.blogspot.com/2008/09/edit-host-file-by-batch-file.html. We were looking for an easy way to edit the Windows hosts file from a script for a larger development environment setup that we wanted to fully automate including IIS website, bindings and new entries in the said hosts file. Of course, for a one-timer echo does the job, but don’t try to run the setup script twice on the same maching – you’ll get duplicate entries very quickly. Reading about how other people are asking the author of the above post how to avoid the duplicates or how to erase entries from the hosts file, I set out to simply write such a utility myself and share it with the rest of the world. If there is one other person out there to whom it will be useful I will be happy. So, HostsEditor does the following: Adds and removes entries to and from the Windows hosts file. Automatically backs up the original hosts file to hosts.bck. Does not add duplicate entries; instead prints a warning. For usage in automation scripts there is /q switch to suppress all info messages. That’s all there is to it for now. But there’s more to come! Stay tuned, Oliver
21. January 2011 20:19
We recently updated one our largest project to use ASP.NET 4.0, and for this matter the new Package/Publish feature including sub-web.configs which is meant to supersede the Web Deployment Project. For a manual deployment there’s a good write-up on the msdn library titled ASP.NET Web Application Project Deployment Overview which shows how and where to set this up. In our case this was not satisfactory because our deployment process is a bit more complicated. We push our changes to a central repository and use JetBrains’ continuous integration server (CIS) TeamCity Professional, which is totally free for our project size, for a continuous integration process. Once TeamCity has pulled and tested the current version, it is supposed to deploy this version to our staging server where we further test the complete site. The key point in an automatic deployment was the management of the different web.config files for the different environments our project is running on. Unfortunately, until yesterday every deployment that included changes to the web.config file – even to the staging server - required a manual step of editing the web.config that live on our staging system (outside of source control!). What we used to do: after a successful build on our CIS we simply copied the web application (files) to our staging server! But as Scott Hanselman wrote: If You're Using XCopy, You're Doing It Wrong! This post inspired us to move along and take advantage of the new possibilities that we were given. In the meanwhile, before switching to .NET 4.0 actually, we also took a shot at the Web Deployment Project way of doing things but never actually got that far as to fully automate the deployment – somehow the setup was not as easy as we hoped. Anyway, we wanted web.config Transforms! So what does our setup look like and what did we want to do? During local development and testing I use a web.config file that talks to a local DB instance and has some more specific settings. To run the web application on our staging server we need to replace certain values or whole sections in the web.config. For this transformation we use the sub-web.config files, one for each build configuration: Now, with all of these web.config files the simple XCOPY deployment we used to use does not work any longer. We need to trigger the web.config transformation on the build server and then deploy the whole application. As easy as this looks using the built-in menus and dialogs in Visual Studio – it took me quite a while to find how to do this in an automated build, more concretely from the command line. After unsuccessfulle skimming stackoverflow.com for a solution I finally tripped over this very informative blog post on publishing a VS2010 ASP.NET web application using MSBuild. Admittedly, the author focuses on how to publish on the local machine as it’s yet a different process but towards the end he posts the solution I was looking for: 1: msbuild Website.csproj "/p:Platform=AnyCPU;Configuration=Release;DesktopBuildPackageLocation=c:\_Publish\stage\Website.zip" /t:Package
This was it! After running this on my machine with my own settings I looked into the folder with the zip file and found the following 5 files:
At first I just wanted to take the zip file, copy it to the staging server, unpack it – done! But then I peaked into it… and deeper… and deeper… and… still deeper… until I finally saw our application files underneath this directory:
This has got to be one of the longest paths I’ve ever seen and used! How would I automate the extraction of web application files from the zip with such a path? I was already seeing myself hacking away on the command line…
But wait: what about those files that appeared next to the zip file? A ci-stage.deploy.cmd and a readme.txt caught my attention – of course, I opened the cmd file first :-D
Well… maybe the readme file gives me a shortcut to understanding this and the rest of the 190 lines:
Looks promising! I convinced myself to give it a shot. So we set up a new configuration in TeamCity with the following settings:
These settings reflect the command line from above with a few minor changes (removed the DesktopBuildPackageLocation and set the /v[erbose] switch to m[inimal]):
msbuild Website.csproj "/p:Platform=AnyCPU;Configuration=Release" /t:Package /v:m
The second step is to use the generated script, ci-stage.deploy.cmd. I recommend to run the script by hand once using the /T switch just to make sure everything looks alright. In our case we found out that the deployment of the package would have deleted a lot of files, most of all images, from our website. This was not what we wanted! After a quick search I found this question on stackoverflow.com: MSDeploy: “Leave extra files on destination” from command line? So I added this switch to the parameters list in the build step configuration as follows:
That’s it! This is all we need on the command line to generate a package that is ready for deployment on the staging server. There are a few more necessary settings, including the DesktopBuildPackageLocation, that can be found in the Package settings window inside the project properties of the web application project:
the DesktopBuildPackageLocation can be set here instead of on the command line
the website and application name on the destination IIS server that this package should be installed to
some more options like excluding debug symbols etc.
These settings are saved in the project file and will be used during generation and deployment of the package.
That’s all I have to say right now.
Happy Coding, Oliver