Sunday, 29 March 2015

Running NUnit tests with Jenkins

In this series:

At the time of writing Jenkins is at version 1.602.

Following on from my previous post - Automatically triggering a Jenkins build on Git commit - the next step is to automatically run unit tests as part of the build.

My unit tests are written using NUnit. As with everything in Jenkins getting unit tests to run as part of a build is a fairly manual process which involves adding a new build step to call the NUnit console application directly. Let’s get started.

Step 1 – Install the NUnit binaries

If you haven’t got it already you’ll need to download and install NUnit. You can do so using an MSI or grab the Zip file and extract the binaries to a folder on your machine. Of course you could use Chocolatey to install NUnit.

chocolatey install nunit

Step 2 - Add a build step to your Jenkins build

Now you have NUnit you’ll need to add a new build step to your Jenkins build. Choose your project from the dashboard.



On the configuration page scroll to the bottom and add click ‘Add build step’. Choose ‘Execute Windows batch command’.


This adds the new build step that you will use to call NUnit. That’s simply a case of calling the NUnit console executable passing in the path to the assembly containing the tests to be run.


Remember to call nunit-console.exe or nunit-console-x86.exe if you’re testing a 32-bit assembly. If you need to run tests against multiple assemblies you can do so by listing them on the command line. NUnit command line documents are here

Once you’ve done all that save the changes and click ‘Build now’.


All things being equal you should have a successful build. If you click on the build in the Build History and look at the Console Output you will be able to see the results of the NUnit run.


The output isn’t very slick though, is it? Lets see about getting a nice unit test report working.

Step 3 – Publish NUnit report

First, make sure the NUnit plugin has been installed in Jenkins. If it’s not there you’ll need to add it. Go to Dashboard > Manage Jenkins > Manage Plugins and check your installed plugins. If it’s not there add it. This plugin makes it possible to import NUnit reports from each build into Jenkins so they are displayed with a trend graph and details about which tests that failed.


Next you need to get NUnit to publish its results as an XML file from your build. To do that add ‘/xml=nunit-result.xml’ to the end of the batch command you added in Step 2.


Now add a post-build action to your build definition and select ‘Publish NUnit test result report’.

jenks009Enter the name of the XML file you added to the batch command (nunit-result.xml) as the ‘Test report XMLs’ value and save the build configuration.


Run a build and when it completes click on the build ID in the Build History list for the project. You’ll find a new item, Test Result, in the build history. Click on the Test Result link to see the report.


Job done!

How to update Chocolatey

I’ve recently started using Chocolatey - a package manager for Windows – to install tools and applications. If you’re not familiar with Chocolatey it’s described thus:

“Chocolatey is a package manager for Windows (like apt-get or yum but for Windows). It was designed to be a decentralized framework for quickly installing applications and tools that you need. It is built on the NuGet infrastructure currently using PowerShell as its focus for delivering packages from the distros to your door, err computer.” [1]

Installing Chocolatey is easy and is explained on their homepage. But how do you upgrade Chocolatey to the latest version after you’ve installed?

Firstly, I opened a command prompt as administrator and ran Powershell - actually I used ConEmu which has a convenient tab for just this purpose. I then ran ‘chocolatey version’ to see what’s what.


Looks like there’s a new version. Let’s update. Doing so is just like updating anything with Chocolatey. Run the ‘chocolatey update’ passing in the package name; ‘chocolatey’ in this case.


I’ve truncated the console output but suffice is to say Chocolatey has been updated.


[1] About Chocolatey

Saturday, 28 March 2015

Using ApprovalTests to test a simple logging framework

I’ve been playing with a simple logging framework that I’ve had lying around for ages and thought I’d add an NLog implementation just for kicks and giggles. It also gave me an opportunity to play with the ApprovalTests framework by Llewellyn Falco.

Very briefly ApprovalTests lets you compare output generated during a test with a predefined example. If they differ the test fails. In this case I wanted to verify that a log file generated during a the execution of a unit test matched an example log file. I could then be sure that the log file had been generated and that the contents were correctly formatted.

I’m not going in to a great deal of detail about the basics of unit testing here. Suffice is to say that I’m using NUnit.

In Visual Studio I created a test project, added a reference to the project containing my logger implementation and added a few NuGet packages to the test project:


Note that I’ve added ApprovalTests and NLog as well as NUnit.

I added an App.config file to the test project and put some basic NLog configuration in it. NLog is very helpful and attempts to automatically configure itself on start-up by looking for configuration files in some standard locations. One of those is – you guessed it – the standard App.config file.

<?xml version="1.0" encoding="utf-8" ?>
    <section name="nlog" type="NLog.Config.ConfigSectionHandler, NLog"/>
  <nlog throwExceptions="true">
      <target name="file" type="File" fileName="log.txt" layout="${level:uppercase=true}|${logger}|${message}|${exception}" />
      <logger name="*" minlevel="Debug" writeTo="file" />

Note the target on line 8. That defines the layout for a log message and therefore what we expect to see for each entry in the log file. The macros (prefixed with $) will be replaced with appropriate values when the log entry is written. Also note that the generated log file will be called log.txt and will appear in the same directory as the compiled dlls.

The next step is to create a test fixture. Here’s some code:

namespace Andy.French.Logging.NLog.Tests
    using System;
    using System.IO;

    using ApprovalTests;
    using ApprovalTests.Reporters;

    using NUnit.Framework;

    /// <summary>
    /// Tests for the <see cref="Andy.French.Logging.NLog.Logger"/>.
    /// </summary>
    public class LoggerTests
        /// <summary>The logger to test.</summary>
        private ILogger logger;

        /// <summary>
        /// Sets up a unit test.
        /// </summary>
        public void SetUp()
            this.logger = new Logger("This.Is.A.Test.Logger");

        /// <summary>
        /// Tests that the information method the with a string argument writes a message in correct format.
        /// </summary>
        public void Information_WithStringArgument_WritesInfoMessageInCorrectFormat()
            // Arrange

            // Act
            this.logger.Information("This is a test message.");
            // Assert
            Approvals.Verify(new FileInfo("log.txt"));

        /// <summary>
        /// Tears down a unit test.
        /// </summary>
        public void TearDown()
            if (File.Exists("log.txt"))

So what’s going on here?

Firstly, we have a setup method that instantiates a new instance of my Logger class prior to execution of each unit test. We also have a teardown method that checks for the log file at the end of each unit test and if it’s there it deletes it. This way we guarantee that each unit test starts with a clean slate.

Update: The teardown method isn’t strictly required because ApprovalTests cleans up after each unit test run. In this case it deletes the log file.

There are a couple of interesting lines in the test. On line 33 there’s a UseReporter attribute. This is part of the ApprovalTests framework and does some interesting things. More on that later.

On line 42 we make a call to Approvals.Verify. Note we aren’t using standard NUnit assertions here but are using the ApprovalTests API instead. This call is going to check that the log file matches the predefined example. If it doesn’t, the test fails.

So where is the example log file and how do we generate it? That’s where the UseReporter attribute comes in.

The first time you run the test in Visual Studio (I use the ReSharper unit test runner) the test will fail because we haven’t got an example for ApprovalTests to use to make a comparison with. Because we’ve added the UseReporter attribute ApprovalTests steps in and reports what’s going on. It actually creates an example file for us, shows us what’s in the generated file and allows us to edit the example.

Here’s the test failure in the ReSharper test runner:


And this is what we’ve got in the Visual Studio editor:


The output in the actual log file is correct so we can safely copy the text from the log file (on the left) into the example (on the right) and save it.


Note that ApprovalTests has created the example file in the project folder and has used a naming convention based on the name of the test class and the name of the test method. ApprovalTests uses this naming convention to matchup an example file with a specific unit test. In this case the name is:



To see this file click the ‘Show All Files’ icon in the Solution Explorer. Right-click on it and select ‘Include In Project’. The example files need to be saved with the project and added to version control.

Next time the test is run we get a pass!


By the way, if you’re wondering about the test method naming convention it’s the one I’ve used for years and was first suggested by Roy Osherove.

Update: I forgot to mention that ApprovalTests doesn’t just work with text-based files like the log file we were testing here. It can work with a variety of file formats including images. You can find the source code here if you are interested in poking around.

Sunday, 15 March 2015

Automatically triggering a Jenkins build on Git commit

In this series:


Following on from my previous post, Getting started with Jenkins, Git and MSBuild, the next step is to get Jenkins to build whenever I commit changes to a local Git repository.

Option1 – Polling the repository

Now, it is possible to setup a Build Trigger to get Jenkins to poll for changes in a Git repository. You do this using the Poll SCM option on the project configuration page in Jenkins.


By enabling Poll SCM and entering a cron expression * * * * * in the Schedule text box we can get Jenkins to poll the repository for changes every minute. You can enter whatever cron expression you like but this serves as a simple example.

Polling the repository is a bit ugly and inefficient. Wouldn’t it be better to get Git to notify Jenkins of a commit and to trigger a build that way? Well you can do just that using ‘Git hooks’.

Option 2 – Triggering a build using Git hooks

Inside every Git repository there’s a hidden .git folder. Inside the .git folder is a hooks subfolder. This folder can hold scripts that are executed by Git when it performs certain operations. This feature is well documented (see Customizing Git - Git Hooks in the Git documentation).


I’m not going to delve into Git hooks here but will just show the steps I took to get Git to trigger a Jenkins build on a Git commit to a local repository by using the post-commit hook.

However, before we get into the details of the post-commit Git hook we need a few other pieces to be in place.

The Git Plugin notifyCommit endpoint

When you install the Git Plugin for Jenkins you get an HTTP endpoint that can be used to trigger Jenkins to check a Git repository for changes and to schedule a build if it finds any. You can test this by hitting the URL directly. Here’s an example that works on my setup (you would need to change the url to match a repository you have access to):


This feature is described in the ‘Push notification from repository’ section of the Git Plugin documentation. The documentation describes what happens when you hit the endpoint:

“This will scan all the jobs that are configured to check out the specified URL and the optional branches.  If they are also configured with polling, polling will be immediately triggered (and if that finds a change worthy of a build, a build will in turn be triggered). We require the polling configuration on the job so that we only trigger jobs that are supposed to be triggered from changes in the source tree.”

NB: There’s a gotcha here! Make sure you have enabled Poll SCM in your Jenkins build configuration. Don’t specify a schedule, just enable the polling. If you don’t, nothing will happen.

If you see an error like the following check you’ve enabled Poll SCM:

No Git consumers using SCM API plugin for: file:///c:/Source/Andy.French.Configuration.Service

If you hit the notifyCommit endpoint and there are changes in the repository a build will be triggered. If there are no changes nothing will happen but you can still see that the repository was checked in the Git Polling Log.



When you installed Git you got curl – a utility for automating HTTP jobs - for free. It’s an exe that lives in the Git bin directory. You can use it to submit requests to an HTTP endpoint like the notifyCommit one we have been looking at and we can do this from a script.

The post-commit Git hook

What we need is for Git to use curl to call the notifyCommit endpoint after a commit takes place. We can do that by using the post-commit Git hook. To get this working add a file called post-commit (no file extension) to the .git\hooks folder in your repository and open it in a text editor. Add the following to the file:

curl http://localhost:8080/git/notifyCommit?url=file://c:/Source/Andy.French.Configuration.Service

You will want to change the URL to one that points to the repository you want to build.

NB: There’s another gotcha here! The first line of the script is essential. This is a bash script and it won’t be executed if that line is missing.


After putting the post-commit Git hook in place I was able to trigger Jenkins builds whenever I committed changes to the repository. Result! The next step is to run unit tests as part of the build.

Live Writer can’t login to Blogger

I have been struggling to get Live Writer to login to Blogger so I can publish posts like this one. Every time I tried to publish I got a login failure.

The answer was simple. I have changed the security on my Google account to use 2-Step verification, something an application like Live Writer can’t do. The answer is to use ‘App passwords’.

A full explanation can be found here:

I added an App password for Live Writer and everything works again.

Update 21/12/2015

If you’ve been using Live Writer it’s now time to move over to Open Live Writer. This integrates nicely with Blogger.

“Today is the day. An independent group of volunteers within Microsoft has successfully open sourced and forked Windows Live Writer. The fork is called Open Live Writer (also known as OLW) and it is part of the .NET Foundation and managed by this group of volunteers. Read the fantastic announcement at the .NET Foundation Blog! Download Open Live Writer now!” [1]

[1] Announcing Open Live Writer - An Open Source Fork of Windows Live Writer, Scott Hanselman.

Getting started with Jenkins, Git and MSBuild

In this series:

I recently changed jobs and at the new place we are using Jenkins for continuous integration. We are also transitioning to Git from Subversion for source control. I’ve used CruiseControl.Net and Team City before but Jenkins is new to me so I thought it would be interesting to replace my Team City installation at home with Jenkins so I can get a feel for it.

At the time of writing Jenkins is at version 1.602.

1. Install Jenkins on Windows

Jenkins installation is pretty straightforward. Hop over to the Jenkins website and downloaded the Windows installation package.


The installation package – a Zip archive – contains an MSI installer. Run it. Once that’s completed Jenkins will have been started for you and your web browser will open at http://localhost:8080/, the default for Jenkins.

It’s not immediately obvious but what the MSI has done is installed Jenkins as a Windows service and started it for you.


If you have problems during the installation you can easily start Jenkins manually from the command line. On my system the command looked like this:

"C:\Program Files (x86)\Java\jre7\bin\java.exe" -jar jenkins.war

You can then go to http://localhost:8080/ to see the Jenkins dashboard. If Jenkins hasn’t installed as a Windows Service you can manually complete that step from the Jenkins management pages. On the Manage Jenkins page you’ll find an ‘Install as Windows Service’ link.

See Install Jenkins as a Windows service.

2. Configure Jenkins to be able to use Git

I had Git installed on my workstation already but if you haven’t you’ll want to do that now.

Once you have Git installed return to the Jenkins dashboard and go to Manage Jenkins > Manage Plugins. Find the Git Plugin on the Available tab (note there’s a filter top-right of the page to help reduce the number of items to look through). Check the tick box next to the Git Plugin and click Install without restart. Note: You may get a message saying to need to restart Jenkins. Just check the restart checkbox and Jenkins will restart for you.

3. Configure Jenkins to be able to use MS Build

If you are building a Visual Studio project you’ll probably want to install the MSBuild Plugin. The process is the same as for installing the Git Plugin. Follow the process in step 2 above to do so.

That’s not all you need to do though. You’ll need to configure Jenkins to know where to find the MSBuild.exe. Go to Manage Jenkins > Configure System.


Scroll down to find the MSBuild section. Here you can add multiple MSBuild entries if you need to to match your requirements (e.g. different .Net versions, 32 or 64 bit, or maybe the version that comes with Visual Studio).


Note: I have been seeing the following warning when adding MSBuild entries:

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\MSBuild.exe is not a directory on the Jenkins master (but perhaps it exists on some slaves)

I’ve checked my configuration and no slaves are configured. I’ve also had a look at StackOverflow (Jenkins can't find msbuild) but none of the suggested answers work for me. However, MSBuild is being invoked and my builds are working. If I find a resolution I’ll post an update.

4. Setup a build

For my first Jenkins build I chose to use a small project I have that has no external dependencies (no NuGet packages etc.). It’s a very simple configuration service that I use in little projects.

I added a new item and chose Freestyle project before clicking OK.


I was then faced with the configuration page for the project. On this occasion I decided to keep things simple and work against a local Git repository. Under Source Code Management I chose Git and pointed the Repository Url at my local repository.


The next step was to get MSBuild to build the source. This is done in the Build section. When doing this make sure you choose the appropriate instance of MSBuild – choose from the MSBuild instances you configured in step 3 above – and make sure it’s not left as ‘default’. I didn’t bother with any Command Line Arguments but if you need to run specific build configurations (e.g. Release) you can add the appropriate MSBuild arguments in the text box provided.


Once I’d saved the configuration I was ready to go and all I had to do was click Build Now to trigger the build.


Once a build has completed the result is visible as a new item in the Build History. If you want to see what happened - maybe you want to start investigating a failed build - you can click on an item in the Build History and from there the Console Output to see the log that was generated by the build.

Wrapping up

So I got my first build to work. First impressions are that Jenkins lacks the slickness of Team City but it appears functional. Next steps for me are to investigate triggering the build when source code is committed and to get my NUnit unit tests to run as part of the build. Updates to follow.

Next: Automatically triggering a Jenkins build on Git commit