Wednesday, 25 June 2014

Using Entity Framework in integration tests

Entity Framework code first includes an interesting feature called database initializers. When I first encountered this feature I wondered if it would be possible to use a database initializer to drop and recreate a database as part of a suite of integration tests. It might prove very useful if we were able to create a set of repeatable and atomic tests around, for example, a data access layer. Of course it turns out that this is possible.

At the time of writing Entity Framework is in version 6.1.1.

 

What is a database initializer?

A database initializer is an implementation of the IDatabaseInitializer<TContext> interface and is used by Entity Framework to setup the database when the context is used for the first time. This could involve dropping and recreating the entire database, or just updating the schema if the model has changed. MSDN describes the interface as follows:

“An implementation of this interface is used to initialize the underlying database when an instance of a DbContext derived class is used for the first time. This initialization can conditionally create the database and/or seed it with data.” [1]

There are several implementations available out of the box:

  • DropCreateDatabaseIfModelChanges<TContext> – will DELETE, recreate, and optionally re-seed the database only if the model has changed since the database was created.
  • DropCreateDatabaseAlways<TContext> - will always recreate and optionally re-seed the database the first time that a context is used in the app domain. To seed the database, create a derived class and override the Seed method.
  • CreateDatabaseIfNotExists<TContext> - will recreate and optionally re-seed the database only if the database does not exist. To seed the database, create a derived class and override the Seed method.

In our scenario we always want to reinitialize the database before each test. This will give us a repeatable baseline at the start of each test case. Also, we probably want to be able to insert some known test data. The DropCreateDatabaseAlways<TContext> class looks like it does exactly what we want: it always recreates the schema and can optionally re-seed the database.

 

Creating an initializer derived from DropCreateDatabaseAlways<TContext>

Creating our custom initializer turns out to be very simple:

namespace Andy.French.Repository.Entity.Framework.Tests
{
    using System.Data.Entity;
    using System.Data.Entity.Migrations;

    using Andy.French.Repository.Entity.Framework.Tests.Domain;

    public class TestInitializer : DropCreateDatabaseAlways<TestContext>
    {
        protected override void Seed(TestContext context)
        {
            context.Customers.AddOrUpdate(
                                c => c.Id,
                                new Customer { Name = "Customer 1" },
                                new Customer { Name = "Customer 2" });

            base.Seed(context);
        }
    }
}

In this example I’m seeding the database with a couple of customers. Naturally you would seed the database according to your needs using your domain classes.

 

Invoking the initializer in our test suite

I use NUnit for tests - and have done so for a very long time – so the example below is based on NUnit.

For true atomic repeatable tests you should drop and recreate the database before each and every test case but for this example I have chosen to do so once for each test fixture - a class that contains tests. To do that I have created a class that’s marked with the SetUpFixture attribute. NUnit will pick this up and will run the method marked with the SetUp attribute for each test fixture and before any of the tests it contains are run.

namespace Andy.French.Repository.Entity.Framework.Tests
{
    using System.Data.Entity;
    using NUnit.Framework;

    [SetUpFixture]
    public class SetUpFixture
    {
        public SetUpFixture()
        {
        }

        [SetUp]
        public void SetUp()
        {
            Database.SetInitializer(new TestInitializer());
            
            var context = new TestContext();
            context.Database.Initialize(true);
        }
    }
}

That’s all here is to it! On line 16 we call the Database.SetInitializer method so Entity Framework will use our custom initializer. Remember, the custom initializer extends DropCreateDatabaseAlways<TestContext> so the initializer is set for the test context type.

On lines 18 and 19 we simply create a context and call context.Database.Initialize(true) which causes the database to be dropped, recreated and re-seeded. MSDN describes the Initialize method in the following terms:

“Runs the the registered IDatabaseInitializer<TContext> on this context. If "force" is set to true, then the initializer is run regardless of whether or not it has been run before. This can be useful if a database is deleted while an app is running and needs to be reinitialized. If "force" is set to false, then the initializer is only run if it has not already been run for this context, model, and connection in this app domain. This method is typically used when it is necessary to ensure that the database has been created and seeded before starting some operation where doing so lazily will cause issues, such as when the operation is part of a transaction.” [2]

 

Running a test

With everything in place we can now run some tests. Here’s an example where we are testing a customer repository:

namespace Andy.French.Repository.Entity.Framework.Tests
{
    using System.Linq;
    using NUnit.Framework;

    [TestFixture]
    public class CustomerRepositoryTest
    {
        private CustomerRepository _repository;

        [SetUp]
        public void SetUp()
        {
            var context = new TestContext();
            _repository = new CustomerRepository(context);
        }

        [Test]
        public void FindAll_WhenCalled_FindsAllInstances()
        {
            // Arrange

            // Act
            var result = _repository.FindAll();

            // Assert
            Assert.That(result.Count(), Is.EqualTo(2));
        }
    }
}

 

A note on configuration

You will need to add an app.config file to your project. The config file will have to tell Entity Framework which database to use. It might look something like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="entityFramework" type="System.Data.Entity.Internal.ConfigFile.EntityFrameworkSection, EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false"/>
  </configSections>
  
  <entityFramework>
    <defaultConnectionFactory type="System.Data.Entity.Infrastructure.LocalDbConnectionFactory, EntityFramework">
      <parameters>
        <parameter value="v11.0"/>
      </parameters>
    </defaultConnectionFactory>
    
    <providers>
      <provider invariantName="System.Data.SqlClient" type="System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer"/>
    </providers>
  </entityFramework>
  
  <startup>
     <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.1"/>
  </startup>
</configuration>

Note that we are using a SQL Server Express local database (v11.0). In my case the result is I get a couple of database files (.mdf and .ldf) named after the context used by the database initializer in my user directory.

 

image

 

References

[1] DatabaseInitializer<TContext> Interface (MSDN)

[2] Database.Initialize Method (MSDN)

Saturday, 21 June 2014

Integrating StyleCop with TeamCity

Having used CruiseControl.Net for some time I thought it was time to try something new: TeamCity from JetBrains. I’m a bit fussy about code quality so one thing I like my integration builds to do is run StyleCop and fail the build if violations are found.

Create an MSBuild file

After some research I tracked down some basic guidance on StackOverflow and adapted it to my needs [1].

I created an MSBuild file that could be referenced from a umber of TeamCity build configurations. This build file invokes StyleCop, counts the violations and fails the build if StyleCop violations are encountered. I saved the build file to a shared location where it could be used from different builds. Here’s the basic script:

 

<Project DefaultTargets="RunStyleCop" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
	<Import Project="$(MSBuildBinPath)\Microsoft.CSharp.targets" />
    <Import Project="$(ProgramFiles)\MSBuild\StyleCop\v4.7\StyleCop.targets" />
	<UsingTask TaskName="XmlRead" AssemblyFile="C:\MSBuild\lib\MSBuild.Community.Tasks.dll" />

	<Target Name="RunStyleCop">
        <CreateItem Include="$(teamcity_build_checkoutDir)\**\*.cs">
            <Output TaskParameter="Include" ItemName="StyleCopFiles" />
        </CreateItem>

        <StyleCopTask ProjectFullPath="$(MSBuildProjectFile)"
                      SourceFiles="@(StyleCopFiles)"
                      ForceFullAnalysis="true"
                      TreatErrorsAsWarnings="true"
                      OutputFile="StyleCopReport.xml"
                      CacheResults="true"
                      AdditionalAddinPaths="$(ProgramFiles)\StyleCop 4.7\Andy.French.StyleCop.Rules.dll"
                      OverrideSettingsFile="$(teamcity_build_checkoutDir)\Settings.StyleCop" />
                      
        <XmlRead XPath="count(//Violation)" XmlFileName="StyleCopReport.xml">
            <Output TaskParameter="Value" PropertyName="StyleCopViolations" />
        </XmlRead>

        <TeamCitySetStatus Status="$(AllPassed)" Text="StyleCop violations: $(StyleCopViolations)" />

        <Error Condition="$(StyleCopViolations) > 0" Text="There were $(StyleCopViolations) StyleCop violations." />
	</Target>
</Project>

 

On line 2 we import the StyleCop.targets from the StyleCop installation directory. This makes the StyleCopTask available on line 10. If you examine this file you’ll find it references the StyleCop.dll in the StyleCop installation directory. The StyleCopTask is actually in that DLL.

On line 3 we import the MSBuild.Community.Tasks.dll. This is an open source project that adds some useful MSBuild tasks including the XmlRead task on line 19 (see [2] below).

You may have to hop on over to the project GitHub site to grab a release [3]. I downloaded the Zip file and extracted the DLLs that I wanted, putting them in a shared location (C:\MSBuild\lib\ in this case).

The RunStyleCop target does all the work. On line 6 we grab all the C# files in the solution. Note that we are using a TeamCity variable here: teamcity.build.checkoutDir. NB: Don’t forget you have to replace all instances of “.” with “_” if you are using MSBuild.

 

“Make sure to replace "." with "_" when using properties in MSBuild scripts; e.g. use teamcity_dotnet_nunitlauncher_msbuild_task instead of teamcity.dotnet.nunitlauncher.msbuild.task” [4]

 

Now I have some custom StyleCop rules and I like to disable a couple of the default rules. To activate my custom StyleCop rules DLL I had to specify the path to it using the AdditionalAddinPaths attribute on line 16. I also include a Settings.StyleCop file with overridden settings with each solution so on line 17 I set the OverrideSettingsFile attribute to point to that file.

The XmlRead task on line 19 reads the output from StyleCop and makes the result available in a property called StyleCopViolations. This is used on line 23 to report the number of violations to TeamCity using the TeamCitySetStatus task on line 23. Then, on line 25, we throw an error – failing the build – if there are any errors.

TeamCity configuration

It’s quite straight forward then in TeamCity. You need to add a Build Step to your Build Configuration. I set the path to the shared build file created above and specified the target:

 

image

 

Note I have set the path to the shared build file and included the target to run. Here’s an example of a failed build, first in the Projects overview where our status message is displayed:

 

image

 

And in the build results page where the Error message can be seen:

 

image

 

There’s more work to do, for example getting the violations to display better but for now I can get at them via the build log tab.

References

Wednesday, 18 June 2014

Download SQL Server Express

Only a few days ago we were moaning in the office about how complicated it was to dowload the correct version of SQL Server Express. Well, it seems we were not alone.

Scott Hanselman has come to the rescue with an awesome blog post that provides easy links to the various SQL Server Express binaries. Here's Scott's short link to his blog post:

http://downloadsqlserverexpress.com

One for the bookmark list. Thanks Scott!



Thursday, 22 May 2014

Minimising deadlocks with READ_COMMITTED_SNAPSHOT

The problem

We have a system that has a number of separate Windows services that all access a shared database hosted by SQL Server 2008 R2. The system is used to provide online hydraulic modelling in support of a water utility company’s maintenance activities. The services are built around SynerGEE Water, a hydraulic modelling product from DNVGL.

One of the Windows services is responsible for monitoring a model library – SynerGEE models are stored as MDB files on the file system – and when a new or updated model is detected it adds a MODEL record to the database. It also indexes all of the pipes in the model and adds them to a MODEL_PIPE table in the database.

A second service checks the database for new MODEL records and then invokes SynerGEE to perform some hydraulic analysis. The results of this analysis are used to update the MODEL_PIPE records.

We observed that if a number of models had been updated in one go the result was that sometimes a database deadlock occurred when the second service was querying the MODEL_PIPE table. This was because the first service was in the process of adding other MODEL_PIPE records for other models at the same time.

We are using NHibernate for all data access and all database queries or updates are wrapped in transactions with the assistance of the Spring.Net transaction template. NHibernate Profiler was used to confirm that all the transactions were correctly formed and we could see that the transactions were using the READ COMMITTED isolation level.

The solution

Firstly, I did some research around Minimizing Deadlocks and noted that using a row-based isolation level can help. In particular, activating READ_COMMITTED_SNAPSHOT on the database can help by allowing SQL Server to use row versioning rather than shared locks.

“When the READ_COMMITTED_SNAPSHOT database option is set ON, a transaction running under read committed isolation level uses row versioning rather than shared locks during read operations.” [1]

Further research around snapshot isolation levels provided further insight:

“The READ_COMMITTED_SNAPSHOT database option determines the behavior of the default READ COMMITTED isolation level when snapshot isolation is enabled in a database. If you do not explicitly specify READ_COMMITTED_SNAPSHOT ON, READ COMMITTED is applied to all implicit transactions. This produces the same behavior as setting READ_COMMITTED_SNAPSHOT OFF (the default). When READ_COMMITTED_SNAPSHOT OFF is in effect, the Database Engine uses shared locks to enforce the default isolation level. If you set the READ_COMMITTED_SNAPSHOT database option to ON, the database engine uses row versioning and snapshot isolation as the default, instead of using locks to protect the data.” [2]

Bingo! If the READ_COMMITTED_SNAPSHOT database option is set to ON row versioning is used instead of locks.

“Once snapshot isolation is enabled, updated row versions for each transaction are maintained in tempdb. A unique transaction sequence number identifies each transaction, and these unique numbers are recorded for each row version. The transaction works with the most recent row versions having a sequence number before the sequence number of the transaction. Newer row versions created after the transaction has begun are ignored by the transaction…

…Snapshot isolation uses an optimistic concurrency model. If a snapshot transaction attempts to commit modifications to data that has changed since the transaction began, the transaction will roll back and an error will be raised. ” [2]

In our case this looked very interesting because the first Windows service adds new MODEL and MODEL_PIPE records in a transaction and is done with them. The second Windows service then reads the new MODEL and MODEL_PIPE records and updates them in a separate transaction. The chances of an optimistic concurrency issue are minimal. Although the two services are accessing the same table they are not accessing the same rows. Therefore, using row version-based locks would allow the two services to work better together.

So, I enabled READ_COMMITTED_SNAPSHOT on the database [3] and found that the deadlocks no longer occurred.

ALTER DATABASE <dbname> SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
ALTER DATABASE <dbname> SET READ_COMMITTED_SNAPSHOT ON;
ALTER DATABASE <dbname> SET MULTI_USER;

Result!

References

[1] Minimizing Deadlocks, Technet.

[2] Snapshot Isolation in SQL Server, MSDN.

[3] Enabling Row Versioning-Based Isolation Levels, Technet.

Monday, 21 April 2014

BitDefender subscription update (2)

OK, an hour or two after submitting the email to BitDefender in step 9 of my previous post I received a reply saying my automatic renewal for BitDefender Internet Security 2014 has been cancelled. Praise be!

BitDefender, if you are listening, please change the way you manage automatic renewals of your product. Allow users to cancel their subscription or automatic renewal from the product page as easily as they can renew them. You are operating in a sector that requires the trust of your users. Engender that trust by making the process honest and transparent. I think your business will benefit from it. 

Sunday, 20 April 2014

BitDefender subscription update

OK, here’s an update to the situation regarding my BitDefender subscriptions (see previous post Are BitDefender (Avangate/Digital River) behaving like confidence tricksters?).

I had a couple of emails waiting for me this morning. There has been some movement, but not on everything.

Step 8 - Subscription cancellation (huzzah!)

BitDefender have cancelled my “BitDefender Internet Security 2013 subscription”. Thank you BitDefender. Thank goodness for that.

However, I note that they have not cancelled the automatic renewal of the BitDefender Internet Security 2014 subscription so I’m going to have to do that separately again. I’m going to use the email address I was given on Twitter (see below).

Step 9 - Twitter power

Well, yesterday I popped an update on Twitter and it looks like BitDefender heard me. I’m now going to contact them using the email address they provided (bitsy@bitdefender.com) to get the automatic renewal of the 2014 subscription cancelled. Let’s see what happens then.

bd21
bd22

Saturday, 19 April 2014

Are BitDefender (Avangate/Digital River) behaving like confidence tricksters?


I have been a user of BitDefender by Digital River for a few years now. In January of this year I ‘upgraded’ my BitDefender installation to BitDefender Internet Security 2014 subscription. I was surprised to find that today – 19th April 2014 – I was charged £40.46 GBP because my subscription had been renewed automatically.

I don’t want auto-renewal of anything and I couldn’t understand why I was being charged again with 281 days left on my subscription.

What I discovered is that the subscriptions for the old versions of the product are still in place and are being automatically renewed and I can’t cancel them!

This seems to be a trend amongst the anti-virus vendors. I had a similar experience with Kaspersky. These are companies that are operating in an environment where you are inclined to trust them. After all, they are working to protect you, aren’t they? What they are actually doing is making you accidentally sign up for automatic renewals (there was probably some small print and an inconspicuous checkbox on their payment page) and then not letting you cancel the subscription or making it very hard to do so.

This post is a description of all the steps I’ve taken to try and cancel the automatic renewal. At this point all attempts have failed but I’ll update this post if I succeed. Please read on and make your own minds up as to whether BitDefender are behaving like confidence tricksters.

Suffice is to say, I would advise anybody to avoid BitDefender like the plague.
  

Step 1 – Do some checks

OK, so BitDefender shows me that I have 281 days left.

bd1

So, lets hop over to their website and see what gives. I log on to my account and head over to the product page and this is what I see:

bd2

Looks like all my previous products are still active. I don’t use them anymore because I’ve upgraded to the 2014 version so how can I cancel the automatic renewals? Well, not on this page and there are no instructions here either.

Isn’t it reasonable to expect to see a button allowing you to cancel a subscription? After all they are very keen for you to renew. Even a little link next to each subscription would be a help. 
 

Step 2 – Check the email

The automatic renewal email I received from BitDefender had some instruction about how to follow up:

bd3 
 

Step 3 – Find my order

So I head over to www.findmyorder.com to see what gives. The site requires an order number and a password. Luckily the email from BitDefender included an order number so a put it in along with my account password:

bd4

What? Incorrect order number and/or password?

OK, lets try the ‘forget your password link’ to see if it’s the password. This gives me a form asking for the order number again. No problem, I enter the order number, click Submit and it sends me an email.

bd5

The weird thing is the password is completely different to my account password (by the way, I strongly suspect everyone is getting the same password back for this page). Never mind, maybe it’s me. Let’s enter the password and see what happens:

bd7

Great, we can manage a subscription. Let’s click the link…

Step 4 – Manage subscription (not)

Ah, another login.

bd8

Never mind. Let’s try logging in.

bd9

Now take my word for it, it doesn’t matter what password I use (my BitDefender password or the one they sent me in the email previously) I get the same thing: “Enter a valid email address”.

It could be that it’s bad validation but by now I’m getting suspicious.

Step 5 – Contact support

So, back on the product page from step 1 I use the Support link and get this dialog:

bd10

OK, where’s the 2014 product (I have a 2014 subscription listed on the product page)? And take my word for it clicking “the full list of products” link doesn’t list it either. Oh well, let’s use the 2013 version for now and see what gives.

bd11

I click FIND HELP and get a useful looking result:

bd12

Let’s click the link:

bd13

Oh good, another link. OK, here goes:

bd14

What you get is a nice form to fill in. The problem is, there’s no 2014 version of the subscription listed but the automatic renewal I want to cancel is listed as BitDefender Internet Security 2014, not 2013.

Anyway, I have submitted this form a number of times, once for each ‘version’ I have asking for automatic renewals of my subscriptions to be stopped. I have also submitted an extra one listing all 3 of the products I own asking the same.

Now I don’t know where this form goes but I haven’t even received an automated response and as far as I can see nothing has happened.

But really, why do I have to go through all these pages to try – and fail – to cancel a subscription or an automatic renewal when all subscriptions are listed on my product page? Why oh why can’t I do it there? Why should I have to contact support for this?

Anyway, this hasn’t worked so what can I do now?

Step 6 – eHow makes a suggestion

Getting desperate I do a Google search with Bing and find a link to an eHow page that suggests going to http://shop.BitDefender.com and completing some simple steps. Now the steps aren’t right; it looks like the article is out of date. So, I ended up clicking Contact Us at the top of the page and then SUPPORT but that gets you right back to the support page from step 5. Bummer.

However, if you click “My BitDefender” at the top of the page you get something that looks quite useful:

bd15

But yet again there are no ways to cancel subscriptions or automatic renewals.

But look, there’s a support link at the top of the page, I wonder where that goes:

bd16

Well, it goes nowhere. You stay on the same page.

Step 7 – The mystery page

OK, now I can’t remember how I found this page. It looks like the page that the eHow article was suggesting in Step 6 but I can’t remember how I found it.

Anyway, I completed the form but it absolutely will not submit it because it tells me that no orders were found! Now remember this is using the order number I was given in BitDefender’s automatic renewal email. Funny that. The same order number worked in step 3.

You really do have to question if any of this is accidental.

bd17 

Step 8 – Email customer support directly

Well, on the mystery page in Step 7 there was an email address listed (customerservice@bitdefender.com). So, I sent an email to that address.

bd18

OK, guess what came back. Now remember this was an email address taken from a publically accessible page on the BitDefender web site.

bd19

What a surprise.