The first day was a session on Rapid Testing presented by Michael Bolton (no, not that Michael Bolton).  It was a shortened version of a three day session he usually presents.  It was a fantastic presentation.  Michael is a great speaker and great teacher.  I have been reading his articles in various magazines for a few years now.  It was great to see him in person and learn from him.  He talked about rapid testing techniques, new ways to look at exploratory testing and how to find the highest risk defects fast.

The keynote speech was from Rob Sabourin on testing in an scrum environment.  Like Michael, I have read many of his articles as well.  He is a dynamic (shall we say animated?) speaker.  He has a wealth of experience in software testing and is willing to share.  As an added bonus, I got to ride with him on the shuttle to the hotel and from the hotel back to the airport.  I greatly enjoyed the conversation.  Thank you for sharing Rob.

There were some other great speakers there as well including Matt Heusser and Hans Buwalda.  They also set up plenty of time during the conference for networking.

It was a terrific conference.  I have been to general software development conferences before  but this is the first one specific to software testing.  Software testing seems like a relatively unknown portion of software engineering.  Sure, there are lots of articles on test cases and test plans.  But really looking at what software testing entails, what a good tester does when hunting that elusive bug, what separates the average tester from the great tester is a vast field ripe for exploration.  This conference renewed my excitement to join in the discovery.

From what I gathered from David Laribee’s article, I like this. I REALLY like this.

By changing namespace, class, and method names, the behavior of each spec (test) is clearly revealed. In a larger suite of tests, if there are failures, I can more easily see what is failing and what trends, if any, are resulting in my failures. Before, I would name my test method names to describe a behavior, "Some concept should do this when given that." Getting the namespace and class name involved provides a more legible test result.

While reviewing a test case for a peer, I took this approach and arrived at the following scenario. I am changing the names of the methods to hide details. The original test method was "FilterToEndPointTestReport." Making this more readable, using my previous approach, we arrived at "FilterShouldUpdateStatusOnReportWhenDefaultFilterValueIsChanged." Great, that is better, but…

Applying the approach described in David’s post, I now have this.

   1: namespace Specs_for_Filters

   2: {

   3:   public class When_default_filter_value_is_changed : FilterTestFixture

   4:   {

   5:     [Test]

   6:     public void Endpoint_should_be_updated_on_report {}

   7:     

   8:     [Test]

   9:     public void Endpoint_should_be_updated_on_grid {}

  10:     

  11:     [Test]

  12:     public void Endpoint_should_be_updated_on_chart {}

  13:   }

  14:   

  15:   public abstract class FilterTestFixture

  16:   {

  17:     // provide common functionality that will be used 

  18:     // among behavior-driven filter test classes.

  19:   }

  20: }

When my tests run, their resulting output is more legible. This approach helps me to think about my tests in terms of behavior rather than implementation. In my opinion, my test cases are now more abstract and more succinct.

After seeing too many duplications across methods in one or more classes, I decided to investigate a way to remove these. I am always looking to remove duplicate code, even code that shares similarities, I look to refator. Removing duplications is important to adapt more easily to change. When a code change is required, we should only have to make it in one place. The following article will show two relatively simple means to address this code smell, delegates and an Aspect-Oriented approach.

The example I am using in this article is seen below. It is a unit test class, StackFixture class, and it is extremely trivial. This is a typical unit test taken from Test-Driven Development in Microsoft .NET. I have added the logging functionality as an easy means to show how these types of DRY violations can occur. While the duplications in this example deal with logging, they could pertain to other, more complex, functionality as well. The approaches to removing duplications across methods in this article can work with these simple scenarios, as well as more complex ones.

Take for example this sample test case. Each test method logs a message before and after executing the test logic. This violates DRY. We want to keep our test logic in our method, but factor out the duplicate logging. Again, if we were not logging, but doing some repetitive logic, we could factor that out as well.

   1: [TestFixture]

   2: public class StackFixture : AbstractBaseFixture

   3: {

   4:     private Stack stack;

   5: 

   6:     [SetUp]

   7:     public void SetUp()

   8:     {

   9:         stack = new Stack();

  10:     }

  11: 

  12:     [Test]

  13:     public void Empty()

  14:     {

  15:         Log.Info("Starting Empty test");

  16:

  17:         Assert.IsTrue(stack.IsEmpty);

  18:

  19:         Log.Info("Ending Empty test");

  20:     }

  21: 

  22:     [Test]

  23:     public void PushOne()

  24:     {

  25:         Log.Info("Starting PushOne test");

  26: 

  27:         stack.Push("first element");

  28:         Assert.IsFalse(stack.IsEmpty, "After Push, IsEmpty should be false.");

  29: 

  30:         Log.Info("Ending PushOne test");

  31:     }

  32: 

  33:     [Test]

  34:     public void Pop()

  35:     {

  36:         Log.Info("Starting Pop test");

  37: 

  38:         stack.Push("first element");

  39:         stack.Pop();

  40:         Assert.IsTrue(stack.IsEmpty, "After Push - Pop, IsEmpty should be true.");

  41: 

  42:         Log.Info("Ending Pop test");

  43:     }

  44: 

  45:     [Test]

  46:     [ExpectedException(typeof(InvalidOperationException))]

  47:     public void PopEmptyStack()

  48:     {

  49:         Log.Info("Starting PopEmptyStack test");

  50: 

  51:         stack.Pop();

  52: 

  53:         Log.Info("Ending PopEmptyStack test");

  54:     }

  55: }

Remove Duplications with Delegates

Quite simply, we can remove our method duplications using a delegate, in this case, the System.Action delegate. Thanks to Ayende for helping me understand this option. We create a method, TestMethod (as shown below), and add it to our StackFixture class. The method takes two strings (string beforeMessage and string afterMessage) and the Action delegate, representing the test method logic. We will call this method within our test cases.

   1: public void TestMethod(string beforeMessage, string afterMessage, Action action)

   2: {

   3:     // before

   4:     Log.Info(beforeMessage);

   5:

   6:     // invoke our method

   7:     action();

   8:

   9:     // after

  10:     Log.Info(afterMessage);

  11: }

Then, in our tests, we replace the following test method…

   1: [Test]

   2: public void Pop()

   3: {

   4:     Log.Info("Starting Pop test");

   5: 

   6:     stack.Push("first element");

   7:     stack.Pop();

   8:     Assert.IsTrue(stack.IsEmpty, "After Push - Pop, IsEmpty should be true.");

   9: 

  10:     Log.Info("Ending Pop test");

  11: }

With the same method using our delegate approach.

   1: [Test]

   2: public void Pop()

   3: {

   4:     TestMethod("Starting Pop test", "Ending Pop test",

   5:                delegate

   6:                    {

   7:                        stack.Push("first element");

   8:                        stack.Pop();

   9:                        Assert.IsTrue(stack.IsEmpty, "After Push - Pop, IsEmpty should be true.");

  10:                    });

  11: }

We can replace each of our test methods with the same TestMethod using the System.Action delegate. It will then easy to omit our before and after strings replacing them with “action.Method.Name,” or some other intelligent logic to determine what to log.

This approach works well, but delegates are often difficult to understand. If this solution suits your needs, make sure the implementation is easily maintainable. What is nice about this approach is how simple it is. There is no need to reference any assemblies outside of the .Net framework, i.e. no third party dependencies.

The Aspect-Oriented Approach

Aspect-Oriented Programming (AOP) is an entirely different animal. It certainly deserves more than just a blog post. Please read more about it online. This article will not explain the details of AOP.

If you are reading this, welcome back. I will assume you are now familiar with AOP. There are several options for AOP frameworks. Some frameworks I have used are Spring.net, Castle Project, and PostSharp. The former two provide IoC capabilities as well. I suggest using a well-supported framework, one with community support and frequent updates. Investigate for yourself, there are several nice options available.

AOP with Castle DynamicProxy

Hamilton Verissimo put together a good sample using Castle’s DynamicProxy. It is a nice, clean implementation.This worked fine for me and would work well in other situations. However, in my scenario, if I were to use this approach, I needed to have each of my test classes extend from MarshalByRef. In addition, I would need to modify the underlying NUnit framework to create my proxies in order to provide advice. Since I could not do the latter, or did not want to investigate, I searched for other AOP options.

The DynamicProxy aproach to solving your AOP needs is a great option. It just did not work in my situation, only because NUnit executes each of my methods. I would need to somehow intercept the creation of my test class, create a proxy of it, and execute the test methods on it.

AOP with PostSharp

Gael Fraiteur’s PostSharp is a great option for AOP needs. It is extremely clean and easy to use. It is the simplest AOP framework I have used. I suggest watching Gael’s video tutorial.

The first thing I needed to do, besides downloading and installing PostSharp, is to add two references to my project, PostSharp.Laos and PostSharp.Public. After that, I create a simple class extending OnMethodInvocationAspect, called LoggingMethodInvocationAspect.

   1: [Serializable]

   2: public sealed class LoggingMethodInvocationAspect : OnMethodInvocationAspect

   3: {

   4:     private ILogger log;

   5: 

   6:     public LoggingMethodInvocationAspect(ILogger logger)

   7:     {

   8:         log = logger;

   9:     {

  10: 

  11:     public override void OnInvocation(MethodInvocationEventArgs eventArgs)

  12:     {

  13:         // before

  14:         log.Info("OnInvocation Before proceed");

  15: 

  16:         // invoke

  17:         eventArgs.Proceed();

  18: 

  19:         // after

  20:         log.Info("OnInvocation After proceed");

  21:     }

  22: }

At this point, all we need to do is apply our aspect on target methods. The “AttributeTargetMembers” attribute can use wildcards. This tells the AOP framework to only intercept those methods in the “MathService” class that start with “Test.” Below is the sample where I specify the target assembly, target type (class or classes to intercept), and target methods to intercept. There are many features beyond what I am showing so do further investigation.

   1: [assembly: ClassLibrary.SampleInterfaces.LoggingMethodInvocationAspect(

   2:     AttributeTargetAssemblies = "ClassLibrary.UnitTesting.Sandbox",

   3:     AttributeTargetTypes = "ClassLibrary.UnitTesting.Sandbox.MathService",

   4:     AttributeTargetMembers = "Test*")]

Finally, my StackFixture class now looks something like this. Notice that the duplicate logging logic is now removed and each test method is prefixed with “Test” in the method name. This is so that PostSharp can filter what methods to intercept.

   1: [assembly: ClassLibrary.SampleInterfaces.LoggingMethodInvocationAspect(

   2:   AttributeTargetAssemblies = "ClassLibrary.UnitTesting.Sandbox",

   3:   AttributeTargetTypes = "ClassLibrary.UnitTesting.Sandbox.MathService",

   4:   AttributeTargetMembers = "Test*")]

   5:

   6: [TestFixture]

   7: public class StackFixture

   8: {

   9:     private Stack stack;

  10: 

  11:     [SetUp]

  12:     public void SetUp()

  13:     {

  14:         stack = new Stack();

  15:     }

  16: 

  17:     [TearDown]

  18:     public void TearDown(){}

  19: 

  20:     [Test]

  21:     public void TestEmpty()

  22:     {

  23:         Assert.IsTrue(stack.IsEmpty);

  24:     }

  25: 

  26:     [Test]

  27:     public void TestPushOne()

  28:     {

  29:         stack.Push("first element");

  30:         Assert.IsFalse(stack.IsEmpty, "After Push, IsEmpty should be false.");

  31:     }

  32: 

  33:     [Test]

  34:     public void TestPop()

  35:     {

  36:         stack.Push("first element");

  37:         stack.Pop();

  38:         Assert.IsTrue(stack.IsEmpty, "After Push - Pop, IsEmpty should be true.");

  39:     }

  40: 

  41:     [Test]

  42:     [ExpectedException(typeof(InvalidOperationException))]

  43:     public void TestPopEmptyStack()

  44:     {

  45:         stack.Pop();

  46:     }

  47: }

The end result, a StackFixture test class with duplicate logging logic removed. The PostSharp AOP framework post-processes the compiled assembly and inserts itself, easily providing points of interception.

The PostSharp approach does involve a third-party dependency, but I feel like it is cleaner than the delegate approach. Gael informed me future versions of PostSharp will provide a more configurable solution than adding the [assembly ...] reference for the aspects, perhaps an XML configuration option. This can be done today, but involves some work.

Whether you use delegates, AOP, or some other approach, remove duplicate code wherever possible. I am happy to use either approach in my projects. There are tradeoffs to either solution, so investigate for yourself.

The problem

You have two methods (Foo and Bar), and Foo calls method Bar.  It will look like this:

public int Foo()
{
    // stuff happens here
    int i = Bar();
    // do more stuff here    return i + 5}

public int Bar()
{
   // do some logic
   return 1;
}

Now you need to test both methods.  Bar can be easily tested like this (this should work for NUnit and MBUnit tests — assuming both methods exist in a class call MyClass):

[Test]
public void Bar_Test()
{
    MyClass c = new MyClass();
   int result = c.Bar();
   Assert.AreEqual(1, result);
}

Now we test Foo:

[Test]
public void Foo_Test()
{
   MyClass c = new MyClass();
   int result = c.Foo();
   Assert.AreEqual(6, result);
}

Here is where we have an issue.  What if the result of Bar changes?  That will break the test.  So now we have a brittle test.  Any time Bar changes, we have to change all of the Bar tests, but also the Foo tests, so we have added complexity to our tests as well.  Now if Foo (or Bar) was in a separate class, this wouldn’t be an issue, but they are in the same class.  And as much as I like small classes, this seems too trivial to split up (even in the context of an extremely trivial example).

The Solution

What I would like to do is Mock the method Bar.  You can do that with a delegate.

Without really getting into what a delegate is, I will just show the code.  So here are my two methods, with a delegate added in for good measure.

public delegate int BarDelegate();

public BarDelegate Bar;

public MyClass()
{
    Bar = BarInternal;
}

public int Foo()
{
    // stuff happens here
    int i = Bar();
    // do more stuff here    
    return i + 5;
}

public int BarInternal()
{
   // do some logic
   return 1;
}

OK, here is what I did:

1. Renamed Bar to BarInternal
2. BarDelegate delegate
3. Created a variable named Bar to point to the method BarInternal
4. Created a constructor to hook up Bar to BarInternal.

Now all of my code can still use the method Bar just like it was there, but in actuality it is calling BarInternal.  I have now created one layer of indirection to my method Bar so it can now be mocked and tested.

Here is my new test for Foo:

[Test]
public void FooTest()
{
    MyClass c = new MyClass()
    c.Bar = BarMethodForTest;
    int result = target.Foo();
    Assert.AreEqual(7, result);
}

private int BarMethodForTest()
{
    return 2;
}

So what is new with the test is the BarMethodForTest that is passed to c.Bar and I have coded BarMethodForTest to always return 2.  When I set c.Bar to my new method, BarInternal (the default method) is no longer called.

Actually, if I wanted to I could simplify this further by setting c.Bar with an anonymous delegate like this:

[Test]
public void FooTest()
{
    MyClass c = new MyClass()
    c.Bar = delegate() { return 2; }
    int result = target.Foo();
    Assert.AreEqual(7, result);
}

Stepping Back

OK, take a step back and look at this again with a critical eye.  Have I really solved the problem?   Yes and no.  The real problem was that my methods were too complex to simply test.  Now my methods are simple to test, but the code is complex to read.  All I’ve done is shifted the complexity from one class to the other.  A better solution would to have created another class and further abstracted out the logic.

At the same time, my class is not overly burdened by the change either.  In lieu of a real refactoring, I do consider this a viable solution to the problem.  So by all means do this — but make sure you are out of alternatives first.

Read anything by you can get your hands on written by Scott Barber (http://www.perftestplus.com/scott_blog.php). Start with this one:

Performance Testing Guidance for Web Applications http://msdn2. microsoft. com/en-us/ library/bb924375 .aspx. It is a Microsoft document written by Scott Barber and others that gives a great overview of the process of performance testing. It is not technical and is not tool specific. It gives you an idea of the steps involved and things to plan for in your performance testing efforts.

Next, read this one:

User experience, not metrics http://www.ibm. com/developerwor ks/rational/ library/4228. html It goes deeper into performance testing. It is written around the IBM Rational toolset, however you can use the information against any performance toolset.

Then go to this one:

Beyond performance testing http://www.ibm. com/developerwor ks/rational/ library/4169. html

For books, I would recommend Performance Testing Microsoft .NET Web Applications by the Microsoft ACE team, especially if you are testing products developed using Microsoft tools. It is written around the older performance testing tools from Microsoft, but there is lots of good info.  There seem to be a couple other performance testing books out there, but I have not read them. As I do, I’ll post book reports.  A good book on statistics can hurt either.

Can anyone out there in the blogosphere comment on books you have read related to performance testing?

For tools, I would recommend playing around with the freely available tools out there.   For example:

Apache JMeter: http://jakarta. apache.org/ jmeter/

Radview Webload: http://www.webload. org/

The Grinder: http://grinder. sourceforge. net/

OpenSTA: http://opensta.org/

Also look at the pay-for tools. You may not use them, but get to know the products, what they do and how they do it. Some examples:

Borland Silk: http://www.borland. com/us/products/ silk/silkperform er/index. html

HP (Mecury) LoadRunner: https://h10078. www1.hp.com/ cda/hpms/ display/main/ hpms_content. jsp?zn=bto&cp=1-11-126- 17%5E8_4000_ 100__

Empirix ELoad: http://www.empirix. com/index. asp

IBM Rational Performance Tester: http://www-306. ibm.com/software /awdtools/ tester/performan ce/

Microsoft Visual Studio Team Systems Testers Edition: http://msdn2. microsoft. com/en-us/ vsts2008/ products/ bb933754. aspx

Happy testing,

Scott

1. Specifications are typically written before code as instructions for what the code SHOULD do.
2. Tests, (written before or after the code), are a set of instructions detailing what the code MUST do. If we can’t pass the tests, then we cant release the product.
3. Specifications have no enforcement mechanism beyond human beings.
4. Tests may be automated and the enforcement of them may be as well. Without passing a suite of regression tests, for example, the software under test (SUT) may not be deployed to the drop point on a network.

Tests act as a gate keeper. Regardless of how few or poor our tests are, they serve to ensure the software meets the specifications. This is why we refer to passing a test as PASSING. Passing a test means that we can get beyond it. Functional requirements written in a document have no such enforcement ability.

Ideally, tests are an expression of the functional requirements such that we have a validation tool for the functional requirements. If this is true, are not tests the functional requirements in an executable form?

This is why tools like Fitnesse exist. Can we as developers accomplish the same level of testing with a Unit Testing framework? You bet, but the frameworks don’t typically provide the nice syntax and user experience business users need to create the tests. This is exactly what Fitnesse strives to provide, a way for business users to specify requirements in an automat-able form.

Some Other Observations

• Tests, while ideally written before code, still rarely are. Most tests are written after the code is written. These tests include regression tests, acceptance tests, unit tests, etc. All kinds of tests are at play here.
• A MS Word document can’t break the build.

If these things are all true, then how can we possibly consider the tests anything BUT the functional requirements. They may not have that label at the top of the page, but tests fundamentally define acceptance of the system.

I just don’t understand how this doesn’t compute for some folks.

The project is interesting and the people are pleasant, not a bad gig at all.  So far I’m manage to put source control in place (SubVersion via the VisualSVN Server — they really have the best install around for windows), they are getting on board with TDD, I’m slowly working in User Stories into their process (by osmosis really), and I decided to try to go the extra mile and get a bit of Continuous Integration going.

In the past I would have hit up CruiseControl, struggled for a few days, swore, kicked a couple of dogs, and then get everything working.  This time I thought I might be able to spare the dog by trying TeamCity from JetBrains.  Result, the dog got an extra treat this week. 

I got the entire friggen process up and working in about three hours.  And that was on the first try!   Most of that time was spend fixing dependency issue in the projects (which were my fault, not TeamCity’s — that is why you want one of these things after all). 

The first process I set up was to create a test build.  The test build did these things:

1. Monitor Subversion to changes in my project, if there are changes, downloaded them and proceed to step 2.  Otherwise: repeat step 1.
2. Compile all the source.  I chose to build the project from the solution file.  If this were an internal project I might use a Nant script, but considering the client I just stuck with the solution file.
3. Run all unit tests. Report any failures.
4. Let me know all is well.

For #4, I had to log onto the TeamCity web site (created by the install), and install the client task tray application (similar to CC.Tray for CruiseControl really).  The only annoyance with any of the tools is that the TeamCity tray application doesn’t make any noise, and it isn’t very configurable (like CCTray is).

But, for an extra bonus step I set up a second build to recreated the test web site on command.  This second build took 30 minutes.  The TeamCity’s configuration web site has an option to copy an existing build.  I made a few modifications and I was good to go.   The two modifications were: change the build directory, and no to monitor subversion for changes.  I want this build to be kicked off manually.

So now, the only part of the process that is not automated is the move to production.  I have to make web.config changes for that (meaning I have to do some custom msbuild work to swap connection strings).  But all in all I am very happy with where things are at considering the amount of time spent.

CruiseControl might be more configurable and have a better tool set, and for a different project on a different team, I might go down that road again.  But for an inexperienced team, or for a project with non-demanding build requirements, TeamCity is an excellent tool.

I am officially a TeamCity convert.

This presentation consists of a Power Point deck with minimal slides (it was for a code camp) and a sample project written MVP style (WinForms – not web forms).  That is where the meat of the presentation was, the slide deck is just there to give a taste of what mock objects are supposed to do.

Anyway, the big change was that I updated the sample projects mocking code to Rhino Mocks 3.3 syntax.  This uses the Record and Playback methods now.  But I did leave one test method using the old syntax for comparison sake.

For all of the tests to run you will have to download and install the Adventure Works LT database from CodePlex.

Download the presentation here.

Enjoy.

Clicking on the Test run error link reveals the following error.

Failed to queue test run ‘starr@TEXAS 2007-11-29 21:13:48′: Test Run deployment issue: The location of the file or directory ‘c:\projects\bowlingkata\bowlingscorelib_test\bin\debug\Rhino.Mocks.dll’ is not trusted.

The code coverage tool wants to be able to run in full trust mode. Makes sense, actually. So what’s the quickest way to patch this up an keep moving? Since Rhino.Mocks.dll is strongly named and signed and all, this turns out to be the easiest thing to do for me:

I am running my Unit Tests with Code Coverage turned on in my test configuration and Rhino Mocks is helping me increase code coverage. What is it, you ask?

Technorati Tags: Rhino Mocks,TDD,Visual Studio

EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.1830, P3 42435be1, P4 mscorlib, P5 2.0.0.0, P6 4333ab80, P7 bd0, P8 59, P9 system.nullreferenceexception, P10 NIL.

Searching for an error this cryptic reassured me that I need to take measures to improve the details of my error handling. As a result, I immediately created two HttpHandlers to deal with unhandled exceptions such as this one, and one for more generic error handling. By logging more detailed information, I can better detect the error and fix it. This is nothing new, but definitely important. A good article on reporting unhandled exceptions can be found here, ASP.NET 2.0 Unhandled Exception Issues.

After implementing this solution, my error reared its ugly head again, and a detailed error log was recorded. Perfect! As I am reading the detailed error message and stack trace, I am finding myself even more confused. Nothing in the stack trace could be found in my code. After another internet search, I found one dedicated thread specifically for this error. The error details are below.

System.NullReferenceException: Object reference not set to an instance of an object.
    at System.Web.SessionState.SessionStateModule.PollLockedSessionCallback(Object state)
    at System.Threading._TimerCallback.TimerCallback_Context(Object state)
    at System.Threading.ExecutionContext.runTryCode(Object userData)
    at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Threading._TimerCallback.PerformTimerCallback(Object state)

   
Closing the chapter on this error, the only thread I found with details on this can be found here, Null Reference Exception in SessionStateModule. The fix turned out to be a Microsoft hotfix that I had to install on my servers that basically updated the aspnet_wp.exe, System.Web.dll, and webengine.dll. With the fix in place, so far so good – no signs of the error after three weeks of peak usage.

Lessons Learned

Errors that occur outside of the main application context will cause the application pool to be recycled in ASP.NET 2.0 by default. While this default can be disabled, it is a better approach to throw more detailed information about each error. Using the unhandled exception HttpModule will help isolate errors that originate on threads outside of the application’s context. When catching errors, provide as much detailed information as possible to give yourself a fighting chance.