A few days ago, I started development on a new project and I am using it to learn some new things. I have been working with Visual Studio Team System 2008 + ReSharper exclusively for awhile now, and I want to try some other tools.
Even though learning some new tools and coding practices was my goal in the beginning, I found myself absolutely paralyzed by the number of choices for every tool along the way. I still don’t think I have an ideal setup, but I’’ tell you what I have experienced so far.
Unit Test Framework
I considered these alternatives:
I figured the tooling support for NUnit would be the best given its maturity. The test runner in ReSharper supports it out of the box, so what could go wrong?
UnitTest framework – No winner, see below
Code Coverage
I love VSTS code coverage and complexity analysis. With that history, I knew good and well that I needed a good code coverage tool to help me with writing effective unit tests. So, I went the only real place I knew to go, NCover.
Well, after some time futzing with NCover versions, I just cant seem to get it to work. Here is the error it gives me when I try and run it in Visual Studio:
—— Test started: Assembly: RoboDojo.Core.Specs.dll ——
NCover couldn’t create a coverage report.
After some Googling, it seems there are several potential causes of this problem, but I tried all the known fixes with no good result. Further, the user experience of NCover simply didn’t seem to provide the tight integration I get from VSTS.
Because my need for a coverage tool is so acute, I went back to MSTest.
NCover – Fail
TestDriven.NET
Somewhere along the way, I figured out I could get a copy of TestDriven.NET for free for use on open source and individual project, which this is. So, after installing TestDriven.NET I have the ability to execute my MSTest unit tests with a different keystroke, but I really don’t see that it buys me anything.
When installing TestDriven.NET, it laid down yet another version of NCover, which produced the same behavior as the standalone version I downloaded earlier.
It will run my MSTest tests with the VSTS code coverage tool, so that’s nice, but at the end of the day I think all I get from TD.NET is some different ways to execute unit tests.
TestDriven.NET – Fail, but only because of NCover
ReSharper
ReSharper is the ubiquitous-must-have-it tool for VS and hardly bears explanation. I did find Sneal, a unit test runner adapter that lets me run my MSTest unit tests with the ReSharper test runner. This is a big win because ReSharper’s unit test runner beats both VS’s and TD.NET’s.
The Sneal plugin has some issues though and I will be trolling for another plugin. Overall, it is win, though.
ReSharper – Win
Source Control
SVN. No experimentation here. I am tired already, so I am sticking with a known quantity.
For clients, I am using AnkhSVN integrated into Visual Studio and TortoiseSVN. Frankly, I don’t get the argument that non integration into the IDE is a good thing. I find having to flip back and forth between Windows Explorer and Visual Studio to be high friction.
Tortoise, is great, but integration lets me add and delete classes without worrying that I will break the build.
AnkhSVN will sometimes get into a loopy state wherein it thinks I have mutliple working copies of my project going on. This happens when deleting or renaming directories and the only fix seems to be restarting VS.
SVN, AnkhSVN, Tortoise SVN – Win
Continuous Integration
Although I have good experience with CruiseControl.NET + NAnt, I really just wanted a simple solution file based build and test execution. Nothing fancy.
At the prompting of a friend (thanks, Chris) I decided to try TeamCity, and I am glad I did. Easy setup. Nice integration. Easy configuration. Talks to my hosted SVN repo with no problems and uses my Gmail account to send build notifications.
TeamCity – Win!
Conclusion
You know, VSTS has its issues and they have been vetted publicly and passionately. Frankly, I am not unhappy with the tooling I am getting and will probably uninstall NCover, NUnit, TestDriven.NET and stay with what I am getting from VSTS.
That is, unless you can tell me I have done something wrong or something else I just MUST try. What are you running on your machine that makes life easier?
What about PartCover for code coverage? Last week i were looking for a free alternative for NCover and found it, but i haven’t tried yet:
http://sourceforge.net/projects/partcover/
Anyone on this?
I tried PartCover a couple of months ago. It seemed to work fine for a while and then it just stopped working. Also, there is little or no activity on the project. It’s a pity.
if i had to guess your issues with ncover are probably bitness issues. i’d guess you’re on a 64 bit dev machine and the free ncover only does 32-bit, my simple workaround is to make your test assemblies x86 only and then you’ll get coverage…
Just finished reading the part about “NCover – Fail”. Since I work on NCover, I’d love to help you work through the problems if you’re still interested, and also do whatever is needed to prevent them in the future. By the way, what does “tight integration with Visual Studio” mean to you?
Finalbuilder is a nice buildserver that lets you automate almost everyting : it can deploy to a virutal machine, set up an IIS site, … almost everything
@bob, You are spot on with the 64 bit issues. I am trying again. Thank you.
@Stephen, I am working with Daniel at NCover and he is helping me out, so I will get an new report up as soon as things come together.
Tight integration: The real bottom line is that I want to be able to see code files with color highlights of covered, not covered lines in the source control editor. I have no idea at this point if NCover does this, but VSTS does and it is huge.
@Tom – Agreed on FinalBuilder. Awesome product. I don’t need it for this little thing I am doing but in a team environment or enterprise application of any size, that is a huge win.
I use UltraEdit for a lot of things that VS cannot do, it’s a must have on my development machine.
A few more pros and one con for these tools…
TestDriven:
Results in Output pane, rather than a window.
If all tests passed, the only message I see is xx Tests Passed, 0 failed (I don’t need to see a green indicator for every test.)
On failed tests, double-clicking on the noted exception (with a line number) takes you to that line in your code.
The ability to rerun a test, no matter where you are in the code.
TeamCity:
Once you get NCover running, integrating it with TeamCity is fairly straightforward.
With a little work, you could probably integrate the output from MSTest as well (using trx files)
MSTest:
– My biggest gripe with MSTest and the MSTest runner is all the files/folders it creates with each test run
When running MSTest with TestDriven, these files are not created/retained.
TestDriven: very fast. much less overhead than either the R# runner or MStest runner (imho of course).
ide integration: If the only dev tool I used was Visual Studio then maybe having everything in the solution wouldn’t be a big deal, but I use all sorts of tools, and I have files and folders that I don’t necessarily want to be strictly under the control of visual studio for the scc integration to work. [Yeah, I know, next we’ll be arguing about which text editor is better.]
I know that the SCC bindings in vs08 are much improved, so maybe I should Give Ankh A Chance and see if my opinions change. 🙂
I’ll end with this: I think its funny that something I am often annoyed by (IDE integration) is, to somebody else, a terrific thing. You just have to love a line of work which allows such an incredibly wide expanse of opinions!
Thank you for the post and for the information about Sneal – I will check it out.
@David it really sounds like NCover is the real reason you dismissed everything else. I personally never had any serious problem running NCover. Sometimes it was tricky to get it right with NAnt and cc.net, but nothing impossible or impractical.
@Tony, if you have not tried AnkhSvn since the 2.0 release, especially the recent update in the last few days, you should give it a second look. It’s got nothing to do with v1.0, forget that thing 🙂
Hi,
Can you please contact us on the AnkhSVN problem you refer to in your mail. (It is not a known or earlier reported issue, but we like to have more details so we can resolve it anyway).
Thanks,
Bert
(One of the AnkhSVN developers)
Update: I just got an NCover license and will be comparing that to VSTS code coverage out of the box in the next few days.
I use VisualSVN and love it.
http://www.visualsvn.com
It sits on top of tortoise svn and gives you the IDE integration you want. It is not a MSSCI plugin so it doesnt have any of the binding issues people mention.