My Resume

  • My Resume (MS Word) My Resume (PDF)


Affiliations

  • Microsoft Most Valuable Professional
  • INETA Community Champion
  • Leader, NJDOTNET: Central New Jersey .NET User Group
Showing posts with label Unit Testing. Show all posts
Showing posts with label Unit Testing. Show all posts

Saturday, October 1, 2011

Approval Testing–better ROI for UI testing?

A recent episode of Herding Code interviewed the creator of the Approval Tests project, Llewellyn Falco.  Initially, I was vehemently against the idea.  Rather, I consider automated UI testing (building scripts that execute the UI and inspect what happens) incredibly time-consuming and flaky. In other words, you spend a lot of time producing something that has limited value - the ROI is just not there.

But, by the end of the podcast, I was sold.  The concept of Approval Tests seems to drastically reduce the time it takes to create – and, more importantly, maintain – automated UI tests.  The value stays the same but the effort is reduced, which means that the ROI numbers start to become much more tolerable.  Frankly, I think the ROI of backend (non-UI) tests way overshadows UI testing…  but at least it’s palatable.

Tuesday, June 7, 2011

What is Test-Driven Development (TDD)?

Test-Driven Development is a development approach that relies on unit tests to drive the development and - more importantly - the design of applications. In order for software to be considered "testable" it must be adequately decomposable, allowing tests to target specific units of logic (e.g. classes, methods, or even specific portions of a method). The requirement for decomposition drives loosely-coupled, "SOLID" architecture which embraces OO principles.

Benefits of TDD

"True", dogmatic TDD – also called “Test-First Development” - dictates that code may only be written to satisfy a failing test, and only the bare minimum code is written to make that test pass. TDD provides several benefits:

  • Loosely Coupled Architecture
    The need for tests to completely control a component’s environment drives loosely-coupled components, which – when extrapolated to the system as a whole - leads to a loosely-coupled architecture. 
  • Focused Development
    Scope of code currently written is limited to the needs of the immediate business requirement. If more code is needed to support future requirements, that work is delayed until future tests will drive that development. This keeps developers focused solely on the task/requirement at hand.
  • Regression Test Suite
    Unit tests act as a regression test for the remainder of the application's lifetime. And, since dogmatic TDD states that no code can be written without a test to back it, this implies that an application developed using TDD will never have less than 100% code coverage (the number of lines of production code covered by unit tests). That said, true 100% code coverage is very impractical for a number of reasons.
  • Documentation
    Unit tests are merely code that executes other code, and act as extensive “real-world” examples of how components are used, thus providing a form of documentation.
  • More Productive Debugging
    Since “units under test” are adequately isolated and have at least one unit test focused specifically on them, it is often incredibly easy to locate a failing component by looking for the failing test(s). What’s more, since unit tests are executable, debug-able code, developers can easily attach their debugger to a specific test and execute it.

Detriments of TDD

  • More Code
    By definition, the test-first methodology produces a test suite which – at a minimum – doubles the size of your solution’s codebase. This leads to:
    • Increased Lines of Code
      Assuming it takes at least the same amount of time and effort to write test code as it does to write production code, TDD literally doubles the time spent writing code produced (and the corresponding time it takes to write said code).
      Perspective: In terms of the SDLC, the time spent actually writing code is only a fraction of the Implementation phase – much more time is spent on developer testing/verification, debugging, and bug fixing. Taking this into consideration, the increased coding time introduced by TDD is easily offset by more targeted and productive debugging, not to mention lowering the number of bugs to begin with (both in the long term and the short term!). 
    • Increased Cost of Change
      Since unit test code is so closely tied to production code, changes to business requirements mean that both production code and its corresponding tests will need to change. The implications of this change are the same as the preceding bullet: writing and changing code is only a fraction of the SDLC Implementation phase.
  • Even More Code!
    Developers can easily become carried away with writing an abundance of unit tests in an effort to achieve the highest level of code coverage they can. The ROI of additional unit tests against an already-tested component can drop quickly as the number of tests goes up.
  • False Sense of Security
    A high level of code coverage can provide a false sense of security if developers are convinced that the level of code coverage equates to the nonexistence of bugs. However, code coverage only measures whether or not a line of code was executed, not how it was executed (i.e. under what conditions). Consider a highway system: just because you drove your car over every foot of road doesn’t mean those same roads will react the same when traversed by a bus.

An Example of TDD in Action

Business Requirement

The application must produce the sum of two numbers

Step 1: Write a failing test

public void ShouldProduceSumOfTwoNumbers() {
    Assert.AreEqual(4, new Calculator().Sum(1, 3));                FAIL!
}

Step 2: Write just enough code to make the failing test pass

public class Calculator {
    public int Sum(int number1, int number2) {
        return 4;                                                                     PASS!
    }
}

And we’re done! Except that what we’ve produced is a method which returns a hard-coded value! This situation is easy to rectify: write another failing test against the same component.

Step 3: Write another test which specifies a different set of parameters

public void ShouldProduceSumOfTwoOtherNumbers() {
    Assert.AreEqual(5, new Calculator().Sum(2, 3));                FAIL!
}

Since the new test asserts a different result based on different inputs, this test fails because the initial implementation of the Sum method returned the hard-coded value different than what this new test expects.

Step 4: Revisit and refactor the production code to pass the new test

public class Calculator {
    public int Sum(int number1, int number2) {
        return number1 + number2;                                         PASS!
    }
}

Though simple and contrived, this example effectively demonstrates the process – and more importantly, the mindset – behind Test-Driven development.

TDD and UI Development

As you move further away from the statically-typed compiled “backend” code and closer to the UI, the unit tests associated with these parts of the system tend to introduce less resilient and reliable methods such as string comparison. As a result, the cost of creation and maintenance grows exponentially.

A word of warning: because of this exponential cost and loss of strong reliability, the ROI of the TDD approach often becomes negative when applied to the UI layers. It is often better to drive the testing of UI layers by professional (QA) testers as they will likely be applying these approaches anyway.

TDD vs BDD (Behavior-Driven Development)

Test-Driven Development – as its name implies – relies on unit tests to drive production code. Ideally, these unit tests derive from business requirements, however strict adherence to the Test First approach often means that developers end up writing unit tests to allow them to write code and ensure that that code works… not that it meets any kind of business requirements.

Behavior-Driven Development (BDD) is a philosophy grown from TDD which focuses on the software requirements of - and human interaction with - “the business” to deliver software that provides value to the business. Though the two approaches are variations on the same theme and the differences are subtle, BDD aims to please customers by satisfying their (ever-changing) requirements, as opposed to simply focusing on “working code”. This usually means less stringent code coverage requirements

Resources

General internet searches for the concepts in this document such as “test driven development” and “behavior-driven development” rarely leave much to be desired. I have not come across many bad resources in regards to Test-Driven Development. Unfortunately, because these are heavily philosophical concepts that go far beyond simply learning a language or syntax, the only way to truly understand it is to find a mentor and do it (and learn from your mistakes).

Regardless, here is a short list of some of the better resources I’ve found recently:

· Test-Driven Development Wikipedia (yes, it’s a great resource!)

· Test Driven Development Ward Bell, et al – the grandfather(s) of XP

· Guidelines for Test-Driven Development Jeffery Palermo

· Introduction to Behavior-Driven Development BddWiki

· Introducing BDD Dan North

· The Art of Agile Development: Test-Driven Development James Shore

· What is a Unit Test? Jess Chadwick

· Test-Driven Development: By Example Kent Beck

· Working Effectively With Legacy Code Michael Feathers (applying TDD to existing codebases)

Friday, February 25, 2011

Presentation: Automated Unit Testing for Mere Mortals

Last weekend I had the immense pleasure of getting my unit testing presentation selected as one in the great Code Camp NYC lineup.  It was a great crowd and this is the first time I tried to record one of my talks.  I think it turned out alright!  I’ve embedded the low-quality version below.  If you prefer the high-def version, here it is:  Unit Testing for Mere Mortals (720p).  

Enjoy, and please feel free to let me know what you think!

Thursday, November 12, 2009

What’s a “Unit Test”?

Photo courtesy of those show cancelling bastards at CBS.

No, I’m not talking about these guys...

Generally speaking, writing any kind of code that exercises the code you've written is a good thing, but the term “unit test” carries with it a very focused and specific meaning. Listed below are what I consider the top-most important qualities of a “unit test”:

  • Atomic


    A unit test should focus on validating one small piece (“unit”) of functionality. Generally, this will be a single behavior or business case that a class exhibits. Quite often, this focus may be as narrow as a single method in a class (sometimes even a specific condition in a single method!). In practice, this equates to short tests with only a few (preferably just one) deliberate and meaningful assertions (Assert.That([…])).

    Common Pitfalls & Code Smells
    • Dozens of lines of code in one test
    • More than 2-3 assertions, especially when they’re against multiple objects
  • Repeatable


    A unit test should produce exactly the same result at any time on any environment, given that environment fulfills a known set of dependencies, e.g. the .NET Framework. Tests cannot rely on anything in the external environment that isn’t under your direct control. For instance, you should never have to worry about having network/Internet connectivity, access to a database, file system permissions, or even the time of day (think DateTime.Now). Failed unit tests should indicate a bug in the code and nothing else.

    Common Pitfalls & Code Smells
    • Tests pass on the first execution, yet some or all fail on subsequent executions (or vice-versa)
    • “NOTE: The XYZTest must be run prior to this or it will fail!”
  • Isolated / Independent

    In a culmination of the first two qualities, a unit test should be completely isolated from any other system or test. That is to say, a unit test should not assume or depend upon any other test having been run or external system (e.g. database) having a specific state or producing some specific result. Additionally, a unit test should also not create or leave behind any artifacts that may trip up other tests. This is certainly not to say that unit tests cannot share methods or even whole classes between each other – in fact, that is encouraged. What this means is that a unit test should not assume some other test has run previously or will run subsequently; these dependencies should instead be represented as explicit function calls or contained in your test fixture’s SetUp and TearDown methods that run prior to and immediately following every single test.

    Common Pitfalls & Code Smells
    • Database access
    • Tests fail when your network or VPN connection is disabled
    • Tests fail when you have not run some kind of external script (other than perhaps an NAnt script to compile, of course)
    • Tests fail when configuration settings change or are not correct
    • Tests must be executed under specific permissions
  • Fast

    Assuming all of the above conditions are met, all tests should be “fast” (i.e. fractions of a second). Regardless, it is still beneficial to explicitly state that all unit tests should execute practically instantaneously. After all, one of the main benefits of an automated test suite is the ability to get the near-instant feedback about the current quality of your code. As the time to run the test suite increases, the frequency with which you execute it decreases. This directly translates into a great amount of time between the introduction and discovery of bugs.

    Common Pitfalls & Code Smells
    • Individual tests take longer than a fraction of a second to run

If one were really clever, they might arrange the above into a cute little acronym like “FAIR”, but the order in which they appear above is very deliberate; it is the rough order of importance that I place on each quality.

Unit Tests vs. Integration Tests

Odds are that if you have written any automated tests recently, you probably violated one of the above guidelines… and probably for very good reason! What you have produced, my friend, is another very valuable form of automated test called an integration test. As opposed to a unit test - whose sole purpose is to validate the logic and/or functionality of a specific class or method – an integration test exists to validate the interaction (or “integration”, as it were) between two or more components. In other words, integration tests give the system a good work-out to make sure that all of the individual parts work together to achieve the desired result – a working application.

As such, integration tests are just as – if not more so – valuable in a business sense as unit tests. Their major drawbacks, however, are their slow speed and fragility. Not only does this mean that they will get executed less frequently than a unit test suite, but the rate of false-positives (or negatives… however you want to look at it) is much higher. When a unit test fails, it is a sure indication of a bug in the code. In contrast, when an integration tests fails it may mean a bug in the code, but could also very well have been caused by other issues in the testing environment such as a lost database connection or corrupt/unexpected test data. These false positives - though a useful indicator that something is wrong in the developer’s environment – usually just serve to slow down the development process by taking the developer’s focus away from writing working code. Assuming you strive to avoid these distractions whenever possible, the conclusion I come to is that you should therefore strive to rely on extensive test coverage via a solid unit test suite and supplement that coverage with an integration test suite and not vice-versa.

References

A great deal of the reason I even took it upon myself to write this blog post was because I couldn’t really find any good online articles or posts concerning “what makes a unit test”!   Below are a few of the great ones I found.  It may seem like I stole from some of them, but the ideas above really are my opinions…  they just happened to be widely shared. :)

However, it seems at this point if you are very interested in learning more about this topic, books are your best bet.  Anything by the “usual suspects” (Fowler, Hunt, Thomas, Newkirk…) is a great bet, but here are a few I have read and loved:

Sunday, November 16, 2008

Easier Automated Database Testing with SQL Express

Scenario

I've got a project in which I actually have full create scripts for my database such that I can build a whole new instance from the bottom up.  I've also got some automated unit/integration tests that I want to run against this database, complete with a bunch of scripts that can build some test data for me (unrealistic, I know...  but bear with me :).  Also, I really don't want to have to worry about configuring connection strings just for my tests - I just want some database available to me when I need it that I can wail on with requests and gets cleaned up for me when I'm done.  Finally, I want to keep my tests as isolated as possible, which to me means a file-based SQL Express database; that way, I can attach, detach, and delete as much as I want with as little exposure and impact to the rest of my build system as possible.

Solution

My solution to the above scenario I found myself in was to create a helper class called TestDatabase whose job is to give me a database when I need one, provide me with a clean version of my test data before each test I run, and clean up after me when I'm done.  To this end, I started searching for how to create a file-based SQL Express database using code, and came up with Louis DeJardin's great blog post that walked me right though it.  After I had that, it was a simple matter of whipping up the class, shown below (Note: this is only a partial listing.  You can get the full listing from my code repository):

TestDatabase.cs (partial listing)
public class TestDatabase : IDisposable
{
private readonly string connectionString;
private readonly string databaseFilename;

public string ConnectionString { get { return connectionString; } }
public string Schema { get; set; }
public string TestDataScript { get; set; }

public TestDatabase(string databaseFilename, string schema, string testData)
{
this.databaseFilename = databaseFilename;
connectionString = string.Format(
@"Server=.\SQLEXPRESS; Integrated Security=true;AttachDbFileName={0};",
Path.GetFullPath(databaseFilename));
Schema = schema;
TestDataScript = testData;
}

public void Dispose()
{
DeleteDatabaseFiles();
}

public void RecreateTestData()
{
EnsureDatabaseCreated();

if (!string.IsNullOrEmpty(TestDataScript))
ExecuteQuery(TestDataScript);
}

// Create a new file-based SQLEXPRESS database
// (Credit to Louis DeJardin - thanks! http://snurl.com/5nbrc)
protected void CreateDatabase()
{
var databaseName = Path.GetFileNameWithoutExtension(databaseFilename);

using (var connection = new SqlConnection(
"Data Source=.\\sqlexpress;Initial Catalog=tempdb;" +
"Integrated Security=true;User Instance=True;"))
{
connection.Open();
using (var command = connection.CreateCommand())
{
command.CommandText =
"CREATE DATABASE " + databaseName +
" ON PRIMARY (NAME=" + databaseName +
", FILENAME='" + databaseFilename + "')";
command.ExecuteNonQuery();

command.CommandText =
"EXEC sp_detach_db '" + databaseName + "', 'true'";
command.ExecuteNonQuery();
}
}

// After we've created the database, initialize it with any
// schema we've been given
if (!string.IsNullOrEmpty(Schema))
ExecuteQuery(Schema);
}
}


Let's analyze the things we've got going on here:


  1. First, we've got the CreateDatabase() method (lines 34-61) - basically ripped right from Louis's blog post linked above - which does the magic of creating a file-based SQL Express database.  It all boils down to a "CREATE DATABASE" and "EXEC sp_detach_db" call on the local SQL Express instance's tempdb database, which everyone has access to.  Then when that's all done, I execute the schema script that the tester passed in to build the database schema and finish the initial setup.
  2. Now that the database has been created and initialized with its schema, we can run some tests against it!  Problem is, at this point it's just an empty database...  Fortunately for us, we've got the RecreateTestData() method, which just executes the TestDataScript against the current database, allowing us to easily populate whatever test data we want!  This script should include everything it needs to clean out the database and rebuild it from scratch with a new set of clean data.
  3. Built-in connection string management.  As you can see, our constructor takes in a database filename, builds a connection string out of it, and then exposes that connection string to our testers via a read-only property.  That is one less connection string that our test project has to worry about managing in its app.config (or whatever), which is pretty nice and clean, IMHO!
  4. Finally, our big finale:  cleaning up after ourselves!  You can see that TestDatabase implements IDisposable, allowing us to create a Dispose() method which cleans up after everything we've done - namely, deleting the database files we've created along the way.  This means that after everything is said and done, we've left not one footprint of our presence on the build system.

Now, after we've got our TestDatabase class available, our unit tests become as easy as this:


public void SomeCoolDatabaseDrivenServiceTest()
{
var mySchema = System.IO.File.ReadAllText("mySchema.sql");
var testData = System.IO.File.ReadAllText("testData.sql");
using (var db = new TestDatabase("TestDatabase.mdf", mySchema, testData))
{
db.Initialize();
var service = new MyService(db.ConnectionString);
service.DoSomething();
}
}

Of course, individual tests can have even less code if you manage the test database outside of the test by using your test framework's setup and teardown methods.  For example, if I had a whole slew of tests against the same database (which is usually always the case), the test class would start out like this:


TestDatabase database;

public void ClassInitialize()
{
var mySchema = System.IO.File.ReadAllText("mySchema.sql");
var testData = System.IO.File.ReadAllText("testData.sql");
database = new TestDatabase("TestDatabase.mdf", mySchema, testData);
database.Initialize(true);
}

public void TestInitialize()
{
// Rebuild the test data from scratch before EVERY test
database.RecreateTestData();
}

public void ClassCleanup()
{
database.Dispose();
}

Now that we have all of that setup and teardown logic out of the way, we can focus on what we're actually testing, so then that test I showed you before becomes a simple one-liner (as it would have been if we were just passing in a connection string from a configuration file):


public void SomeCoolDatabaseDrivenServiceTest()
{
// No TestDatabase setup - just use its connection string!
var service = new MyService(database.ConnectionString);
service.DoSomething();
}

What's cool about this is that not only do we not have to worry about where to get our connection string from, our entire suite of test data is also being rebuilt for us before every test is run!

Try It Out For Yourself!


If you like what you've seen in this post and want to try it out for yourself, you can grab the full source file (complete with in-line comments and unit tests) from my repository: TestDatabase.cs.  Just drop it in your project and start using it!
Note: The full source file has unit tests included. If you don't want them, you can simply delete them without affecting the main class.


As always, I'd love to hear your comments and feedback on all this.  If you've found this useful or - better yet - if you have a better way of doing it, please let me know!

Tuesday, April 15, 2008

Unit Test Project Structure

I recently came across Phil Haack's interesting blog post and associated poll about structuring your unit test projects - do you have separate unit test assemblies or not? - and was not entirely surprised by the results.  They were more or less in line what what I'd expect.  At the beginning of my TDD "career" I'd actually considered for a brief second whether it'd be better to put my unit tests in line with the code they're testing (not "inline" as in the same file, but close) or in a completely separate assembly.  After a few minutes of thought, I came away with what it seems most other people come away with - and emphatic "NO!"  I'm honestly left wondering - and I don't mean any disdain, but - what are these 8% of "Yes" voters are thinking!?  Is it laziness, or some other motive that I'm not getting?  Really - I'd love to know!

Wednesday, August 1, 2007

Dummy Page Handler

My current project at Infragistics is revamping our entire redirect and tracking backend. Since we are dealing with redirects and rewrites, my tests naturally ventured into the realm of doing something to/with the current HttpContext. As anyone who's dealt with this can attest to, I had my fair share of difficulties along this road, but decided to blog about one problem and solution I thought might be particularly interesting and/or useful for anyone testing redirects.

My situation

I have a list of request URLs along with a corresponding redirect path for each of them. The project I'm working on is a library - it's meant to be used in a web site, but is not actually run as a standalone site. You can unit test with your favorite framework all you want, but at some point when you're dealing with redirection and the HttpContext, you're eventually going to want to try it out in a hosted environment. Using blog posts and code from Scott Hanselman and Paul Haack, I set myself up a nice little WebDev process and ran my integration tests within that. Everything worked great, except... RED BAR?!?

My redirection tests had failed. "404 Error" messages filled my Output box. Oh... yeah - using the hosted approach actually creates a website using physical files, just like any normal website. The fact that my integration test class library was the thing starting it up was irrelevant - it still expected the physical pages to exist and it wouldn't allow me to just redirect to any hypothetical path that I wanted to (even though my tests didn't really care whether the destination path actually existed or not - just that our response was redirected properly).

(Note: If you're not partial to my storytelling, you can just jump down to the Solution section below for what I eventually came up with.)

My first attempt at was a knee-jerk reaction: WebDev wanted a physical file? I'll give it one. So, I added "dummy.html" to my project and used TestWebServer.ExtractResource("dummy.html", destinationPath) before every request. Echk. This approach gave me a few reasons to feel uneasy: I now have an extra (rather meaningless) static file around to clutter up my project; I'm writing a file before EVERY request; and well, what if sometimes I actually wanted the 404 error to occur?

Onto attempt #2. Well, let's call it Attempt 1.1, because I didn't actually solve either of the last two issues, but that stupid extra file in my project was bugging me, so I replaced the call to ExtractResource("dummy.html", destinationPath) with CreateDummyPage(destinationPath). The CreateDummyPage() method did away with reading the dummy HTML page from the resources and just used a StringBuilder to populate the content... then wrote it to a file. But, like I said - this didn't really help fix my two other issues.

Solution

The solution ended up being ridiculously easy: write an HttpHandler to serve up dummy pages. The code ended up looking like below. Of course, you can boil this code down even more and simply use a string constant for the page content. But, as you can see, I wanted some kind of content on the page (to make sure the Handler itself was actually working, at the very least!).

using System;
using System.Text;
using System.Web;

namespace Infragistics.Guidance.Web

public class DummyPageServer : IHttpHandler
{
private string _PageTitle = "Dummy Web Page";

public bool IsReusable
{
get { return true; }
}

public void ProcessRequest(HttpContext context)
{
string body = String.Format("Requested path: {0}", context.Request.Path);
StringBuilder sb = new StringBuilder("<html>");
sb.AppendFormat("<head><title>{0}</title></head>", _PageTitle);
sb.AppendFormat("<body>{0}</body>", body);
sb.Append("</html>");
context.Response.ClearContent();
context.Response.Write(sb.ToString());
context.Response.End();
}
}
}







Then, you register the module like so:

Oh, but wait - what about my third concern from before? What if I don't want every .aspx request to be mocked up?? Just be more specific in the handler definitions, such as:

This approach seems to be working great for all of my tests so far. No more 404's - yay! I hope it can help with yours.

Friday, May 11, 2007

NUnit and MSTest working in harmony

For my latest project at Infragistics, we decided it would be prudent to take the TDD approach. We do all our development in Visual Studio 2005, however most of us do not have the Team System edition and are left without the MSBuild/MSTest framework it provides. Those of us without VS2k5: TS who want to try our hand at TDD have to find another suitable framework and - as anyone who has ever searched for .NET unit testing knows - the natural course of Googling will inevitably bring you to NUnit. During a discussion about which testing framework to "standardize" on in our group, Ambrose came across this MSDN Magazine article which - in addition to being an informative article overall - has a great tidbit about writing tests that work under both NUnit and MSTest. And, like any great tip, it's incredibly simple and boils down to defining aliases for the various attributes you use. A little more searching, and I found this complimentary blog entry that sheds a little more light on just what attributes match up together.

Unfortunately, choosing to go this route means that you're limited to the lowest common denominator when it comes to what attributes and Asserts you can use (at least without having to add "#if NUNIT" directives everywhere), but when you're looking for a cross-test-platform solution like we were, it's a pretty good compromise!