If yoda were a developer…

What’s in a name?

Google “Pex” and what do you expect to find? According to Wikipedia:

“Cross-linked polyethylene, commonly abbreviated PEX or XLPE, is a form of polyethylene with cross-links.”

That’s not really the definition I was hoping for, so if we qualify the search with “Microsoft” then we have a better chance of encountering a more meaningful software related defintion:

“Pex (Program EXploration) produces a traditional unit test suite with high code coverage. A parameterized unit test is simply a method that takes parameters, calls the code under test, and states assertions. Given a parameterized unit test written in a .NET language, Pex automatically produces a small unit test suite with high code and assertion coverage. To do so, Pex performs a systematic white box program analysis.”

Well, now we have got that cleared up, let’s have a look at what we can do with it.

For a start, the most powerful aspect of Pex is the white box analysis that it performs on an existing unit of code. This analysis will create the appropriate boundary test (as best as it can). The generation of test code is not extensive; it covers the simple cases and where complex data type are used, the developer will most likely still have to fall back on utilizing test methods such as mocking.

Part of the secret of getting Pex to work your code properly is to provide some form of Parameterized Unit Test to get it started. Another important factor is to use the Stubs Lightweight Framework (also a research project from Microsoft) to create appropriate stubs for the Parameterized Unit Tests. Both methods can be generated quite easily; however a few tweaks are required to get it just right.

In order to illustrate the work involved we shall revisit the previous example of one of the Managers (either EmployeeManager or CustomerManager). The following walkthroughs assume that you have Pex installed and that you have either Visual Studio Test edition or Team edition installed.

How it is done using defaults…

In the first stage we need to instruct the assembly being tested that it will give the new unit test assembly access to the internal methods and properties (see my previous post):

The second stage is to create the Unit Test project that will “house” the Pex generated unit Tests. The easiest way to do this is to use the Pex auto-generation functionality from the Code Context menu:

1. Right click the class name that will have the unit tests generated (in our case CustomerManager) and select the Pex->Create Parameterized Unit Test Stubs:
   
   
   
2. Configure the Unit Test project that you want to target the auto-generated code to (in this case we will select from the drop down combo box) and click Ok when you are ready:
   
   
   
3. Specify (if you wish) the location of the new Test project and click OK:
   
   
   
4. The result will be the generation of the new Unit Test project complete with auto-generated Stubs and Parameterized Unit Test Stubs:
   
   
   

Having a quick look at what is generated, we see that the stubx file is the configuration xml for the Microsoft Stubs engine that indicates which assembly we are generating the stubs from – thereby creating the stubs designer file:

As we will see later on, the stubs that are generated in this “default” manner might not be exactly what we want, although the concept is going to come in handy.

Next is the auto generated parameterized unit test stubs. These methods are intended to be used by Pex to generate the unit test methods when the analysis is run. In their current format they do little but to serve as place holders:

At this stage, as an educational exercise, we can run the Pex Explorations to see what gets thrown up:

Looking through the results there is one glaring issue and that is the fact that the way we have created the architecture of the managers (using Dependency Injection and having default behavior); we have inadvertently generated integration tests (the clue is in the fact that we are seeing SqlExceptions – this means that we have also explored the underlying data access layer, which we had tried so hard to inject the dependency for:

The thought through way…

In my mind we don’t want to generate stubs of the classes that we are actually testing, we want to generate stubs of the interfaces that we are injecting into the classes we are testing – a subtle difference, but nevertheless one that is important.

Therefore, strip out the constructor tests (parameter-less and single DI parametered constructor):

After those test methods are removed, delete the Manager stubs (we aren’t interested in stubbing out the class we are testing):

Finally we want to generate the stubs for the Interfaces, so we need to create a stub of the BaseData assembly:

Now that the stubs are generated, we can go about changing the parameterized test methods to fit our own purposes. Taking the first method Delete, we make the changes to the method in such a way that the method itself creates the class instance we want to test and the parameter provides us with the boundary conditions. This means that our code changes from this:

to this:

Once all of the parameterized unit tests are prepared we can run the Pex Explorations again:

The result this time is quite different to what we had previously with the outcome being more conclusive than the first. Upon inspection we actually find some test scenarios that we didn’t cover in our code:

The general idea is to continue to adjust the code of your class to make sure that all of the test cases are handled correctly until you can achieve, as far as possible, test case completeness:

The next stage is to run the tests to see what the code coverage is:

You will see that we didn’t get 100% code coverage because of the method “<LoadAllCustomers>b__0(class DiDemo.BaseData.Entities.Customer,class DiDemo.BaseData.Entities.Customer)”;which is the following area of code:

Conclusion

What is on offer by using Pex is quite astounding, but by no means bullet proof. There are still cases that you will want to write your own test method to try to reach the nirvana of 100% code coverage; however what Pex does do is to take care of the majority of cases that you would grudgingly have to knock out by hand.

The other niggling issue in my mind is the fact that the stubs have to be re-generated anytime there is a change made to a dependant class, which might seem like nit-picking, it is still something that could be overlooked during development – thereby rendering the unit test as unreliable.

On the whole I am quite impressed by what is on offer and I could foresee it’s use when the wider .NET development community starts to walk around and kick the tires a bit more, maybe even take it out for a few laps.

In continuation with the Dependency Injection entry of my previous blog, now we look at how we can leverage the work we did to use it with a mocking framework to perform proper unit testing.

There are numerous blogs and articles about different Mocking frameworks and after taking a couple of them around the block and kicking the tires I settled on RhinoMocks as the best fit for my needs. I put together a little table to try and capture my thought process on choosing the appropriate framework and although this is not an extensive list it met my needs at the time:

So what is mocking?

Well, for those readers that are not familiar with mocking it can be summed up with a reference to Martin Fowler in his Mock’s aren’t Stubs post

“Mocks are objects pre-programmed with expectations which form a specification of the calls they are expected to receive.“

Whilst doing my research into the use of Mocking Frameworks and what I wanted to get out of them I came across a few references to “Isolation Frameworks” that seemed to be used on top of the reference to Mocking Frameworks and the idea is effectively that you use the mocking framework of your choice to isolate the unit under test by specifying the expected behavior of the mocked object. We are still mocking, but with in the context of testing an acute code path of a particular class.

How does it fit in with the example?

In my previous blog, I introduced the very basic example of dependency injection to illustrate the idea behind isolating the dependencies in order to easily “injection” an alternative. Now that we have the ability to inject an alternative, we’ll create mock instances of those dependencies and use those instead of the real ones.

As a reminder, here is the definition of the IEmployeeConnector interface that we created when setting up the Dependency Injection in the EmployeeConnector class:

With this in mind, we can then test the functionality of the LoadAllEmployees function:

This then implies that we need to create a mock instance of the IEmployeeConnector that will be passed into the constructor of the EmployeeConnector class. The following MSTest method provides an example of how we can do this:

This test method will test the Sort functionality of the LoadAllEmployees function on the EmployeeConnector instance in isolation, because we have told the mocking framework exactly how we want it to behave with the Expect.Call() function.

When the method does get called within the LoadAllEmployees of the EmployerConnector method the RhinoMocks framework will pass back the filledCollection instance that we had constructed:

As a brief note, the Test utility method FillCollection() will fill the passed in List collection with dummy data that the Test method can then check against in the Assert statements:

Visual Studio Test Edition

One of the wonders of the Visual Studio Test Edition is the fact that you can run the MSTests and also specify the Assembly that you want to perform code-coverage:

Once this is done, then when you run the MSTest you can view the code paths that were covered by the test (with highlighting):

And also those code paths that were not covered (with highlighting):