What’s in a name?
Google “Pex” and what do you expect to find? According to Wikipedia:
“Cross-linked polyethylene, commonly abbreviated PEX or XLPE, is a form of polyethylene with cross-links.”
That’s not really the definition I was hoping for, so if we qualify the search with “Microsoft” then we have a better chance of encountering a more meaningful software related defintion:
“Pex (Program EXploration) produces a traditional unit test suite with high code coverage. A parameterized unit test is simply a method that takes parameters, calls the code under test, and states assertions. Given a parameterized unit test written in a .NET language, Pex automatically produces a small unit test suite with high code and assertion coverage. To do so, Pex performs a systematic white box program analysis.”
Well, now we have got that cleared up, let’s have a look at what we can do with it.
For a start, the most powerful aspect of Pex is the white box analysis that it performs on an existing unit of code. This analysis will create the appropriate boundary test (as best as it can). The generation of test code is not extensive; it covers the simple cases and where complex data type are used, the developer will most likely still have to fall back on utilizing test methods such as mocking.
Part of the secret of getting Pex to work your code properly is to provide some form of Parameterized Unit Test to get it started. Another important factor is to use the Stubs Lightweight Framework (also a research project from Microsoft) to create appropriate stubs for the Parameterized Unit Tests. Both methods can be generated quite easily; however a few tweaks are required to get it just right.
In order to illustrate the work involved we shall revisit the previous example of one of the Managers (either EmployeeManager or CustomerManager). The following walkthroughs assume that you have Pex installed and that you have either Visual Studio Test edition or Team edition installed.
How it is done using defaults…
In the first stage we need to instruct the assembly being tested that it will give the new unit test assembly access to the internal methods and properties (see my previous post):
-
[assembly: InternalsVisibleTo("DiDemo.Managers.Tests")]
The second stage is to create the Unit Test project that will “house” the Pex generated unit Tests. The easiest way to do this is to use the Pex auto-generation functionality from the Code Context menu:
- Right click the class name that will have the unit tests generated (in our case CustomerManager) and select the Pex->Create Parameterized Unit Test Stubs:
- Configure the Unit Test project that you want to target the auto-generated code to (in this case we will select
from the drop down combo box) and click Ok when you are ready:
- Specify (if you wish) the location of the new Test project and click OK:
- The result will be the generation of the new Unit Test project complete with auto-generated Stubs and Parameterized Unit Test Stubs:
Having a quick look at what is generated, we see that the stubx file is the configuration xml for the Microsoft Stubs engine that indicates which assembly we are generating the stubs from – thereby creating the stubs designer file:
As we will see later on, the stubs that are generated in this “default” manner might not be exactly what we want, although the concept is going to come in handy.
Next is the auto generated parameterized unit test stubs. These methods are intended to be used by Pex to generate the unit test methods when the analysis is run. In their current format they do little but to serve as place holders:
At this stage, as an educational exercise, we can run the Pex Explorations to see what gets thrown up:
Looking through the results there is one glaring issue and that is the fact that the way we have created the architecture of the managers (using Dependency Injection and having default behavior); we have inadvertently generated integration tests (the clue is in the fact that we are seeing SqlExceptions – this means that we have also explored the underlying data access layer, which we had tried so hard to inject the dependency for:
The thought through way…
In my mind we don’t want to generate stubs of the classes that we are actually testing, we want to generate stubs of the interfaces that we are injecting into the classes we are testing – a subtle difference, but nevertheless one that is important.
Therefore, strip out the constructor tests (parameter-less and single DI parametered constructor):
After those test methods are removed, delete the Manager stubs (we aren’t interested in stubbing out the class we are testing):
Finally we want to generate the stubs for the Interfaces, so we need to create a stub of the BaseData assembly:
Now that the stubs are generated, we can go about changing the parameterized test methods to fit our own purposes. Taking the first method Delete, we make the changes to the method in such a way that the method itself creates the class instance we want to test and the parameter provides us with the boundary conditions. This means that our code changes from this:
-
/// <summary>Test stub for Delete(Customer)</summary>
-
[PexMethod]
-
public void Delete([PexAssumeUnderTest]CustomerManager target, Customer i_customer)
-
{
-
// TODO: add assertions to method
-
// CustomerManagerTest.Delete(CustomerManager, Customer)
-
target.Delete(i_customer);
-
}
to this:
-
/// <summary>Test stub for Delete(Customer)</summary>
-
[PexMethod]
-
public void Delete(Customer i_customer)
-
{
-
// Create a stub instance of the ICustomerRepository
-
stub.Delete = (a) => { };
-
ICustomerRepository repository = stub;
-
// Create a real instance of the CustomerManager that we want
-
// to put under test
-
manager.Delete(i_customer);
-
}
Once all of the parameterized unit tests are prepared we can run the Pex Explorations again:
The result this time is quite different to what we had previously with the outcome being more conclusive than the first. Upon inspection we actually find some test scenarios that we didn’t cover in our code:
The general idea is to continue to adjust the code of your class to make sure that all of the test cases are handled correctly until you can achieve, as far as possible, test case completeness:
The next stage is to run the tests to see what the code coverage is:
You will see that we didn’t get 100% code coverage because of the method “<LoadAllCustomers>b__0(class DiDemo.BaseData.Entities.Customer,class DiDemo.BaseData.Entities.Customer)”;which is the following area of code:
Conclusion
What is on offer by using Pex is quite astounding, but by no means bullet proof. There are still cases that you will want to write your own test method to try to reach the nirvana of 100% code coverage; however what Pex does do is to take care of the majority of cases that you would grudgingly have to knock out by hand.
The other niggling issue in my mind is the fact that the stubs have to be re-generated anytime there is a change made to a dependant class, which might seem like nit-picking, it is still something that could be overlooked during development – thereby rendering the unit test as unreliable.
On the whole I am quite impressed by what is on offer and I could foresee it’s use when the wider .NET development community starts to walk around and kick the tires a bit more, maybe even take it out for a few laps.
