Typemock Isolator++: A Run-time Code Modification Method to Unit Testing

Typemock Isolator++ is a library to incorporate a run-time code modification approach for unit testing in C++. Run-time code modification is a method to patch the application without terminating it. Typemock Isolator++ uses trap instructions to intercept and redirect function calls at run-time and it can fake almost anything: global, static methods and members, free functions, non-virtual methods and live instances. It works seamlessly with GoogleTest, Boost.Test, UnitTest++, CppUnit and Microsoft Test and it is compatible with both Windows and Linux.

Licensing System

The licence comes in two flavours: annual and perpetual. The price for the annual licence starts at €399 and €990 for the build server version. The perpetual version is €799 and €2499 respectively. Support for the perpetual version is not included and it is €170 for the base version and €450 for the build server. For more information visit the pricing page. Typemock is free for MVPs and there is also support for open source projects. If you’re interested get in contact with them through the contact us form and send a brief description of your project. Otherwise you can obtain a 15-day licence directly from the download page.

Installation

Isolator++ is compatible with both Windows and Linux. Before beginning the download it is possible to choose the package for the desired platform:

  • Windows: it has a classic Windows installer with minimal user input needed.
  • Red-hat Package Manager (RPM)
  • Debian package (DEB), created from the RPM using Alien. On some Linux distribution, like Ubuntu, it’s possible to use the GUI to install it if don’t want to use the terminal.

The evaluation license is emailed straight after the download, and once you’ve installed Isolator++ you need to activate it. In the
Getting Started section the installation process is well explained. Bear in mind that the evaluation licence sent with the activation email is only valid for Windows, and you will need to get in contact to receive a valid one for Linux as well.

Integration

Isolator++ works seamlessly with many existing unit testing frameworks:
GoogleTest, Boost.Test, UnitTest++, CppUnit and Microsoft Test. It also integrates with many code coverage tools for both Windows and Linux.

From Visual Studio 2017 15.6, GoogleTest and Boost.Test are both supported
natively by default. Unit tests are automatically discovered after a successful build and available in the Test Windows for execution or debugging. They can be also run directly from the code window by clicking on the icon above the test name.

Running the Examples

There is a good array of examples that comes with the installation and it can be a good place to start. In Windows there are solutions for Visual Studio while in Linux there are Makefiles for both GCC and CLANG.

I decided to compile and run the examples using Ubuntu Cosmic 18.10. Although they already come with an older version of the Google Test framework I decided to use the latest one. Here are the bash commands you need to get the latest GoogleTest sources:

sudo apt-get install libgtest-dev
sudo apt-get install cmake # in case you don't have it 

cd /usr/src/gtest 
sudo cmake CMakeLists.txt 

sudo make
#let's copy the libs to the default path
sudo cp /usr/src/gtest/*.a /usr/lib

I had some initial issues using the provided Makefile for GCC. The linker failed because Isolator++ wasn’t compiled for position independent code. From GCC 6.0 onward position independent execution is enabled by default to support address space layout randomization.
I contacted Typemock support and they quickly replied explaining that I had to disable PIE in the MakeFile for compilation and linking:

-no-pie added to compile the Isolator++ examples

Unexpectedly I ran into another issue where the linker wasn’t able to bind some references to the GoogleTest library. It turned out to be the dual ABI setup between the IsolatorCore library, which is probably compiled using a compiler version pre GCC 5.1 for backward compatibility, and GoogleTest. By setting to 1 or deleting the following define in the stdafx.h everything linked just fine:

#define _GLIBCXX_USE_CXX11_ABI 1

Before running the tests successfully I needed to set these two environment variables: LD_LIBRARY_PATHand LD_BIND_NOW.

# this is used to locate isolator++
export LD_LIBRARY_PATH="/usr/lib64/typemock"

# tells the linker to load dynamic libraries at startup
export LD_BIND_NOW=1

The library path must be added to the environment variable LD_LIBRARY_PATHwhile in order to mock global and static functions
LD_BIND_NOW must be set to 1. There is a brief explanation on the RedHat documentation but in short LD_BIND_NOW instructs the operating system to load dynamic libraries during the application startup:

Finally unit tests ran fine

The cost of writing tests

The creation of unit and integration tests is frequently perceived as a cost rather than a long term investment. The risk of changing or refactoring untested code increases dramatically as the project grows. Such risk is like a debt, it increases over time and it is more expensive to pay back in the long term than investing in writing tests at the very early stages of development.

In new projects teams can draw techniques from test-driven development to make the creation of tests easier, if enough time is allocated by project managers to hit the desired test coverage. However nowadays it is more common to work on existing projects than brand new ones. C++ is an old language and it has been around for almost 40 years, since its creation in 1979. In fact it is common to find systems built on C++ that span over two or even three decades, especially in banking, space and defence industry and also in game engines. Writing tests in isolation for such systems is likely to be very difficult if not impossible without refactoring existing code.

Mock Frameworks and the Proxy Pattern

Mock frameworks are used to automate the creation of mock classes, which are complicated and time-consuming to put together. Mocks are used to imitate behaviours of real objects or to ‘fake’ dependencies of a class. Mock objects have the same interface as the real objects they mimic, allowing a client object to remain unaware of whether it’s using a real object, or a mock object.

In my previous article, in the section ‘Testing using the Proxy pattern’, I discussed how mocking frameworks work under the hood and why they are limited in what they can do. Although the article covered the topic for C# only, mocking frameworks in C++ also relies on polymorphism to hook calls to real objects.

In C++ proxy-based frameworks don’t support mocking of static, free (old C-style functions), global and non-virtual functions. Luckily the flexibility of C++ makes it possible to workaround some of these limitations: for example using composition with templates instead of composition with interfaces.

There are a lot of open source mocking frameworks available for C++: GoogleMock, CppUTest, Boost.Turtle and FakeIt are very popular among developers. MockItNow is no longer maintained and it is the only open source alternative to using binary patching for mocking.

Runtime Binary Patching

Isolator++ doesn’t rely on inheritance to intercept calls. It uses a very different technique that involves patching the test executable at run-time. Run-time code modification has been a subject of different research to find method to fix bugs and security issues at run-time. Mission critical systems, like telecom or banking cannot be stopped to maintain the required level of system availability. Run-time code modification is also referred to as Run-time Binary Patching.

Function interception is achieved through the manipulation of the prologue of the target function and re-routing the function call with an unconditional jump to the user-provided function. The purpose of the prologue is to store the previous stack frame location on the rbp or ebp register
(depending if the executable is compiled for x64 or x86) on the stack and to set the current stack frame location. Instructions from the target function are then placed in a trampoline and the address of the trampoline is placed in a target pointer.

Let’s have a look at this very simple example:

struct Foo {
  int m_anInt;
  Foo(){ m_anInt = 0; }
  
  void DoSomething() const { m_anInt = 10; }
};

TEST(FooTest, FakeMethod) {
  //Arrange
  Foo* foo = new Foo();
  WHEN_CALLED(foo->DoSomething()).Ignore();
   
  //Act
  foo->DoSomething(); //after this call m_anInt should be 10
   
  //Assert
  EXPECT_EQ(foo->m_anInt, 0);
}

The above snippet shows how to use Isolator++ in order to set the expectations in a unit test. There is no need to use a mock for the struct Foo. Binary interception works regardless of the type of object or method being intercepted. Thus Isolator++ can set the expectations on live instances.

Let’s have a look at the assembly code generated for a function before and after the expectations are set by Isolator++. When the test FakeMethodstarts there is no modification to the prologue of the target function DoSomething(left side of the image below). The prologue is changed and the trampoline instruction is injected when the WHEN_CALLEDmacro is called.

The trampoline instruction is a call to a naked function that unconditionally jumps on return to a predetermined location or interceptor function. In this case the trampoline function simply returns to the caller of the function DoSomething() because the expectation is to ignore it. 

Runtime Binary Patching in action

Binary patching is very powerful when used in testing. It reduces the amount of refactoring before it’s possible to start writing tests in isolation because it relaxes any constraint imposed by proxy-based mocking frameworks.

Proxy Pattern vs Binary Interception

In this section I am going to compare the two aforementioned approaches to show how unit tests are written by using both Isolator++ and GoogleMock.

Let’s create a simple class called DbAuththat authenticates users. The class has two member pointers, one for database access called DbUser, and another one for logging called Logger.

struct DbConnection {
 void* m_socket;
 bool OpenConnection();
 void CloseConnection();
};

struct Logger final {
 Logger(const char* name) { /* open the file stream*/ }
 ~Logger() { /* close the file stream*/ }
 void Info(const char* as, ...) { printf("Logging Info"); };
 void Error(const char* as, ...) { printf("Logging Error"); }
 void Warning(const char* as, ...) { printf("Logging Warning"); }
};

//class to access the User table in the db
class DbUser final {
 DbConnection* m_connection;
public:
 DbUser() { 
  m_connection = new DbConnection(); 
  m_connection.OpenConnection();
 }
 ~DbUser() { 
   m_connection.CloseConnection(); 
   delete m_connection; 
 }
 bool IsValidUser(const char*, const char*) const;
};

//used to store any error 
struct Error {
 const char* m_message;
 int m_code;
};

//concrete class to test
class DbAuth final {
 int m_authCode;
 Error m_error;
 DbUser* m_database;
 Logger* m_logger; // a logging class
public:

 DbAuth() : m_authCode(0) { 
   m_database = new DbUser(); //open a connection to the db
   m_logger = new Logger("DbAuth"); //open a file stream
 }
 ~DbAuth() { 
   delete m_database; 
   delete m_logger;
 }
 const Error& GetError() const { return m_error; }
 
 bool AuthenticateUser(const char* username, const char* pwd) {
   m_logger->Info("Authenticating user %s", username);
   if (m_database->IsValidUser(username, pwd)) 
   {  
     m_logger->Info("user %s successfully authenticated", username);
     m_authCode = 10; 
     return true; 
   }
   m_logger->Error("Failed to authenticate the user %s", username);
   m_error.m_message= "Failed to authenticate the user";
   m_error.m_code = -2;
   return false;
 }
};

The class DbAuth has a function called AuthenticateUser to authenticate users given username and password . In its constructor it creates an instance of DbUserandLogger. They respectively open a connection to a database and a file stream. If we wanted to write unit tests for the class DbAuth, it would require some refactoring to avoid hitting the db and the file system. Let’s see how it is possible to test the DbAuthclass without changing its design:

class TestAuthClass : public ::testing::Test {
public:
  void TearDown() override {
   ISOLATOR_CLEANUP();
  }
};

TEST_F(TestAuthClass, AuthenticateValidUser) { 
  DbUser* dbUser = FAKE_ALL<DbUser>();
  FAKE_ALL<Logger>(); 
  
  DbAuth* authObject = new DbAuth();

  auto isValidUser = IS<const char*>([](const char* s)-> bool 
    { return  strcmp(s, "typemock") || strcmp(s, "isolator++"); });
  
  WHEN_CALLED(dbUser->IsValidUser(isValidUser, isValidUser))
  	.Return(true);

  bool validUserSuccess 
       = authObject->AuthenticateUser("typemock", "isolator++");
  
  EXPECT_TRUE(validUserSuccess);
}

TEST_F(TestAuthClass, AuthenticateInvalidUser) {
  DbUser* dbUser = FAKE_ALL<DbUser>();
  FAKE_ALL<Logger>(); 

  DbAuth* authObject = new DbAuth();
  auto isInvalidUser = IS<const char*>([](const char* s)-> bool 
     { return  strcmp(s, "An") || strcmp(s, "Intruder"); });
  
  WHEN_CALLED(dbUser->IsValidUser(isInvalidUser, isInvalidUser))
  	.Return(false);
  
  bool invalidUserFail = authObject->AuthenticateUser("An","Intruder");
  
  EXPECT_FALSE(invalidUserFail);
  
  EXPECT_EQ(-2, authObject->GetError().m_code);
  EXPECT_STREQ("Failed to authenticate the user",
     authObject->GetError().m_message);
}

In the Isolator++ API, the FAKE_ALL macro fakes current and future instances of a class. An enum that specifies the strategy to create the fake can also be passed in and it can be either Call Original or Recursive Fake.
Recursive Fake is the default option used by Isolator++ and it’s one if its special features: it fakes recursively every dependency of a class with a fake instance. This option also initialises every primitive, which is handy to avoid long setup, exceptions or undefined behaviour during testing.

At the beginning of each unit test, the functions FAKE_ALL<Logger> and FAKE_ALL<DbUser> tell Isolator++ to inject/return a mock instead of calling their original constructors.

DbAuth() : m_authCode(0) { 
   m_database = new DbUser(); // Faked instance injected
   m_logger = new Logger("DbAuth"); // Faked instance injected
}

In the two unit tests above, I tested the DbAuth class without hitting the Db, the file-system and without changing its design.

However, one could argue that the design of DbAuth is not ideal. In fact dependencies to other classes should be abstracted away and supplied through the constructor. Such a design approach is called the Bridge pattern.

Following the Bridge pattern DbUser and Logger should be passed in through the constructor of DbAuth. If I were to write the same unit tests using a proxy-pattern framework, I would also need to make DbUser and Logger non-final and change every method I want to set my expectations on to be virtual:

struct Logger {
  Logger(){ /* does nothing, testing only */ }
  Logger(const char* name) { /* open a file stream*/ }
  virtual ~Logger() { }
  virtual void Info(const char* as, ...) { printf("Logging Info"); };
  virtual void Error(const char* as, ...) { printf("Logging Error"); }
  virtual void Warning(const char* as, ...) { printf("Logging Warning"); 
 }
};

//class to access the db
class DbUser {
 DbConnection* m_connection;
public:
 DbUser() { /*Does nothing, testing only */ }
 DbUser(DbConnection* connection) :
 m_connection(connection) { 
    m_connection.OpenConnection();
 }
 virtual ~DbUser() { 
   m_connection.CloseConnection();
 }
 virtual bool IsValidUser(const char*, const char*) const;
};

class DbAuth { 
  int m_authCode;
  Error m_error;
  DbUser* m_database;
  Logger* m_logger; // a logging class
public:
  DbAuth(DbUser* database, Logger* logger) 
   : m_authCode(0)
   , m_database(database)
   , m_logger(logger)
   { }
};

Now that DbUser and Logger can be inherited, let’s re-write the previous tests by using GoogleMock this time:

class MockDbUser : public DbUser
public:
  MOCK_CONST_METHOD2(IsValidUser, bool(const char* username, const char* pwd));
};

class MockLogger : public Logger {
public:
  MOCK_CONST_METHOD2(Info, bool(const char* msg, const char* param));
  MOCK_CONST_METHOD2(Error, bool(const char* msg, const char* param));
  MOCK_CONST_METHOD2(Warning, bool(const char* msg, const char* param));
};

TEST(TestAuthClass, AuthenticateValidUser) {
  MockDbUser mockDbUser;
  MockLogger logger;
  DbAuth dbAuth(&mockDbUser, &logger);
  
  EXPECT_CALL(mockDbEntity, IsValidUser(StrEq("typemock"), StrEq("isolator++")))
     .WillRepeatedly(Return(true));
  
  bool userIsValid = dbAuth.AuthenticateUser("typemock", "isolator++");
  EXPECT_TRUE(userIsValid);
}

TEST(TestAuthClass, AuthenticateInValidUser) { 
  MockDbEntity mockDbEntity;
  MockLogger logger;
  DbAuth dbAuth(&mockDbEntity, &logger);
  
  EXPECT_CALL(mockDbEntity, IsValidUser(StrEq("An"),    StrEq("Intruder")))
  .WillRepeatedly(Return(false)); 
  
  bool userIsInvalid = dbAuth.AuthenticateUser("An", "Intruder");
  
  EXPECT_FALSE(userIsInvalid);
  EXPECT_EQ(-2, dbAuth.GetError().m_code);
  EXPECT_STREQ("Failed to authenticate the user",
    dbAuth.GetError().m_message);
}

In GoogleMock there’s no automatic mock generation and it requires a bit of setup. The mock class must inherit from the real class and each virtual method of the class has to be wrapped with a specific macro
MOCK_METHODn() or MOCK_CONST_METHODn() where n is the number of arguments of the function being mocked. There’s also command-line tool that is written in Python to generate the mock definition given a file name and the abstract class defined in it. However, as stated in the GoogleMock documentation, such a tool may not always work due to the complexity of C++.

Dynamic Dispatch

Dynamic dispatch is the process of selecting which implementation of a polymorphic function to call at run-time. This process has a cost, i.e. virtual methods cannot be in-lined because the linker doesn’t know in advance the address of virtual functions. Despite this, the cost may be negligible in most cases, but if there is a need to squeeze the performance out of a CPU the cost of a v-table look-up could be a problem. In game or physics engines static polymorphism is sometimes preferred for those classes with functions called thousands of times per frame.

Let’s imagine that the class DbUser is performance critical. To avoid a v-table lookup when the IsValidUser function is called let’s declare the dependencies of DbAuth as templates, and let’s make DbUser and Logger final again:

struct Logger {
 ....
}
class DbUser final {
  ....
  bool IsValidUser(const char*, const char*) const;
};

template <class T = DbUser, class U = Logger>
class DbAuth {
  ...
  T* m_database;
  U* m_logger;
public:
  DbAuth(const T* database, const U* logger )
   : m_authCode(0)
   , m_database(database)
   , m_logger(logger)
  { }
  ....
};

DbUser and Logger are final and the IsValidUser function is non-virtual again. The dependencies of DbAuth have been templated away and passed through its constructor. This is a template variant of the bridge pattern that Google defines high performance dependency injection. The two classes have a nicer and cleaner design while using static polymorphism. Let’s write the unit tests using both GoogleMock and Isolator++:

class MockDbUser {
public: 
   MOCK_CONST_METHOD2(IsValidUser,bool(const char* username, const char* pwd));
};

TEST_F(TestAuthClass, AuthenticateValidUserGmock) {
  MockDbUser mockDbUser;
  MockLogger mockLogger;
  
  EXPECT_CALL(mockDbUser, IsValidUser(StrEq("typemock"), StrEq("isolator++")))
  	.Times(1)
  	.WillRepeatedly(Return(true));
  
  DbAuth<MockDbEntity, MockLogger> dbAuth(&mockDbUser, &mockLogger);
  bool userIsValid = dbAuth.AuthenticateUser("typemock","isolator++") ;
  
  EXPECT_TRUE(userIsValid);
}

TEST_F(TestAuthClass, AuthenticateValidUserIsolator) {
  const DbUser* dbUser = FAKE<DbUser>();
  const Logger* logger = FAKE<Logger>();
  
  DbAuth<DbUser, Logger> dbAuth(dbUser, logger);
  
  auto isValidUser = IS<const char>([](const char* s)-> bool
  	{ return  strcmp(s, "typemock") || strcmp(s, "isolator++"); });
  
  WHEN_CALLED(dbUser->IsValidUser(isValidUser, isValidUser))
  	.Return(true);
  
  bool validUserSuccess = dbAuth.AuthenticateUser("typemock", "isolator++");
  
  int timesCalledTrue = TIMES_CALLED(dbUser->IsValidUser(isValidUser, isValidUser));
  
  EXPECT_TRUE(validUserSuccess);
  EXPECT_EQ(timesCalledTrue, 1);
}

TEST_F(TestAuthClass, AuthenticateInValidUserGmock) { 
  MockDbUser mockDbUser;
  MockLogger mockLogger;
  DbAuth<MockDbUser, MockLogger> dbAuth(&mockDbUser, &mockLogger);
  
  EXPECT_CALL(mockDbEntity, IsValidUser(StrEq("An"), StrEq("Intruder")))
  	.Times(1)
  	.WillRepeatedly(Return(false)); 
  
  bool userIsInvalid = dbAuth.AuthenticateUser("An", "Intruder");
  
  EXPECT_FALSE(userIsInvalid);
  EXPECT_EQ(-2, dbAuth.GetError().m_code);
  EXPECT_STREQ(2Failed to authenticate the user", dbAuth.GetError().m_message);
}

TEST_F(TestAuthClass, AuthenticateInvalidUserIsolator) {
  const DbUser* dbUser = FAKE<DbUser>();
  const Logger* logger = FAKE<Logger>();
  
  DbAut<DbUser, Logger> dbAuth(dbUser, logger);
  auto isInvalidUser = IS<const char*>([](const char* s)-> bool 
  	{ return  strcmp(s, "An") || strcmp(s, "Intruder"); });
  
  WHEN_CALLED(dbUser->IsValidUser(isInvalidUser, isInvalidUser))
  	.Return(false);
  
  bool invalidUserFail = dbAuth.AuthenticateUser("An", "Intruder");
  
  int tc = TIMES_CALLED(dbUser->IsValidUser(isInvalidUser, isInvalidUser));
  
  EXPECT_FALSE(invalidUserFail);
  EXPECT_EQ(tc, 1); 
  
  EXPECT_EQ(-2, dbAuth.GetError().m_code);
  EXPECT_STREQ("Failed to authenticate the user", 
  dbAuth.GetError().m_message);
}

The GoogleMock API allows mixing of setup and assertion. Despite it being cleaner from a coding style point of view this approach violates the AAA paradigm. For example:

// Arrange
EXPECT_CALL(mockDbUser, IsValidUser(StrEq("typemock"), StrEq("isolator++")))
  	.Times(1) // it asserts here if IsValidUser 
                  // is called more than once
  	.WillRepeatedly(Return(true));

The call to Times(1) will assert whenever IsValidUser is called more than the number of times expected during the Act section. In the Isolator++ API it’s not possible to mix assertions with expectations. By design the API strictly follows the AAA paradigm. For example in the following snippet the Arrange is separated from the Assert:

auto isInvalidUser = IS<const char*>([](const char* s)-> bool 
  	{ return  strcmp(s, "An") || strcmp(s, "Intruder"); });

// Arrange
WHEN_CALLED(dbUser->IsValidUser(isInvalidUser, isInvalidUser))
  	.Return(false);

// Act
 bool invalidUserFail = dbAuth.AuthenticateUser("An", "Intruder");

// Assert
int tc = TIMES_CALLED(dbUser->IsValidUser(isInvalidUser, isInvalidUser));
EXPECT_EQ(tc, 1);

For conditional behaviour faking Isolator++ has several argument matchers for generic comparison and also custom matchers for object comparison.

Conclusions

Despite C++ offering some niche workarounds through templates to overcome some of the limitations imposed by proxy-based mocking frameworks, it is still up to developers and software architects to come up with an ad hoc design to make testing in isolation possible. Otherwise the alternative is to go through a slow, painful and very risky refactoring process.

Binary patching is a powerful technique for testing because it sets no constraint on code design. It can be very useful on existing code-bases, especially those that are hard to change and to test, in order to achieve good test coverage before any refactoring is needed.

With great power comes great responsibility. In fact, binary patching is not meant to be a shortcut to write poorly designed software. Good design principles and techniques drawn from test-driven development should always be the preferred approach to include tests from the very early stages of development.

Isolator++ is a great tool to automate the creation of mocks and to start writing tests quickly before changing a line of code.

As for the cons, binary patching relies heavily on specific OS libraries, like Microsoft Detour or dlsym for Linux, and CPU architectures. While Isolator++ works on both Windows and Linux, there is no support for embedded devices, MacOS or Console SDKs. If users want to run their tests directly on the target platform it would simply not be possible.

I hope you enjoyed reading the article, I’ll see you soon!

Typemock Isolator: A CLR Interception Approach to Unit Testing

Recently I have contacted Typemock and I was offered a license of Typemock Isolator, their flagship tool, in support of my OSS project LINQBridgeVS. Typemock Isolator is a viable tool to incorporate CLR interception for unit testing in .NET.

CLR is the acronym for Common Language Runtime and it is the virtual machine that manages the execution of .NET programs. Typemock Isolator intercepts CLR calls at run-time and offers a fluent API to fake almost anything: static and sealed classes, instances, virtual or non-virtual synchronous and asynchronous methods, P-invoke calls, private and static constructors and also LINQ queries.  Typemock Isolator comes as a Visual Studio extension compatible with Visual Studio from 2010 through 2017 with any .NET Framework version up to 4.7.2 and incorporates four different components: 

  • Typemock Isolate contains the Typemock Isolator Mocking API, a fluent API to fake and mock objects based on the Arrange-Act-Assert pattern, and Insight which gives detailed information about the behaviour setup during test debugging. 
  • Typemock Suggest suggests and creates tests, for either NUnit or MsTest, for methods and classes in legacy or mainstream code that have partial to no test cover. 
  • Typemock SmartRunner is the built-in test runner and identifies and runs only the impacted tests after the solution’s tests have run for the first time. This includes all modified tests and those tests that need to run because the code they test has changed. 
  • Typemock Coverage is a code coverage tool that displays code coverage for tests, methods, all classes or only those that have recently changed or the entire solution. It also display real-time code coverage while code is written.

Licensing System

Typemock Isolator for .NET has three different types of licenses which are offered in two versions: annual or perpetual. The price starts at €479 pa or (€899 perpetual) for Isolator Essential which comes with the mock framework, the test runner and method coverage. Isolator Complete adds Auto-generated test suggestions and it costs €799 pa (€1599 perpetual). The last one is Isolator for Build Server which adds support to run tests on up to five machines and comes at €1119 pa (€2499 perpetual) . For more information visit the pricing page. Typemock is free for MVPs and there is also support for open source projects, although not directly advertised through their website. If you’re interested in having a license for your open source project get in contact with them through the contact us form and send a brief description of your project.

Introduction

Writing unit and integrations tests is a vital part of the software development life-cycle. It is often underestimated and left as the last part of the development. Achieving an extended test coverage is extremely important for the creation of a reliable and stable product. Techniques and principles from Test Driven Development could be used by teams to make the creation of tests easier.

Reality is often different though. It is a rare opportunity to work on a brand new project where both the design and the development process can be decided from the beginning. Often enough a team determined to follow TDD principles ends up not allocating enough time to hit the desired coverage or tests are not strictly written first. In such cases it is preferred to offload developers and delegate QA teams to write automation tests or to test an individual feature manually. Clearly this approach is not scalable and doesn’t guarantee stability over time. When code changes are not monitored by both unit and integration tests it is very hard to estimate and also to predict the impact that a specific change will have to other parts of the system. 

Unless TDD principles are followed and a robust and clean design is made from the very beginning, the process of writing tests in isolation becomes increasingly more difficult as the project evolves and cannot be achieved without proper (and often painful) refactoring.

The Limitation of Mock Frameworks

Mock object frameworks simulate objects’ behaviour. A mock is something that is setup as part of a test so that it returns an expected result. Mocks in a way are determined at runtime since the code that sets the expectations has to run before they do anything. As a rule of thumb mocks are used to ‘fake’ dependencies of a class. When a method or a class is tested in isolation, i.e. when a a unit test is written for it, dependencies to other systems should be mocked.

There are a lot of free open source mock framework for C#: MoqRhinoMocksFakeItEasy and NSubstitute just to name a few. There is also a built in Visual Studio version called Microsoft Fakes.

The aforementioned frameworks (except for Microsoft Fakes) have one peculiar aspect in common: they can only mock a class given its interface or its base class as long as the base class is inheritable and it declares virtual methods for the behaviours to be setup. 

Testing using the Proxy pattern

Sealed classes, static classes and non-virtual methods cannot be mocked with the proxy pattern approach most mock frameworks use. A proxy is a class functioning as an interface to something else. The proxy has to implement the same interface as the Subject so that the Client doesn’t realise that it is using a proxy. Mock frameworks in fact create proxies at run-time based on interfaces or abstract classes and they are able to override any virtual method. 

The Proxy Pattern
The Proxy Pattern class diagram.
public class Auth
{
   public int AuthStatus { get; private set; }
   public virtual bool IsAuthValid(string user, string pwd)
   {
      return false; //this could be an external call to an auth server
   }

   public void AuthenticateUser(string user, string pwd)
   {
      if ( IsAuthValid(user, pwd) )
           AuthStatus = 10;
   }
}

 // Arrange
 var moqAuth = new Mock();
 moqAuth
   .Setup(p => p.IsAuthValid(It.IsAny<string>(), It.IsAny<string>()))
   .Returns(true);
 // Act
 moqAuth.Object.AuthenticateUser("myUser", "myPwd");
 // Assert
 Assert.IsTrue(moqAuth.Object.AuthStatus == 10);

In the arrange section of the snippet above, Moq creates a proxy that inherits from the class Auth. During the setup, a call to IsAuthValid is overridden so an expected value is returned by default when the method is invoked. The method IsAuthValid however must be virtual, otherwise Moq won’t be able override it and will raise an exception. 

Testing using CLR Interception

Let’s see how the test and the class Auth could be re-written using the CLR interception approach adopted by the Typemock API:

public sealed class Auth 
{ 
   public int AuthStatus { get; private set; } 
   public bool IsAuthValid(string user, string pwd) 
   { 
     return false; //this could be an external call to a server or db 
   } 

   public void AuthenticateUser(string user, string pwd)
   { 
      if ( IsAuthValid(user, pwd) )            
         AuthStatus = 10; 
   }
}

//Arrange
var auth = new Auth(); 
Isolate 
 .WhenCalled(() => auth.IsAuthValid(string.Empty, string.Empty)) 
 .WillReturn(true);

//Act
auth.AuthenticateUser("myUser","myPwd");

//Assert 
Assert.IsTrue(auth.AuthStatus == 10);

I changed the class Auth to be sealed instead and I also made the method IsAuthValid non virtual. Classes should either be designed for inheritance or prohibit it. There is a cost to designing for inheritance, and unless there is a good reason, it should be avoided.

In the test code above there is no mock created. The behaviour is overridden directly on the instance of the object. Typemock intercepts the call to the object’s method at runtime and redirects it to return the value setup instead. If I wanted to make the method private it would also require a small change in the test:

Isolate.NonPublic.WhenCalled(auth, "IsAuthValid").WillReturn(true);

Private or non-virtual methods cannot be tested using a proxy based mock framework. The alternative is using Microsoft Fakes, which I mentioned earlier, and it has similar features to Typemock Isolator.
Microsoft Fakes has support for either proxy-based mocks, which are called stubs, and method interception mocks called shims, which diverts method calls at run time. With Microsoft Fakes every assembly needs to be faked manually. For example if you need to generate a mock for a .NET system type, like DateTime, then a Fake for the System.dll must be created and referenced in the test project. Microsoft Fakes has also some other limitations. Please refer to this link for a detailed comparison of Microsoft Fakes vs Typemock.

Improved design using the Bridge pattern

At this stage one could argue that the current design for the Auth class is poor and a better one would be to abstract away the dependency that authenticates the user from the Auth class.

The Bridge Pattern class diagram.

The Bridge pattern, initially designed by the Gang-of-four, is the best structural pattern to decouple an abstraction from its implementation so that the two can vary independently. It removes the compile-time binding and the implementation can be selected at run-time. Let’s use the Bridge pattern in the Auth class, and let’s add two more dependencies for logging and for db access:

public interface IAuthSystem
{
   bool IsAuthValid(string user, string pwd);
}

public interface ILogging
{
   void Log(string msg);
}

public interface IRepository
{
   IEnumerable<T> Get<T>();
}

public sealed class Auth
{
  private readonly IAuthSystem _authSystem;
  private readonly ILogging _logging;
  private readonly IRepository _repository;

  public int AuthStatus { get; private set; }
  
  public Auth(IAuthSystem authSystem, ILogging logging, IRepository repository)
  {
     _authSystem = authSystem;
     _logging = logging;
     _repository = repository;
  }

  private bool FindUser(string userName)
  {
     var result = from user in _repository.Get<User>()
                where user.UserName.Equals(userName)
                select user;

     return result.Any();
  }
  public void AuthenticateUser(string user, string pwd)
  {
       _logging.Log($"User logging in {user}");

      if ( FindUser(userName) && _authSystem.IsAuthValid(user, pwd))
         AuthStatus = 10;
  }
}
//Moq
  //Arrange
  var authSystem = new Mock<IAuthSystem>();
  var logging = new Mock<ILogging>();
  var repository = new Mock<IRepository>();
  
  authSystem
    .Setup(p => p.IsAuthValid(It.Is<string>(s => s == "Coding"), It.Is<string>(w => w == "Adventures")))
    .Returns(true);

  repository
    .Setup(p => p.Get<User>())
    .Returns(new List<User>{ new User{UserName = "Coding"}});
  
  //Act
  var auth = new Auth(authSystem.Object, logging.Object, repository.Object);
  auth.AuthenticateUser("Coding", "Mock");

  //Assert
  Assert.IsTrue(auth.AuthStatus == 10);

//Typemock
  //Arrange
  var auth = Isolate.Fake.Dependencies<Auth>();
  
  var fakeAuthSystem = Isolate.GetFake<IAuthSystem>(auth);
  var fakeRepository = Isolate.GetFake<IRepository>(auth);
  
  Isolate
     .WhenCalled(() => fakeAuthSystem.IsAuthValid("Coding", "Adventures"))
     .WithExactArguments()
     .WillReturn(true);

  Isolate
     .WhenCalled(() => fakeRepository.Get<User>())
     .WillReturnCollectionValuesOf(new List<User> {new User {UserName = "Coding"}});

   //Act
   auth.AuthenticateUser("Coding", "Adventures");
   //Assert
   Assert.IsTrue(auth.AuthStatus == 10);

With this new design it is much easier to mock away the dependency to IAuthSystem, ILogging and also IRepository. In Moq a mock must be created, setup and an instance passed to the constructor of Auth for each dependency declared in the constructorIn Typemock API there is a shortcut instead to fake all dependencies declared in a constructor using the Isolate.Fake.Dependencies<T>() method. With Isolate.GetFake<T>(object) the corresponding faked dependency can be retrieved to setup behaviours on it:

Auth authObject = Isolate.Fake.Dependencies<Auth>();

IAuthSystem fakeDependency = Isolate.GetFake<IAuthSystem>(authObject);
IRepository fakeRepository = Isolate.GetFake<IRepository>(auth);

Mocking static methods

One thing I always wanted to do freely is using static methods or static classes and also have the ability to override behaviours without having to wrap dependencies with an abstraction layer. For example in my OSS project I have a class called CommonRegistryConfigurations that has a lot of helper methods to access the Windows registry. I always refused to create an abstraction layer that wrapped the Registry class to access the windows registry. I  would eventually end up having parts of my code that rely on these helper class un-tested. Typemock supports faking static methods on static classes, so I was able to setup my object behaviour to mock the call to the static method that access the registry:

public static bool IsSolutionEnabled()
{
   string keyPath = "aKey";

   using (var key = Registry.CurrentUser.OpenSubKey(keyPath))
       return (bool)key.GetValue("myValue");
}

// Test 
Isolate
  .WhenCalled(() => 
     CommonRegistryConfigurations.IsSolutionEnabled())
  .WillReturn(true);

There is one important caveat to keep in mind though. When Isolate.WhenCalled is used to define a fake behaviour on static methods and/or static classes, the behaviour has a global scope. This means that the setup will redirect calls from other unit tests also. It is in fact good practice to mark the test class with a specific attribute available in the Typemock API called IsolateAttribute which makes sure all behaviors are reset at the end of a test method.

Typemock and LINQ Queries

Abstracting away the access to a database is no easy task. One common route is to implement the Repository pattern to abstract away the data access logic, so unit tests can use stubs or mocks as implementations of the IRepository interface. Another solution, if an ORM like Entity Framework is used, could involve faking the DbContext itself instead of wrapping everything with the Repository pattern. There’s a good article here with a thorough explanation on the topic. 

A feature that has really caught my attention is the ability to fake the result of a LINQ query.  Let’s look at the following code:

public bool FindUser(string userName)
{
   var result = from user in _repository.Get<User>()
                where user.UserName.Equals(userName)
                select user;

   return result.Any();
}

// Typemock
 //Arrange
 var auth = Isolate.Fake.Dependencies<Auth>();
            
 Isolate
    .WhenCalled(() => from user in Enumerable.Empty<User>()
                    where user.UserName.Equals(string.Empty)
                    select user)
    .WillReturnCollectionValuesOf(new List<User> { new User() });
 //Act
 bool userExists = auth.FindUser("Adventure");

 //Assert
 Assert.IsTrue(userExists);

In the snippet above, the method FindUser fetches users from a repository using a LINQ query. Instead of changing the behaviour of the method, or faking the repository, it is also possible to fake the result of the query itself. It’s a bit obscure to me though the criteria the API uses to swap the original LINQ query with the one setup in the unit test. I looked up at their official documentation on the topic, but I couldn’t found anything that hints at how linq expressions are swapped around. Regardless of the gore details this feature is very powerful.

Build Server Integration

Typemock Isolator integrates with the several famous build and continuous integration servers like CruiseControl.NET, Jenkins CI, TeamCity and Team Foundation Server, as well as custom build servers. There is a good bit of documentation on their website about Server-side integration. I personally use Appveyor, a free continuos integration solution for Windows and Linux that integrates nicely with GitHub.

Typemock doesn’t integrate with Appveyor as easily as it does with the official supported build servers. Tests that use CLR interception must be run by TMockRunner, a utility part of Typemock Isolator that launches external programs like code coverage tools or a unit test runner like MSTest.

There is some documentation that explains how to run TMockRunner and also how to install it on a custom build server. I’d have preferred to have everything in a single page (or section) so I didn’t have to jump around to find what I needed.

It took me a while to figure out a powershell script that worked nicely with Appveyor. Appveyor runs any custom script with low privileges, thus access to the windows registry is forbidden. TMockRunner needs to access the registry for the initial setup, so I had to come up with a hack to set some specific environment variables to trick TMockRunner into thinking there was no windows registry available. Luckily the support team in Typemock offered a better solution by suggesting to elevate the privileges of my custom powershell script.

I’ll share with you both solutions, in case you need to setup TMockRunner in Appveyor like I did. In the first solution I had to setup the following environment variables in my yml file:

TM_INSTALLDIR: c:\PathToTMockRunner\ 
TMOCK_NO_REGISTRY: 0x1 #this actually tells TMockRunner
there's no registry available
TM_LICENSE:
secure: the encrypted licence number
TM_COMPANY_NAME:
secure: the encrypted company name

Which lead me to this powershell script:

TMockRunner.exe 
   -deploy AutoDeployFolder 
    vstest.console.exe "mytest.dll" "mytest2.dll" 
    /logger:Appveyor

The proper fix though involved elevating the privileges of the powershell script. Then the setup of those four environment variables was no longer needed:

$args = TMockRunner.exe 
   -deploy AutoDeployFolder 
   -register yourLicense vstest.console.exe 
   "test.dll" "test2.dll" /logger:Appveyor

Start-Process powershell -Verb runAs -ArgumentList '$args'

Conclusions

The ability to override methods and classes everywhere, and also in some .NET  classes, is very powerful. It enable developers to write unit tests in isolation in both legacy code where there is none to very little degree of freedom and also in mainstream code. Such freedom shouldn’t be an encouragement to write poorly designed software and TDD (or BDD) should always be the preferred approach when possible so that tests are guaranteed to be part of the product from the very early stages.

The other side of the coin is that a software architecture should always be designed around the problems it solves and the tasks it performs. Being forced to declare methods virtual, avoid sealed or static classes or create a lot of interfaces to abstract any dependency, if not for some technical limitation of mock frameworks, is in my opinion unnecessary. 

On the cons Typemock is not yet compatible with .NET Core and I guess the reason is the different common language runtime (the CoreCLR) .NET Core uses. Also Typemock Suggest needs to be disabled or it will crash if you’re working on a .NET Core project.

I also struggled a bit to setup Typemock in Appveyor, adding the relevant steps I highlighted in this article in their official documentation would be of great help. Another improvement is keeping the Visual Studio extension in sync with the version of the Typemock API. If they mismatch, e.g. you update the Typemock API to a newer version using NuGet but not the extension then an exception is thrown when unit tests are run. 

I really had fun creating tests on my OSS project using Typemock.
For long time I didn’t have much test coverage and finally I was able to increase it in relative short time without doing a huge refactor. The fluent API is very intuitive and easy to use. I hope Typemock will add support for .NET Core in the near future.

This is it, I hope you enjoyed reading this article, see you soon!