Jonathan Boccara's blog

FSeam: A Mocking Framework That Requires No Change In Code (Part 2)

Published July 5, 2019 - 0 Comments

This post is the second part of guest writer Quentin Balland‘s series on FSeam, his testing framework that allows to test legacy code without heavy refactoring. Today Quentin walks us through how FSeam works. Quentin is a French C++ developer eager to learn and share his knowledge on his free time. You can find Quentin online on @FreeYourSoul68 and on his blog.

FSeamThe purpose of this article isn’t to explain everything about FSeam into details as it could be too long and getting boring to read if we go into specifics features that everyone wouldn’t obviously use. We are going to see what are the limitations of GMock and how to use FSeam to try to solve those issues via a quick tutorial.

In this article and in the FSeam API, I call the action that change the behaviour of a function a dupe. It will prevent confusion between a mock, which is an instance of a class that has its behaviour altered, and the actual alterations done on this mock.

In order to understand this tutorial, it would be required to have the following:

  • A Basic understanding of how do unit test and testing framework works, here we will use Catch2.
  • Basic CMake knowledge.

If after reading this tutorial, you want to know more about what you can do, and how far you can go with FSeam, go to the GitHub repository. It contains a pretty complete documentation which explains everything in more depth with examples.

How does FSeam work?

I said in the previous articles that you could basically use FSeam in order to mock your classes without even touching your production code. It looks like a silver bullet. But I never explained how it does this.

FSeam is actually going to generate a mocked implementation of your production code by parsing your header files, and compile it instead of yours for your test.

To do so FSeam is split into three distinct parts:

  • A Code Generator: Developed in Python, it will parse a C++ header file and generate the implementation of the methods and functions that it encounters (of course if the implementation is done in the header file, FSeam shouldn’t re-implement them). This generator is based on an open source C++ header parser formerly developed by Jashua Cloutier, it is now maintained by robotpy on this git repository.
  • A CMake Helper: This is actually a very important part of FSeam, as everything happens at compile time. If we asked users to link the generated files themselves when they need to, FSeam would be nearly impossible to use correctly. Which is why we provide CMake functions in order to handle the generation of the source files and link those generated files in the test binary. We will see later how using CMake almost doesn’t make the CMake code grow, and is pretty easy to use.
  • A C++ Header only library: This library has to be used in order to manipulate the generated mock (check how many times a method/function has been called, with what arguments, or to dupe a method/function)

Changing the behaviour of a class at compile time has been named link seam in Michael Feathers book: Working effectively with legacy code, hence the name FSeam.

The classic way to do

Before entering into “how FSeam works”, I would like to show how we could test this code with the standard methods, explain the pro/cons of such method, to finally have an understanding of how FSeam do things differently.

The above code contains the class we are going to test with GMock.

This is a quite classical external dependency issue we encounter in our everyday code when it comes down to unit testing. The above class contains a DatabaseAccessor object which is our own abstraction of the database connector (could be MySQL, Redis, PostgreSQL, whatever, it is not interesting in this case, let’s just assume that the DatabaseAccessor need a connection to a DB).

If we want to test the function getKnowledgeablePeopleFromCountry, we need a database connection… Well it is inconvenient and there is two way to get around this problem by using a mocking framework (GMock for instance):

  • By inheritance: we can take advantage of the dynamic polymorphism mechanism. To do so we should modify the code above in order to contain a pointer on an Abstract type or an interface representing a DatabaseAccessor. We also need a way to provide our mocked implementation (or production implementation) to the class, the most classical way to do so is to set the value via the constructor.
  • By templatization: or we could templatize external dependency away, by doing so, it will be required to add a getter on the dbAccess instance in order to be able to manipulate it (you could also inject it via the constructor as in the inheritance method)
 

Those techniques work fine but have the issue that you need to have your code comply with some requirements in order to use them (Interface usage, template usage). Which mean you need to refactor some of your code in order to use those techniques. The previous article already explained what were the other downsides of each of those techniques so we won’t go back to that into further details.

Now let’s see how FSeam works and solve the above explained issues. Note that the code examples of this tutorial are available on GitHub.

#1 Installation

It is needed to install FSeam first, you just have to follow this link to know how to do so. Some dependencies for the installation; catch2, python (and python package ply), C++17.

#2 FSeam Test Case

In this tutorial, we will have two different classes to test, one that contains a dependency on a Database connection represented by an object (GameOfThronesDatabase) and another one that will have dependency on free functions and static functions.

I will expose the Catch2 test cases and explain more or less line per line what FSeam does. I won’t explain too many features in order to keep it simple, these examples will be enough for you to start and use FSeam in most cases. For more complex needs, the framework can still help you, but I redirect you to the GitHub documentation that explains everything in further details.

In order to see how to mock classes with FSeam, we are going to test the class above.

This one is composed of two simple functions:

  • isWinnerOfGameOfThrones: that is just checking in the database if the given name is the winner of the Games of Thrones. This first function is interesting because it has the flaw of a lot of legacy code, it creates an instance on a dependency on the fly (databaseConnectionHandler instantiated in the body of the function) and it would be needed to extract this dependency in order to be able to mock it. We will see that it is not needed with FSeam.
  • isGoodGameOfThronesSeason: that is going to use two different instances of database connection (_dbSql and _dbCouchbase). One representing the cache (couchbase) and the other one representing the persistent database (sql).
    Fetching the data from one or the other and verifying if the given season is good or not that good.

#2.1 Test a class: isWinnerOfGameOfThrones

Usually, mocking frameworks require direct access to the instance of the object that you need to mock. But the advantage of having our mock linked at compile time make us able to alter the behaviour of any object instantiated at any time easily; the below example shows you how:

Let’s go through this code step by step.

Get the FSeam MockHandler:

First, we instantiate the ClassToTest we want to unit test, the second line is more important. FSeam works with MockHandler (an object of type FSeam::MockClassVerifier), this object contains the behaviour you want your mock to have. It also stores how the mock has been used (how many times each method has been called, with what argument and so on). Those pieces of information can then be used to do your assertion.

There are multiple ways to get those Handlers, FSeam::getDefault<TypeToMock> return the default mock handler used for the given mocked type.

Actually, the only thing that can differ when mocking a static function/a free function or class with FSeam, is the way to get the MockHandler to manipulate.

Dupe return value and Assert number of calls:

Here is how FSeam is going to be used in most cases. Under the namespace FSeam::ClassMockedName, a “blank type” is generated for each method of the type mocked (here GameOfThronesDatabase).

Without entering into implementation details, you can use dupeReturn in order to decide what is going to be the returned value of your method. This method is present at the MockHandler level and takes only one parameter and has to be of the type that is returned by the function (a compilation error “undefined reference” is generated otherwise).

Then we call the function of our test class, it returns true (as expected). The string we send has actually no impact on the answer as isWinnerOfGameOfThrones will always return true.

Then for the fun of it, we change the return value (because we can) to false, and call the isWinnerOfGameOfThrones method of the ClassToTest some more.

Another important function of the FSeam MockHandler, verify, this function is used in order to check that the given method has been called a certain number of times (4 times in this case).

If you don’t specify any number, the verify method is going to check if the method has been called at least once (you can find additional details here

Expect your methods to be called with specific arguments:

We can add expectations on the arguments that are going to be sent to our mocked method by using expectArg<FunctionName>. Here we basically set expectations as follow: We want isPlayerWinnerOfGameOfThrones to be called at least once with “The Broken” as input, we also want it to be called exactly 3 times with “Know nothing guy” but never with “LittleFinger”.

We then launch our method (following our expectations) and we call the verify method on the mock handler. The verify function is going to validate our expectations.

It is important to use the verify method when using expectArg as it is at this moment that the expectations are checked

There are two important things to note when using expectArg:

  • It is required that the expectations are set before starting the test.
  • In order to validate those expectations, verify method has to be called.

Do not forget to cleanup:

FSeam is working with a singleton object that lives for the entirety of the test. You need to cleanup FSeam mock at the end of each test in order to ensure that there is no pollution for the next test case started. To do so, you need to write this line:

#2.2 Test multiple instances of the same class type:

isGoodGameOfThronesSeason

Now let’s try to test the isGoodGameOfThronesSeason method of our ClassToTest. Previously, we defined a default mocked behaviour for the class, GameOfThronesDatabase, but it wouldn’t work here as we would like to have different behaviour from different instances of the mock on the same methods (like we would normally do with GMock for example).

In this case, we decided to have a getter method returning a handler on the connector objects (but we could have just injected the dependency via the constructor).

As previously said, the only thing that will actually change now is the way to retrieve the MockHandler. When the mock handler is retrieved, the way to dupe, add expectations and verify are exactly the same.

Looks scary? Well not at all, actually, you already know everything about how this code works now! The only real difference is the way we retrieve the MockHandler, and it is summed up in those 3 lines.

As before, we create the instance of the ClassToTest, but this time we retrieve the specific instance MockHandler by using FSeam::get(PointerOnMock). And the rest is exactly the same, dupeReturn, expectArg and verify works the same way than before on fseamMock_SqlDatabase and fseamMock_CoucheBase.

The test case is then quite straightforward, we mock separately each instances using dupeReturn in order to enter in the piece of code we want.

And again (at the end of each test) do not forget to call FSeam::MockVerifier::cleanUp() to clean up the FSeam context.

#2.3 Test a free function or a static method

For this example, let’s change our ClassToTest (but I keep Games Of Thrones theme :p). This one is split into two different files (in order to show a different way to compile in the last part of the tutorial).

In the above ClassToTest, we need to mock a free function called generateRandomNumber() and two static methods of the class DatabaseAccessor, getFavoriteGameForUser(string user, string game) and getAllGames().

Testing the Free Function:

Here we go, the third and last way to retrieve a MockHandler (last because we use the same way for a static method), FSeam::getFreeFunc(). And the rest is the same.

The method name will be found in the namespace FSeam::FreeFunction.

Testing the static method:

As said, getting the static method MockHandler is exactly the same as for the free function, I know it may look counter-intuitive, but the name of the functions are also in FSeam::FreeFunction. The reason is that static methods act exactly like free functions so it was more convenient from an implementation point of view.

#3 Let’s compile it!

Our tests are using Catch2, it has the advantage to be fully integrated with FSeam (FSeam does an automatically register of the Catch2 test via the CMake function catch_discover_tests). Nothing is preventing you from using any other testing framework, but you will need to do the CTest registration by yourself.

The above function declaration is directly taken from the GitHub repository. It is the only function needed to use FSeam with CMake which is using cmake_parse_arguments.

First of all, you need to write is those two lines in your CMake files in order to include the functions you will need (and check that FSeam is correctly installed).

And here is the file system with the files we are going to compile below (everything coming from the GitHub)

#3.1 Compile with a defined set of files

 

The above snippet shows you how to compile your tests by providing a comprehensive list of the file you need to compile. Here is a detailed explanation of the arguments:

  • FILES_AS_SOURCE is the list of files that contain the code you want to test.
  • FOLDER_INCLUDE is the list of folders containing your includes (in order to have your source files finding their includes)
  • TST_SRC is the list of files that contain the testing code (basically the files containing GTest / Catch2 / AnyTestingFramework tests)
  • TO_MOCK is the list of header files that contain the classes / functions signatures you want to mock with FSeam.

#3.2 Compile with a binary target

 

The above snippet shows that it is possible to compile your test with a binary. To do so you need to give the target of your binary into the TARGET_AS_SOURCE argument instead of FILES_AS_SOURCE. It is also required to give the parameter MAIN_FILE which provide the file that contains the main. This file is going to be ignored during the compilation of the test (as the main is generated or provided by your test files).
The advantage of this method is the fact that you don’t need to specify all the files needed for the test.
The drawback is that unnecessary files are going to be compiled.

#3.3 Compile with a library target

 

The above snippet shows how to compile your test with a library.

To do so you need to give the target of your library into the TARGET_AS_SOURCE argument. It is the same as compiling with the binary target (previous snippet) but you don’t need to give a MAIN_FILE to not take into account in the compilation of the test.

As when compiling with a binary target, the advantage of this method is the fact that you don’t need to specify all the files needed for the test. The disadvantage is that unnecessary files are going to be compiled.

As you can see the impact on using the CMake functions provided by FSeam isn’t null but at least it is not too significant. Instead of compiling the tests files via add_exectutable, you just use the addFSeamTests function (which takes some additional arguments). I think it is a correct trade-off for the possibility to test legacy code easily. Please leave a comment if to express your opinion about this trade-off.

In big project, it is recommended to take the extra time to specify each file you need to compile for each test in order to not have really long compilation time for little tests. Using another target (binary or library) to compile your test is useful for little projects.

FSeam can do more

This tutorial cover pretty much 90% of the use cases you would need for your test. But it can do more (customComparator, custom logging, etc…) check it out directly in the documentation.

The project is open source, any feedback on how to improve it is welcome.

You will also like

Become a Patron!
Share this post! Facebooktwittergoogle_pluslinkedin    Don't want to miss out ? Follow:   twitterlinkedinrss

Get a free ebook on C++ smart pointers

Get a free ebook of more than 50 pages that will teach you the basic, medium and advanced aspects of C++ smart pointers, by subscribing to our mailing list! On top of that, you will also receive regular updates to make your code more expressive.

No spam. You can unsubscribe at any time.