August 25, 2024

1000x Faster Mocking with C# Source Generators

The title of this post emphasizes performance, but in reality, simplicity was my primary focus. From the outset, I knew there was significant potential for performance improvements; everything just needed to come together smoothly—and for the most part, it did. I understand that being fast isn’t the most crucial factor in every aspect of programming. In some cases, “good enough” really is good enough. So, if you don’t prioritize the performance of your mocking tool, that’s perfectly fine. However, performance is measurable and generally objective, while simplicity and ergonomics are highly subjective—hence the title of this article.

Overall, this post is the result of exploring a “what if we…” idea. I found the outcomes of this exploration compelling enough to share them as a blog post.

Mocking benchmarks

C# source generators are an incredible tool for certain use cases. The increasing use of ahead-of-time compilation, despite its limitations, is also contributing to the broader adoption of this technology. I believe we have yet to see some great .NET applications and frameworks based on compile-time source code generation, as there seem to be promising opportunities in this area.

How would you mock something without using a framework?

I imagine everyone with sufficient experience knows what mocking is in the context of testing, so I won’t spend time explaining why and when it is needed. However, I’d like to start with a very basic example of manual mocking because:

  1. It will illustrate my thought process and motivation for this approach.
  2. It’s exactly what we’ll implement in the end—basic and simple (but with all the boring parts delegated to the compiler). So, please follow along.

Let’s mock a simple interface:

public interface IService
{
   string GetData(string arg);
}

For now, we’ll avoid using any frameworks or libraries and do all the work ourselves. How would you mock it manually? It depends on what exactly is needed. We could simply implement the interface with some hard-coded behavior for the GetData method:

public class HardcodedService : IService
{
   public string GetData(string arg)
   {
       return "some data";
   }
}

That might actually be good enough for some use cases. It’s even possible to write custom logic based on the data passed into the method. But now, let’s consider a situation where this approach becomes difficult to scale.

This is getting repetitive

Consider an interface with more methods:

public interface IService
{
   string GetData(string arg);
   string GetSomeOtherData(int arg);
   // ... 10 more methods
   string GetEvenMoreData();
}

Some might say this is a poorly designed interface—10+ methods? Perhaps we’ve violated one or more of the SOLID principles, and maybe we should consider refactoring. Who knows? But in reality, interfaces like this do exist. That’s not the main issue here. The real problem is that the amount of monotonous and boring code we have to write in such cases increases exponentially.

Even an interface with just 4-5 methods would effectively demonstrate the problem. Imagine mocking this interface for different use cases (let’s say 10+ unit tests, each with slightly different logic):

public class FooService : IService { }
public class BarService : IService { }
// ... 10 more implementations
public class BazService : IService { }

You can already see the problem: a lot of code. We could somewhat reduce the amount of code by adding, say, boolean properties to control how a particular implementation behaves. But there’s a better solution.

The strategy pattern. The method implementation can be defined outside the method itself by using the strategy pattern. In our particular case, we don’t even need the classic form of this pattern; Functional style and C# delegates (Funcs and Actions) will do the job:

public class Service : IService
{
   public Func<string, string> GetDataImpl;
   public string GetData(string arg) => GetDataImpl(arg);

   public Func<int, string> GetSomeOtherDataImpl;
   public string GetSomeOtherData(int arg) => GetSomeOtherDataImpl(arg);

   public Func<string> GetEventMoreDataImpl;
   public string GetEventMoreData() => GetEventMoreDataImpl();
}

I hope you can see how this allows us to shift the actual implementation from the service class itself to the client code that uses it. Now, we don’t need a separate implementation for each use case—we can control both behavior and state.

That’s great, but there’s still a chance you don’t want to write that code. If so, you’re not alone, and .NET developers have already solved this problem. However, before we move on, I ask you to take another look at that code—it’s the essence of our future implementation. This example is a bit simplified, and we won’t write it ourselves, but instead, we’ll kindly ask Roslyn to do it for us.

Magic to the rescue

As mentioned earlier, mocking is a solved problem. Now, let’s take a look at something that already exists and is widely used. The most popular solution is probably Moq (though most other options look quite similar):

var mock = new Mock<IService>();
mock.Setup(s => s.GetData(It.IsAny<string>()))
   .Returns("some data");
var service = mock.Object;
service.GetData("arg"); // -> "some data"

Despite all the bells and whistles, this approach essentially follows the same idea: generate an implementation and set up state and logic based on the current needs. We provide the rules for a particular use case, and Moq generates the implementation for us at runtime—simple.

Shifting the magic to compile time

This is where the “what if we…” scenario comes into play. What if we shift most of this ceremony to compile time? Will we achieve comparable convenience? Can we improve on it (arguably) and eliminate DSLs such as Setup, Returns, and other methods? Let’s find out.

First, we need to decide what we want to generate. The answer is: we want something similar to what we created manually—a class that can be controlled via Func delegates. After some quick prototyping, it became clear that separating the mock builder from the actual mock implementation makes sense due to the complexity of resolving method overloads and for the sake of separation of concerns.

Let’s look at an example of what will be generated for an interface with one read-only property, one write-only property, and a method that takes a string and returns a string:

public class ServiceMock
{
   private Func<string> _getterOfReadProperty;
   public ServiceMock ReadProperty(Func<string> getter)
   {
       _getterOfReadProperty = getter;
       return this;
   }

   private Action<string> _setterOfWriteProperty;
   public ServiceMock WriteProperty(Action<string> setter)
   {
       _setterOfWriteProperty = setter;
       return this;
   }

   private Func<string, string> _implOfSingleArgReturnMethod;
   public ServiceMock SingleArgReturnMethod(Func<string, string> impl)
   {
       _implOfSingleArgReturnMethod = impl;
       return this;
   }

   public IService Build()
   {
       return new @class(this);
   }
   // ...

The job of this generated class is straightforward: define builder methods for each interface method or property and accept a Func or Action with a signature that matches the target method. Notice how we can generate arbitrary names for the builder methods (which happen to resemble the actual names of the interface methods by design). Unlike traditional frameworks, we are not restricted to working with a predefined builder class that acts as a DSL for constructing implementations at runtime.

You might have noticed the Build method, which we haven’t covered yet. As you might guess, it constructs the actual instance of the object. Let’s examine the second important piece of generated code: a class that implements the given interface:

private class @class : IService
{
   private readonly ServiceMock @var;
   public @class(ServiceMock @var)
   {
       this.@var = @var;
   }

   public string ReadProperty
   {
       get
       {
           if (@var._getterOfReadProperty == null)
           {
               throw new NotImplementedException();
           }


           return @var._getterOfReadProperty();
       }
   }

   public string WriteProperty
   {
       set
       {
           if (@var._setterOfWriteProperty == null)
           {
               throw new NotImplementedException();
           }


           @var._setterOfWriteProperty(value);
       }
   }

   public string SingleArgReturnMethod(string arg)
   {
       if (@var._implOfSingleArgReturnMethod == null)
       {
           throw new NotImplementedException();
       }


       return @var._implOfSingleArgReturnMethod(arg);
   }
}

This class is also generated at compile time, and its role is to implement the target interface using the provided delegates. There’s nothing more to say about it, other than that it’s a private nested class inside ServiceMock.

And that’s it—nothing more. This example fully matches what is actually done by the library.

A note on inheritance and classes

It turns out this model works well with classes and inheritance. Two small additions are needed:

  1. A class can have multiple constructors, so the generated code must provide corresponding constructors and build methods.
  2. For virtual methods, if no setup is performed, the generated code must call the base implementation from its parent class by default.

Roslyn will do all the work

This post is not intended to be a tutorial on writing source generators—there are plenty of those already. However, there is one missing piece we need to discuss: how to specify which interfaces or classes the generator will work on.

One approach could be to use comments, like this:

// generate mock for IService

We could register a syntax receiver to capture comments and parse type names from them to handle the code generation. While this might work in theory, it is not ideal. This method is error-prone and unsafe—any part of the comment could be misspelled, IService could be renamed later, and so on. Instead, we will use attributes—an established and reliable way to declare metadata, which is exactly what we need:

[assembly: Mock(typeof(IService))]
[assembly: Mock<IEmptyService>] // for C# version >= 11

Now, a syntax receiver is registered to listen for attributes of a given type and extract the necessary metadata at compile time to generate the code described above. Again, this is slightly outside the scope of this post and is a significant topic in itself.

Testing

Finally, let’s test everything together in a setting that closely resembles the intended use:

using Mockup;
using Xunit;

// This will generate mock; use Mock(typeof(IService)) for C# < 11.0
[assembly: Mock<IService>]

namespace Mockup.Tests;

public class ServiceMockTests
{
    [Fact]
    public void TestReadWriteProperty()
    {
        object value = "Value";
        
        var service = new ServiceMock()
            .ReadWriteProperty(() => value, v => value = v)
            .Build(); // This will produce IService
        
        // Your test code...
        var result = service.ReadWriteProperty; // returns "Value"
    }

    [Fact]
    public void TestSingleArgReturnMethod()
    {
        var service = new ServiceMock()
            .SingleArgReturnMethod(v => "Changed" + v)
            .Build(); // This will produce IService

        // Your test code...
        var result = service.SingleArgReturnMethod("Value"); // returns "ChangedValue"
    }
}

Benchmarks

Let’s create two simple scenarios and compare the results to Moq, NSubstitute, and FakeItEasy. The numbers will speak for themselves.

BenchmarkDotNet v0.13.12, macOS Ventura 13.6.6 (22G630) [Darwin 22.6.0]
Intel Core i5-7267U CPU 3.10GHz (Kaby Lake), 1 CPU, 4 logical and 2 physical cores
.NET SDK 8.0.301 [Host]     : .NET 8.0.6 (8.0.624.26715), X64 RyuJIT AVX2

Return a string hardcoded into a variable:

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
Mockup 34.60 ns 0.599 ns 0.531 ns 0.0688 - - 144 B
Moq 4,380.75 ns 58.442 ns 51.808 ns 1.8616 - - 3905 B
NSubstitute 5,410.08 ns 105.646 ns 121.662 ns 3.7384 - - 7833 B
FakeItEasy 5,701.07 ns 107.159 ns 114.659 ns 2.4109 0.0153 0.0076 5057 B

Return a string passed as an argument:

Method Mean Error StdDev Gen0 Gen1 Gen2 Allocated
Mockup 16.39 ns 0.392 ns 0.347 ns 0.0268 - - 56 B
Moq 76,471.38 ns 1,119.135 ns 934.529 ns 4.1504 1.2207 0.4883 9118 B
NSubstitute 6,309.30 ns 115.183 ns 107.743 ns 3.7537 - - 7905 B
FakeItEasy 6,377.44 ns 121.293 ns 129.783 ns 2.7771 0.0305 0.0305 5861 B

We need to keep in mind that the frameworks mentioned above are designed to handle much more than what we’ve used here. More features mean more overhead, and everything is done at runtime. Therefore, the comparison may not be entirely fair. Nevertheless, the results are impressive. What’s even more impressive is that the speed is comparable to the hand-coded version; they are essentially the same. We’ve just asked the computer to generate the mock for us.

Next steps

As I mentioned, this is an experiment, and there are many TODOs: What if we need to mock generic interfaces or classes? What if we need to control the name of the generated mock or even the namespace? There are many more interesting questions to explore. Stay tuned.

Links

First of all, take a look at the code: https://github.com/x2bool/mockup. There are also NuGet packages available: Mockup and Mockup.Analyzers. Give them a try, and feel free to leave your feedback on GitHub if you have any comments or suggestions.