Example Based Testing

Almost every engineer writes example-based tests. You pick some inputs, determine what the output should be, and then write some sort of test harness that will provide those inputs to the code under test and check that the output matches your expectations. It’s the first kind of testing most of us learn and it makes up the bulk of nearly every test suite I’ve ever worked in. It’s so familiar that it has become more or less invisible, in many cases when we talk about tests, we are implicitly assuming that we’re talking about example-based tests, and rarely do we actually stop to reflect on what it is (and isn’t) doing for us as a testing methodology.

So, despite its ubiquity, I’m going to spend this blog post talking about the strengths and the weaknesses of example based testing. Although, given that the strengths are generally pretty widely understood already, I’m going to be a touch imbalanced and focus a bit more on the pitfalls of the methodology than its strengths.

A Way to Think About Tests

Before getting into example-based testing itself, I find it useful to think about tests in general as being made of three pieces:

• The inputs: whatever you feed into the thing you’re testing.
• The code under test: the thing you’re actually trying to verify.
• The oracle: the part that decides whether the result was correct.

We’re going to ignore the middle one for the rest of this post, and focus on the first and last, which, in my opinion, really define example-based testing. You can, after all, apply example based testing to whatever scope of code you wish from a single function to an entire system (although if you’re interested in a dicussion of testing scope, my previous post on testing levels touches on the topic).

The input as a concept is fairly straightforward, but I want to take a second to explain what I mean by ‘oracle’. While this is a fairly common term in the testing world, you rarely hear it in more general software development circles, so I think it’s worth clarifying. The oracle is simply the thing that knows the right answer. When you write if got != want, the got != want is your oracle. This is, in my opinion, the simplest and most common test oracle, a value that the programmer chose and is performing an equality check against. That said, oracles can take many forms: sometimes the oracle could be a property of that you’re checking (“no duplicates present”, “ascending order”, etc.), sometimes it’s a second implementation that you’re comparing against, or, if you’re doing usability testing, it could be a cohort of people telling you how usable something is. The important thing is that every test needs something that can tell right from wrong, and that something is the oracle.

Sidebar: Inputs Are More Than Just Arguments

One gotcha in this model is to just think of inputs as arguments to the function, but inputs also include the full set of everything which could affect the result of the test. For a narrow little unit test, this distinction barely matters. Consider a pure function:

func Add(a, b int) int {
  return a + b
}

The only inputs here are a and b. There’s nothing else in play; the function’s behaviour is entirely determined by its arguments. The rest of the environment is largely irrelevant unless you want to expand the scope of your test to encompass hardware issues like “is the ADD instruction implemented correctly” or “what if a cosmic ray hits the RAM module”. You could make an argument for the endianness or the register width as being part of the correctness-relevant inputs here, but, in most scenarios, it’s really just the function arguments a and b that matter.

But a lot of code isn’t like that, consider something more typical:

func GetActiveUserCount(ctx context.Context, db *sql.DB) (int, error) {
  var count int
  err := db.QueryRowContext(ctx,
    "SELECT COUNT(*) FROM users WHERE active = true",
  ).Scan(&count)
  if err != nil {
    return 0, err
  }
  return count, nil
}

The function arguments here are a context and a database handle, but just as important is the state of the database. The rows in that table, the schema, whether the connection is even alive, all of that is input, even though none of it shows up as a function parameter. And anyone who has ever had to debug a test suite that is flaking because tests don’t have isolated access to the database (eg they are sharing state) will have the scars to show why forgetting this is a Bad Idea.

Example-based testing (for real now)

With that conceptual jibber-jabber out of the way, it’s very easy to see how example-based testing fits into that paradigm.

• The inputs are the specific examples that the test author chose to test
• The oracle is a set of assertions that are used to determine if the result matches what is expected for the example.

Go’s table-driven test paradigm make this structure especially visible:

func TestAdd(t *testing.T) {
  cases := []struct {
    name     string
    a, b     int
    expected int
  }{
    {"zeros", 0, 0, 0},
    {"positive", 2, 3, 5},
    {"negative cancels", -1, 1, 0},
  }

  for _, c := range cases {
    t.Run(c.name, func (t *testing.T) {
      got := Add(c.a, c.b)
      if got != c.expected {
        t.Errorf("Add(%d, %d) = %d, want %d", c.a, c.b, got, c.expected)
      }
    })
  }
}

The slice of cases is your hand-picked inputs. The value of expected and the got != c.expected check comprise your oracle. An idiomatic Go test is often a literal diagram of the model: here are my examples, here is the answer I expect for each one.

The good

Example-based testing is simple, easy, and usually quite effective. The mental model is about as straightforward as testing gets, and, when a test fails, it’s quite easy to understand why the test failed (assuming your assertions are labelled reasonably anyway): you got X, you wanted Y, here’s the difference. You don’t even really need a tutorial to understand this and early in most peoples’ programming journeys they will intuitively adopt this pattern in its manual form before they even write their first test. (remember commenting out some of your logic logic, adding a few prints, and running the program to figure out why it’s not working?)

Even once you start automating your tests, they’re still easy to write, and you don’t even need something like Jest, JUnit, etc. All you need is an if got != want { panic(...) } in a throwaway script and suddenly you have automated tests. This is a big contrast to other approaches like mutation or fuzz testing that depend on fairly sophisticated tooling. In other words, the barrier to entry for example-based testing is essentially zero: you don’t need to read a paper to understand it and you don’t need to learn a complex library to apply it, you just need to know how to program, which is a skill you already have if you have written a piece of software to test. That said, it has its limits, and I find it very helpful to keep them in mind when testing.

The bad

The Unknown Inputs Problem

Your test cases are a direct encoding of your imagination. If you didn’t think to test a particular situation, that situation simply has no coverage. This is pretty obvious, but it is one of the biggest downsides of example based testing in my mind.

That said, to be fair, this sounds worse than it really is in practice. Engineers tend to have pretty good instincts for where things will go wrong. The classic move with any numeric operation is to throw 0, -1, 1, and the largest and smallest values the type can hold at it, which can catch a lot of issues. Beyond extreme or “interesting” values, if you are doing white-box example-based testing, you probably know what parts of the code make you uneasy and can craft some inputs to exercise them. The intuition for where these spots are does take a bit of experience to build, but it develops pretty quickly once you start getting paged when things break.

However, intuition does not scale. As a system grows and accumulates more moving parts, the interesting failures increasingly live in the interactions between pieces, in combinations that no single person pictured while writing any individual test. You can’t hand-pick an example for a scenario that never occurred to you.

The Blind Oracle

This is the same blindness, but on the oracle side. Even if you happen to feed in exactly the input that triggers a bug, a weak oracle will cheerfully wave it through. If the code produces a wrong result and you never wrote an assertion that would notice that particular wrongness, your test passes and you’re none the wiser. Having the right input doesn’t help if you forgot to actually check the thing that broke.

That second point leads directly into one of my favourite testing tropes, because it’s so common and so quietly damaging. It is entirely possible to have code with excellent computed test coverage that is, in practice, barely tested at all. This often happens when the assertions are too vague to catch aberrant behaviour, and in the worst of cases, the only thing your “test” actually verifies is that the code didn’t crash since that oracle is always in play regardless of how vague your assertions are. And worse, if you’ve mocked out a lot of dependencies, you may not even be testing that in a meaningful way since your pass on “it didn’t crash” may be an artifact of the mocks and not actually apply when wired up with real-world dependencies.

Here’s a test that looks productive and verifies almost nothing:

func TestProcessOrder(t *testing.T) {
  order := Order{Items: []Item{{SKU: "ABC", Qty: 2}}}

  result, err := ProcessOrder(order)
  if err != nil {
    t.Fatalf("unexpected error: %v", err)
  }

  if result == nil {
    t.Fatal("expected a result")
  }
}

Coverage tooling will look at this and tell you ProcessOrder is covered. After all, technically every line of it was executed in the context of a test, but what did we actually check? All we know is that it didn’t crash, it didn’t return an error, and that it returned something. Everything could be wrong on that order and we would be none the wiser. Maybe there’s a flaw in the subtotal calculation and every order is adding up to $0.00. The lines were executed, but almost nothing of value was checked.

There’s a closely related trope worth mentioning: code that is supposedly “well tested” but will fall over in exciting ways in production because it has no error handling and no test inputs that exercise the error paths. In this case, both of these issues are “synergizing” since the lack of error handling means that the computed coverage will look fine and the lack of sad-path test cases means that it won’t fail in testing. So you can be fully in the grips of this issue and have no clear signal that you even have a problem.

This particular issue is especially pernicious in languages that lean on exceptions for error handling without forcing you to acknowledge them. In Python or Javascript, or with Java’s unchecked exceptions, any call you make can throw, and nothing obliges you to deal with it. That means that a function with an untested, unhandled error path looks identical to a function that has no error path at all. The landmine is invisible until someone steps on it in production. Compare that to Go errors, Rust’s Results, or Zig’s error unions, where an unhandled error is obvious from the syntax (and may not even compile). As a result, if you’re working with an exception-driven error handling paradigm, I think it is very important that you exercise a high level of mental rigour when it comes to error handling and testing error paths (and ideally use a static code analysis tool that can flag these issues), because otherwise you might never notice.

In any case, the key point is that it is important that you are honest with yourself about what your oracle actually checks, and it is critical that you don’t take test coverage metrics as an indication that something is well tested.

Overly strict oracles

So far I’ve mostly talked about tests that are too loose, but it is fully possible (and quite common) to err in the other direction too, and it has its own distinct cost.

The strict equality check that powers most example-based tests makes for a very rigid test suite. When your assertions pin down every last detail of the output, perfectly legitimate changes that happen to touch irrelevant details will break tests that had no business caring about those details.

Snapshot testing and golden-file testing are often the worst offenders here. While they’re extremely convenient and quite useful for some things (UI renders for instance), the flip side is that any incidental change to that output, a reordered field, some different whitespace, a newly added key, will fail the test regardless of whether the change was within the scope of what that test was intended to check.

With every one of these brittle tests you add, you are creating a tax that will be levied on every future change to the code. To make matters worse, when the pain starts being felt, teams regularly fall into a second, follow-up trap. When
tests start to fail for dumb irrelevant reasons often enough, the reflex starts to shift from “lets debug this” to “eh, I’ll just look at it briefly and then re-record the snapshot,” and that shift rapidly erodes the test’s ability to catch a real regression, because now nobody is paying attention to it.

I want to acknowledge a tension here before moving on. Earlier I was cautioning against oracles that are too permissive and here I am cautioning against ones that are too strict. This can sound a bit self-contradictory, but two things can be true at the same time: more assertions give you more power to catch bugs and more assertions give you more surface area for brittleness. The important point isn’t really that you need to have more or fewer assertions, it’s that you need to assert on the right things so that your test is only testing things that should matter to it.

Getting the Most Out of Example-Based Testing

None of these critiques mean that I think people should abandon example-based testing, but I do think people should be a bit more deliberate and mindful of its weak points when they use it. To that end, here are a few habits that I personally find helpful for maximizing the effectiveness of example based testing:

Ask “how could the code be wrong and still pass this test?” This is, by a wide margin, the single most valuable habit I know of for writing better example-based tests. Look at the test you just wrote and genuinely try to imagine a broken implementation that sails through it anyway. Once you think of a gap, add a test case that closes it and repeat. Sometimes it is also useful to turn this practice around and ask yourself “how could the code be correct and still fail this test”, which can help identify tests that are likely to be brittle/flakey in the future.

Don’t over-specify the oracle: Assert on what the test is actually about and leave the rest alone. This ties straight back to the brittleness problem: every assertion on a detail you don’t care about is a little brittleness bomb you’ve planted for your future self. If your test is about the order total, check the order total. Don’t also pin down the exact formatting of an unrelated log message.

Use looser checks for the oracle: This is not always applicable, but sometimes we can make an oracle that asserts on properties of the output that are looser than equality checking. To give a more concrete example of what I mean, below is a test that validates some properties of an Add function without computing an “expected” value:

func TestAdd(t *testing.T) {
  cases := []struct {
    a, b int
  }{
    {0, 0},
    {2, 3},
    {-1, 1},
  }

  for idx, c := range cases {
    t.Run(fmt.Sprint("%d",idx), func(t *testing.T) {
      // a + 0 == a
      if res := Add(c.a, 0); res != c.a {
        t.Errorf("Add(%d, 0) = %d, want %d", c.a, res, c.a)
      }
      // a + 0 - a == 0
      if res := Add(c.a, 0) - c.a; res != 0 {
        t.Errorf("Add(%d, 0) - %d = %d, want %d", c.a, c.a, res, 0)
      }
      // b + 0 == b
      if res := Add(c.b, 0); res != c.b {
        t.Errorf("Add(%d, 0) = %d, want %d", c.b, res, c.b)
      }
      // b + 0 - b == 0
      if res := Add(c.b, 0) - c.b; res != 0 {
        t.Errorf("Add(%d, 0) - %d = %d, want %d", c.b, c.b, res, 0)
      }
      // commutative
      if one, two := Add(c.a, c.b), Add(c.b, c.a); one != two {
        t.Errorf("Add(%d, %d) = %d, Add(%d, %d) = %d, %d != %d", c.a, c.b, one, c.b, c.a, two, one, two)
      }
    })
  }
}

This shouldn’t be taken as an example of how you should test Add functions, but rather as an example of how you can create a very loose oracle that still verifies a fair bit of behaviour. Another example would be testing a sort function by verifying that the output is sorted instead of by computing a sorted result for an equality check. Both of these examples are relatively trivial, but, while not always viable to do, this kind of oracle can make your tests a lot more flexible (and that oracle can then be used for generative testing).

Make sure your assertions can actually fail. A test that cannot turn red tests nothing, and it’s extremely easy to write one by accident. In my opinion, this is the biggest benefit of the “red-green-refactor” cycle in test-driven development: when you write the test first and watch it fail before you write the code, you have some evidence that the test is wired up correctly and is actually capable of catching a problem. Granted, TDD isn’t always the right approach for every situation (personally I don’t actually use the TDD paradigm very often), but you can borrow that specific idea without committing yourself to the methodology. As an example, you can flip some key conditionals while rerunning your test to make sure it fails as expected (this is effectively manual mutation testing).

Guard your setup with assertions too. This one wastes an astonishing amount of engineering time when it goes wrong, so I think it’s always worth doing. If your test depends on some setup having succeeded, assert that it actually did. The classic example of this going wrong is testing a delete operation:

func TestDeleteUser(t *testing.T) {
  userID := uuid.New()	
  _, _ = CreateUser(userID, "alice@example.com")

  err = DeleteUser(userID)
  if err != nil {
    t.Fatalf("DeleteUser returned an error: %v", err)
  }

  _, err = GetUser(userID)
  if err != ErrUserNotFound {
    t.Errorf("expected user to be gone, got err = %v", err)
  }
}

Without setup assertions, this test has a nasty failure mode. If CreateUser fails, your delete might “succeed” simply because there was never anything there to delete, and your test passes while testing absolutely nothing. By asserting that the creation worked first, you make sure the thing you’re actually testing is the thing being exercised.

The subtler benefit is about where the failure shows up. When setup breaks, you want the setup assertion to be the thing that fails, loudly and with a clear message, right at the point where things actually went wrong. Otherwise you get a failure three steps downstream in some assertion that’s a complete red herring and end up burning an hour chasing a bug that doesn’t exist. Assert at the point of failure, not three steps past it.

The usual caveat applies: this can tip over into over-specifying if you get carried away. Guard the things that would invalidate the test if they failed, not on every incidental detail you can get your hands on. As with most things in testing, it’s a balance.

Wrapping Up

Example-based testing is, in my opinion, the most straightforward to test your software. You just pick the inputs and configure an oracle with what each answer should be, technically it doesn’t even have to be automated (although, seriously, please automate your tests).

Unfortunately, that hand-made and straightforward quality that makes it easy to understand is also what provides a limit to its effectiveness. It can only test the cases you thought to write down, and it can only catch the kinds of wrongness you thought to check for. It is unlikely to uncover unexpected failure cases. It is also quite likely that at some point you’ll write tests that don’t test anything, pass by accident, or are otherwise useless without realizing it.

Future Posts

The techniques I want to write about next are, in one way or another, heavily focused on ameliorating the weaknesses of example-based testing that were discussed here, hence why I started with example-based testing. In particular, what I want to write about next are:

• Property-based and Fuzz-testing which are types of generative tests where a testing framework generates inputs in the hopes of finding ones the ones that break your system. This helps you discover the failure modes that you didn’t think about.
• Mutation testing turns your testing apparatus around and tests your tests. By deliberately breaking the code and checking whether your suite notices, it’s basically an automated version of the “how could the code be wrong and still pass?” This helps you detect blind oracles in your test suite.

Although before I get there, I might take a detour into mocking libraries to discuss something that I wanted to include in this post, but decided it would make more sense to discuss on its own.