We all know that the API has a specification. We all write tests to cover “happy paths” and to cover “unhappy paths” when work with the API.
It may seem that we’re pretty safe if we have tests and follow the specification. Well, actually no. Usually it works well in theory, but doesn’t really work in practice.
We can write as many tests as we need to cover all edge cases, but that is both time consuming and error prone. The best choice would be is to automate this process.
So we made a small library to make our life easier - Fuzzer.
The basic idea is to generate many many derivatives of original sample data and feed it to a consumer.
For instance, from this sample
1 2 3 4
we could get two mutants by using
ReplaceNode mutation, such as:
1 2 3 4
1 2 3 4
There are many decisions our serializer/mapper/whatnot can make when it gets such a result: return an error explaining why it cannot convert the dictionary into a model, or silently ignore “number” field, or any other action. In general this decision is up to you and your domain model.
But I’m pretty sure there is no domain model that accepts exceptions like
-[__NSCFNumber length]: unrecognised selector sent to instance 0xDEADBEEF.
The use case may vary and depend on a workflow. The recommended way is to create another test target and put these verifications there.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
It’s not very elegant, but it does the job very well.
user.name.length; may look weird, but it’s the easiest way to validate that
There might be a category
NSNumber+length, but AFAIK our team members doesn’t have such mutations.
Example looks pretty straightforward, but I’d like to highlight couple of things.
The block here does nothing but catching exceptions and recording them into internal storage within mutant that caused the exception:
The engine records only failures, so it’s enough to assert count of returned reports. One can easily examine what caused failure by looking at report, e.g.:
1 2 3
The library provides just a couple of built-in mutations.
Delete Node Mutation
Returns sample without some node so that
1 2 3 4
1 2 3
1 2 3
Replace Node Mutation
This mutation replaces value of some node with another, ‘random’ value as shown in the very first example above.
‘Random’ quoted not because of nature of ‘randomness’ in general, but because the replacement can be provided using
That is, mutator will generate mutant with this replacement for each key/value.
The library provides finite set of built-in mutations and replacements which is, probably, not enough for real life usage.
Fortunately, there is an easy way to extend it.
Previously we were changing only content of sample dictionary, now it’s time to go deeper (actually shallower).
Let’s create a mutation that will return random value (using
NodeReplacement) instead of changing a sample.
Simplest interface ever:
1 2 3 4 5 6 7 8
Implementation is not sophisticated as well:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
The usage is trivial:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
That’s pretty much it
Please, don’t crash when your server lies!
P.S. I do consider to include
ReplaceSampleMutation into the set of built-in mutations.
I will be more than happy to review the Pull Request.