I often hear statements like “Let’s increase code coverage” or “What’s our code coverage?” (mostly from managers). And I think to myself “Oh, boy…”.
Not because I don’t like unit tests or think they’re useless. On the contrary. I think unit tests are very important and useful.
What I’m against is using code coverage as a metric. Because it can mean nothing. And it usually doesn’t.
Let me explain why
Consider the following class that validates and email address:
class EmailValidator { public bool ValidateEmail(string email) { // just for demo purposes // I know it's not a very good email regex :P return new RegEx("[a-zA-Z0-9_]+[@][a-z]+[.]com").matches(email); } }
And a class used for unit testing the email validator:
class EmailValidatorTest { void TestValidEmail() { Assert.IsTrue(EmailValidator.ValidateEmail("someone@email.com")); } }
What’s my code coverage?
Well, I only have one class with one method. And my unit test goes though that code. That means my entire “codebase” is covered. Yay! 100% coverage!
That means my code is bullet-proof, right? WRONG!
Why? Because my regex, although not very complex, covers a lot scenarios.
I’ve only written a unit test for small letters.
I haven’t written unit tests for emails that are comprised of:
- empty string
- caps
- numbers
- underscore
- alphanumeric characters
- combinations of the above (and there are plenty)
So, what’s my REAL coverage? I would say less than 15%.
What about real life?
I’m not saying code coverage is always unreliable. But it only works as expected for simple cases. Like the following:
bool IsEven(int n) { if(n%2 == 0) return true; else return false; }
Two unit tests for this code will give 100% coverage and that’s actually all the unit tests you need for this.
But you don’t have code like this in real life. From my experience, less than 5% of any codebase has code that’s dead simple like this.
Most of the time, you use third party libraries that have classes/functions like the regex I used above. Even though I only wrote one line, there are a bunch of if/else statements hiding behind that.
Conclusion
What I disagree with isn’t code coverage per se. That actually helps the developer to visualize the areas in which s/he hasn’t written any unit tests at all.
The problem is most people don’t understand that
high code coverage ≠ code reliability.
You can have >95% code coverage and your application can be unacceptably buggy (I’ve actually witnessed this myself).
Source: https://dev.to/conectionist/why-code-coverage-is-not-a-reliable-metric-327l