I often hear statements like “Let’s increase code coverage” or “What’s our code coverage?” (mostly from managers). And I think to myself “Oh, boy…”.

Not because I don’t like unit tests or think they’re useless. On the contrary. I think unit tests are very important and useful.

What I’m against is using code coverage as a metric. Because it can mean nothing. And it usually doesn’t.

Let me explain why

Consider the following class that validates and email address:

class EmailValidator
{
    public bool ValidateEmail(string email)
    {
        // just for demo purposes
        // I know it's not a very good email regex :P
        return new RegEx("[a-zA-Z0-9_]+[@][a-z]+[.]com").matches(email);
    }
}

And a class used for unit testing the email validator:

class EmailValidatorTest
{
    void TestValidEmail()
    {
        Assert.IsTrue(EmailValidator.ValidateEmail("someone@email.com"));
    }
}

What’s my code coverage?

Well, I only have one class with one method. And my unit test goes though that code. That means my entire “codebase” is covered. Yay! 100% coverage!
That means my code is bullet-proof, right? WRONG!

Why? Because my regex, although not very complex, covers a lot scenarios.
I’ve only written a unit test for small letters.
I haven’t written unit tests for emails that are comprised of:

• empty string
• caps
• numbers
• underscore
• alphanumeric characters
• combinations of the above (and there are plenty)

So, what’s my REAL coverage? I would say less than 15%.

What about real life?

I’m not saying code coverage is always unreliable. But it only works as expected for simple cases. Like the following:

bool IsEven(int n)
{
    if(n%2 == 0)
        return true;
    else
        return false;
}

Two unit tests for this code will give 100% coverage and that’s actually all the unit tests you need for this.

But you don’t have code like this in real life. From my experience, less than 5% of any codebase has code that’s dead simple like this.

Most of the time, you use third party libraries that have classes/functions like the regex I used above. Even though I only wrote one line, there are a bunch of if/else statements hiding behind that.

Conclusion

What I disagree with isn’t code coverage per se. That actually helps the developer to visualize the areas in which s/he hasn’t written any unit tests at all.

The problem is most people don’t understand that
high code coverage ≠ code reliability.

You can have >95% code coverage and your application can be unacceptably buggy (I’ve actually witnessed this myself).

Source: https://dev.to/conectionist/why-code-coverage-is-not-a-reliable-metric-327l