The issue of testing is divisive. When it comes to testing methods, people are adamant. One of the most well-known examples is Test Driven Development (TDD). There is a lack of clear empirical proof, allowing for bold statements. I believe that testing should be looked at from an economic standpoint. Secondly, I contend that overemphasizing unit testing is not the most cost-effective strategy. Lean Testing is the name I’ve given to this testing concept.
Different kinds of tests have different costs and benefits. You have finite resources to distribute into testing. You want to get the most out of your tests, so use the most economic testing approach.
Building Confidence by Writing more Tests
The article ‘Write tests. Not too many. Mostly integration’ and the related video by Kent C. Dodds express the ideas behind Lean Testing well. He introduces three dimensions with which to measure tests:
- Cost (cheap vs. expensive)
- Speed (fast vs. slow)
- Confidence (low vs. high or click doesn’t work vs. checkout doesn’t work)
The following is the ‘Testing Trophy’ suggesting how to distribute your testing resources.
Compared to Fowler’s ‘Testing Pyramid’ confidence as a dimension is added. Another difference is that unit tests do not cover the largest area.
An end-to-end test has a greater return on investment (ROI) than a unit test. This is due to the fact that an end-to-end test covers a considerably larger portion of the code base. Even when larger expenditures are factored in, it delivers disproportionately more confidence.
Furthermore, end-to-end tests examine the essential pathways that your users walk through. Unit tests, on the other hand, may test edge cases that are never or extremely seldom seen in practice. Individual pieces may function properly, but the overall may not. Martin Sústrik’s blog article “Unit Test Fetish” which demonstrates the prior points.
Code Coverage vs. Unit Tests
Another issue to consider is that code coverage diminishes over time. In practice, most people agree because most projects put the coverage lower bound at roughly 80%. There is studies to back this up, such as ‘Exploding Software-Engineering Myths.’ The following are some general points to consider.
Even with 100% code coverage you trust your dependencies. They can, in principle, have 0% code coverage.
It’s okay for many software products to have the common cases properly function but not the unusual ones (‘Unit Test Fetish‘). You could survive if you miss a corner case bug that impacts 0.1% of your consumers due to inadequate code coverage. In contrary, you may not be able to survive if your time to market rises as a result of high code coverage needs.
Code Quality vs. Unit Tests
It is claimed that unit-testing your code would increase its quality. There are several arguments and some actual proofs in support of that claim, therefore I’ll put the other viewpoint in spotlight.
The article ‘Unit Test Fetish’ states that:
Unit tests are an anti-architecture device. Architecture is what makes software able to change. Unit tests ossify the internal structure of the code
Assume you have three parts: X, Y, and Z. To test them, you’ve created a comprehensive unit test suite. You later decide to modify the architecture such that Y’s functionality is shared across X and Z. You’ve now got two new components, each with its own interface. All of the unit tests are instantly ineffective. Although some test code may be reused, resulting in the fact that the full test suite must be rewritten.
Unit tests raise maintenance costs since they are less resistant to code changes. The burden of recoupling the new modules and their tests will be added to the receipt because tests are also system modules. Check ‘Why Most Unit Testing is Waste’ for these points.
There are significant psychological considerations as well. For example, if unit-testability is important to you, you would choose a program architecture that is simple to test over one that is complex to test but otherwise superior, because you know you’ll spend a lot more time writing tests.
To be clear, I am not a proponent of never doing unit tests. I hope I’ve given you a new viewpoint on testing. I intend to write few articles on how to concretely create a solid integration test for both Flutter and Android projects in the near future.
Here’s what you should do if you want clear – if unnuanced – instructions: Use a strongly typed language like Dart or Java. Concentrate on end-to-end testing and integration. Unit tests should only be used when they are absolutely necessary (e.g. pure algorithmic code with complex corner cases).
Be economic. Be lean.