First
of all this article is not a result of any particular project I
worked for. It's intention is not to criticize, but should be
regarded as a personal opinion.
When we start a project it is difficult to explain what my opinion is because it is not in the standard “not at all” or the other side “90%+ code coverage”.
When we start a project it is difficult to explain what my opinion is because it is not in the standard “not at all” or the other side “90%+ code coverage”.
These days most of us
are fans of testing. The most inspiring thing I heard in the recent
years is about companies that are doing several deployments in an
hour. The first thing that comes to mind is that it should have very
hard testing of all kind and testing almost every possible state of
the application and most of the possible combinations in which the
users use it.
I was at a devops
presentation where the presenter said that Amazon is doing these 7
deploys to their life system every hour, amazing stuff, but then the
addition was – don't ask me how exactly they do it, because if I
knew I would probably not be having this conversation with you, but
being somewhere with my million dollars yacht - fishing. So the real
question is how exactly are they doing it.
One widely spread idea
is that of the "test coverage above 90%”, to most business
people this sounds like there is no way such software could have
bugs.
I start my thoughts with this “test coverage above 90%” thing.
Lets imagine we have a small Java class with public method called “pub” and 5 private methods called “priv1” - “priv5”. Every private method has one parameter of type String and converts it to Byte, Short, Integer, Float and Double with a code like Short.valueOf(String) etc. The public method has a signature:
public
SomeResultType pub(String age, String someShortString, String
someIntString, String balance, String someLong);
It looks kind of odd
having the parameters as strings, but lets imagine the input comes
from some text file. To do its job the “pub” method calls the
private methods.
Now lets write JUnit
test for this class.
In a not well designed
code, that test would just call each private method and constructs
the response , If we call the “pub” method with the arguments
given bellow:
SomeResultType
res = pub(“38”,
“123”, “1234567”,
“0”, “1234567890”)
then we will most probably get 100% code coverage with just one JUnit test. On the other side there is something smelly here. Every one of the private methods can accept different type of String inputs, for example null, empty string, not a number string, string that is beyond the data type's boundary or scientific notation number. So lets assume 5 different types of String for each of the “priv” private methods' parameters.
then we will most probably get 100% code coverage with just one JUnit test. On the other side there is something smelly here. Every one of the private methods can accept different type of String inputs, for example null, empty string, not a number string, string that is beyond the data type's boundary or scientific notation number. So lets assume 5 different types of String for each of the “priv” private methods' parameters.
Imagine that if it was
possible to write JUnit test for a private method in java. We would
need 5 different JUnit tests for each of the private methods or 25 in
total.
Because we cant test
private methods in java, can we do the same thing through the “pub”
method?
In this case we could
call the “pub” method with 4 null parameters and change only one.
That will make again 25 in total.
Now lets look at the
signature of the “pub” method without looking at its
implementation. There is no guarantee that the input parameters are
used in isolation. For example the balance could be divided to the
age minus 21 to get how much the person has earned per year after
lawful age etc. If we call the method with empty string for “age”
then we would get negative value which could be a potential bug. Now
even though we already have 100% code coverage we have lots of
potential bugs and this 100% code coverage is only hiding them in the
face of the customer. This also leads to the practice of actually
“hiding” the potential bugs under the carpet. You could say that
we should check your parameters and throw InvalidArgumentException,
to which I would fully agree, but I am talking just for the testing
at the moment. In the real world the realistic combination of
parameter values to a use case can become much larger.
So to make the “pub”
method 100% tested for all the combinations we would need to write
5^5(5 parameters with 5 possible values) or 3125 tests and this is
only one of the public methods. No project has budget to do this.
I think this gives a
good picture that with our single JUnit test that did “100% code
coverage” we didn't do a big job.
I think that more
efforts should be put in testing the private methods which should be
made “package private”, because of the limitation to test private
methods in java, but not protected because this will add them to the
class's contract. I see the private methods as small lemmas which
become theorems when tested and become “package private”. This is
actually also my vision of good code design. So it is easier to prove
these small theorems with JUnit test than the whole, fundamental,
ever changing public method.
Now there are more
problems with our testing approach to the “pub” method. It is
possible that other public methods in the class will use the same
private methods in the future so then we would duplicate our efforts
to test the internals of the class.
Also the chance that
our public methods will change over time is bigger that that for the
private methods if they are well designed. Actually if the private
methods start changing very frequently this is a sign that they are
not well designed and they don't make sense.
You will not see a
detailed article about how really continues delivery is achieved in
reality as you will not see how big investors create their profiles
for example – this is where the real money come to them. You will
read many tails in all kind of books social networks etc... which go
on the surface of the things. This leads to many projects looking at
testing as some kind of panacea and putting most of their budgets on
it and projects fail because of overspending.
On the other hand there
are senior developers doing successful projects by mostly ignoring
it with the smile “Testing is for ….ies”:). Why is this
happening why the “good guys” doing everything right and
everything according to the “good practices” fail more often that
the “bad guys” ignoring it:
1. Testing real life
If you forget about
being software developer for a while try to imagine testing all the
aspects of real life, it would be hundred times more complex than the
life itself. Not something a human can do.
2. Testing fundamental
bad things
You can test
everything, you can test that nobody drinks alcohol or that everybody
is wearing the same uniform at school, the fact that you are testing
it doesn't mean it is true or right.
3. Code coverage
If you tell the
business that your code has 99% coverage, they think it is very
reliable and has only one percent possible bugs, but this is only
part of the story. Most bugs actually happen in combination of
factors like: the customer is 18 years old AND he purchases at 31
December 23:59 AND the product is “xyz”. If you write one test
for customer being 18 years old, another for purchases at 31
December 23:59 and another one for purchases of product “xyz”
you may get 100% code coverage, but you are still not testing all the
possible combinations which may be unlimited.
My will is to design my
software at very small testable pieces of code that make sense.
Trying to test at this very low level. Testing at higher levels is
always desirable but don't expect it to be complete and it should
involve bit of intuition and art, don't go after the 99% code
coverage .
Other things should
also help like reducing the combination with guiding the user's path
through the application, especially for authenticated access to
reduce possibility of security attacks.
Of course some of the
“Agile” principles also help like the “Fail Early” principle,
but “Agile” is also not a panacea.
