Regarding "when I noticed that some people fixate on a single kind of test, a single point in that space, and claim that those are the only valuable tests"
hope it's clear that I wasn't suggesting that only one kind of test has value. The FIRST qualities of microtests (we didn't have the term back then) suggested that these are the ones best for TDD, because you can afford them in the tight, inner loop of coding.
Other tests have value, but slower tests are unlikely to be run after each minor refactoring step, like a variable rename.
I still remember those 1990s usenet days, when you or Ron or someone would mention these ideas and people would just lose their minds. The development of the coding practices has been joyful for me to participate and to watch. Great stuff.
Inspired by your tweet, I define a unit test as "a unit test is a small test." It is not about the size of the code under test. Small means fewer lines of test code, specifically the first A of the AAA pattern. If the tests can differentiate from each other by just a few lines of code, they are unit tests to me. I usually can achieve everything in your list, even if my test exercises thousands of lines of code, using a database, and there are hundreds of them. My e2e tests using multiple services can achieve almost everything on your list, except they are a bit slow. You might not want to run all of them 2 or 3 times for no reason, but they are bearable.
In summary, I don't do unit testing. My units (whatever that means) do not have their tests. I do unit tests, which are small tests decoupled from the code structure.
Yes, and I like that. I find the "debate" about mocking depressing, particularly when it gets into the region of "mocking is fine, but mocking frameworks are bad".
I am close to the GOOS style, and I do not suffer from the problems that those who say "mocks are bad" suffer from. I see unit tests written without mocks and I often find them hard to understand, due to the size of the setup and lack of specificity.
But others clearly don't have the problems that I do with that style and can easily work with the non-mocking style. So my assumption is that there is something else that is going on.
Just as some people like small functions of a few lines and struggle with comprehension of long functions, and others like longer functions and struggle with comprehension of small functions. Neither kind of person is stupid or bad. It's just different ways of perceiving and working with code. And in that case too, it would be interesting to understand why there is that difference in perception.
Would you consider tests that touch database and/or the file system isolated if they dont affect other tests? I dont have much experience programming professionally but there have been some instances where I found out I wasn't really testing anything by verifying the mocks were called with the right arguments. The "behvaior" I wanted to test was does this record gets inserted into the database given the username is not taken.
Using Docker can help with resetting the database (just start a fresh instance) of course to keep tests fast it does require quite some hardware at the moment. Sometimes I like having one container using multiple databases (one for each test case)
I agree the order wont matter in this case but tests may not be able to run in parallel. The thing that confuses me is: "Isnt the updating a database a part of the behavior? And if so shouldn't tests written in TDD fashion check whether the database was updated rather than checking whether a mock was called with right arguments?"
Do we need to test that the database actually works? I would rather not need a running database for unit tests. I probably would like to have some post deployment smoke tests that prove all my plumbing is in place.
How do I know if my software works as per requirements if I dont test the database? Plumbing is not the only place where database interaction can go wrong. I might execute the wrong insert/update. Maybe I have a new column and I forget to update that?
What about attributes like performance, security, etc.? Wouldn't they need to be kept in mind while writing the tests? In which dimension are they included in?
I've never found a way to test for security. Performance can be tested for, usually at considerable expense. The only unit-scale test you can do is, for example, test that a certain block of code doesn't exceed some number of database calls.
Have heard many say that performance, security, etc. have to be consciously kept in mind when every piece of code is written, so, during TDD, I presume these attributes have to be consciously kept in mind.
Great list. Glad it started from isolation. I've seen too many code bases treating tests as grand architecture of reusable crap that no one can reason about.
Regarding "when I noticed that some people fixate on a single kind of test, a single point in that space, and claim that those are the only valuable tests"
hope it's clear that I wasn't suggesting that only one kind of test has value. The FIRST qualities of microtests (we didn't have the term back then) suggested that these are the ones best for TDD, because you can afford them in the tight, inner loop of coding.
Other tests have value, but slower tests are unlikely to be run after each minor refactoring step, like a variable rename.
That wasn’t meant to be a personal callout. This time 😉
:-D
Thanks for mentioning the article.
I'll try to be on my best behavior.
I still remember those 1990s usenet days, when you or Ron or someone would mention these ideas and people would just lose their minds. The development of the coding practices has been joyful for me to participate and to watch. Great stuff.
Inspired by your tweet, I define a unit test as "a unit test is a small test." It is not about the size of the code under test. Small means fewer lines of test code, specifically the first A of the AAA pattern. If the tests can differentiate from each other by just a few lines of code, they are unit tests to me. I usually can achieve everything in your list, even if my test exercises thousands of lines of code, using a database, and there are hundreds of them. My e2e tests using multiple services can achieve almost everything on your list, except they are a bit slow. You might not want to run all of them 2 or 3 times for no reason, but they are bearable.
In summary, I don't do unit testing. My units (whatever that means) do not have their tests. I do unit tests, which are small tests decoupled from the code structure.
“Specific” and “Structure Insensitive” are in tension.
Yes, too much mocking *can* cause structure sensitivity. And not enough mocking *can* cause lack of specificity.
Different people like different poisons.
I'm not satisfied with "different poisons". I take tensions as challenges--what can I create to turn this into win/win?
Yes, and I like that. I find the "debate" about mocking depressing, particularly when it gets into the region of "mocking is fine, but mocking frameworks are bad".
I am close to the GOOS style, and I do not suffer from the problems that those who say "mocks are bad" suffer from. I see unit tests written without mocks and I often find them hard to understand, due to the size of the setup and lack of specificity.
But others clearly don't have the problems that I do with that style and can easily work with the non-mocking style. So my assumption is that there is something else that is going on.
Just as some people like small functions of a few lines and struggle with comprehension of long functions, and others like longer functions and struggle with comprehension of small functions. Neither kind of person is stupid or bad. It's just different ways of perceiving and working with code. And in that case too, it would be interesting to understand why there is that difference in perception.
Would you consider tests that touch database and/or the file system isolated if they dont affect other tests? I dont have much experience programming professionally but there have been some instances where I found out I wasn't really testing anything by verifying the mocks were called with the right arguments. The "behvaior" I wanted to test was does this record gets inserted into the database given the username is not taken.
If the database is guaranteed to return to the initial state, then the order of the tests can’t matter. The tests may not be able to run in parallel.
Using Docker can help with resetting the database (just start a fresh instance) of course to keep tests fast it does require quite some hardware at the moment. Sometimes I like having one container using multiple databases (one for each test case)
I agree the order wont matter in this case but tests may not be able to run in parallel. The thing that confuses me is: "Isnt the updating a database a part of the behavior? And if so shouldn't tests written in TDD fashion check whether the database was updated rather than checking whether a mock was called with right arguments?"
Do we need to test that the database actually works? I would rather not need a running database for unit tests. I probably would like to have some post deployment smoke tests that prove all my plumbing is in place.
I think that might depend on the complexity of the database. Mocking the database is usually great for performance, but might hide some problems.
How do I know if my software works as per requirements if I dont test the database? Plumbing is not the only place where database interaction can go wrong. I might execute the wrong insert/update. Maybe I have a new column and I forget to update that?
What about attributes like performance, security, etc.? Wouldn't they need to be kept in mind while writing the tests? In which dimension are they included in?
Do you mean tests of system attributes like performance & security? Or do you mean that the tests themselves should be performant & secure?
I meant the system attributes. But yes, tests themselves should be performant & secure as well.
I've never found a way to test for security. Performance can be tested for, usually at considerable expense. The only unit-scale test you can do is, for example, test that a certain block of code doesn't exceed some number of database calls.
I've heard about "Secure by Design", although I haven't personally gone through it yet:
https://www.amazon.in/Secure-Design_p1-Dan-Bergh-Johnsson/dp/1617294357/
Also, DevSecOps by Glenn Wilson gives a layered framework on a general overview level:
https://www.amazon.in/DevSecOps-producing-compromising-continuous-improvement-ebook/dp/B08QRRNX6K
Have heard many say that performance, security, etc. have to be consciously kept in mind when every piece of code is written, so, during TDD, I presume these attributes have to be consciously kept in mind.
Great list. Glad it started from isolation. I've seen too many code bases treating tests as grand architecture of reusable crap that no one can reason about.