What's wrong with Test Driven Development?

🗄️ Archive
1

In a recent webinar on test automation trends for 2018, there was a question and some discussion about BDD and TDD, in which Gil Tayar said “nobody mentioned TDD which from my point of view is a good thing”. I think the sentiment was that as long as you’re testing you’re fine and not to worry about adhering strictly to TDD.

But it did make me wonder: what do testers think of TDD? Is it wonderful when done right or completely overrated? Maybe it’s not worth extra effort, or doesn’t add anything, or has some specific drawbacks even when done correctly?

I have no direct experience with it, but usually when I hear it mentioned it seems to be in a way that positions it as an ideal that nobody ever achieves, which is why the comment suggesting it’s not even worth mentioning struck me as interesting.

1 Like
2

I may be in a lonely group of one with this opinion but I think TDD is largely tangential to the work that we do (those who directly test APIs and white box testers who write unit tests probably exempted). It traditionally states that developers write their unit tests and their methods interfaces before writing their implementation - utilising the principle of “red, green, refactor”. Done properly, it is a really good practice, however a bit too low level for the work most of us do.

It is really more of a code quality and design technique.

7 Likes
3

It depends on what you mean by all of those terms. If TDD means “write a check, see it go red, write code, see it go green” then TDD is a checking technique for programmers to keep themselves from making mistakes. It is testing in the same way that me unblocking a sink is plumbing. It is plumbing, but there’s much more to plumbing and I’m not a plumber. Is it test automation? Not really, although there are automated checks involved - but their purpose is to support a design and programming strategy, not a test strategy. Some people like to stretch this idea until it snaps by using TDD/BDD artefacts as automated tests, and I think this is why there’s some relief about it not being mentioned in the Webinar.

I could go on and on about the boundaries and responsibilites of automation suites in a test strategy, but I won’t unless you really want me to.

If you’d like to explore the gap between BDD and testing this series won’t steer you too far wrong, I don’t think: http://www.mwtestconsultancy.co.uk/bdd-testing-part-4/

Other resources:
https://www.linkedin.com/pulse/bdd-isnt-test-automation-adrian-kerry
http://www.satisfice.com/blog/archives/638

4 Likes
4

So is it that TDD, as a way of generating unit tests, is just a small part of what testing needs to consider?

@paulmaxwellwalters - Why would a code quality technique be tangential to what testers do? Isn’t quality (by some measure appropriate to the context) what we’re in this for?

@kinofrost - why wouldn’t you consider it test automation, assuming the tests from TDD become the unit tests? Aren’t they tests? Aren’t they automated?

I wonder if the comment in the webinar meant that it’s not something testers need to mention since it’s a thing developers do. I generally don’t think in those kinds of terms though (as my questions above probably make clear) so that meaning would have been lost on me.

1 Like
5

@gpaciga Code Quality itself is very much not tangential to what we do, however most testers in my experience (apart from those who are deemed SDETs and/or are charged writing or looking at the output of unit tests - a minority of testers I would say) would never deal with the code during TDD. It is mostly done by developers and software architects at the design and implementation stages.

However I believe that this may change over time as testing teams “shift left”.

2 Likes
6

Aren’t they tests?

No, they are coded checks.

Aren’t they automated?

No, testing is never automated. What they are are structured input for a tool which have to be written, and the results observed, interpreted and acted upon by a human.

Why wouldn’t you consider it test automation?

“Test automation” is a bad way of saying what it really means: “using tools to help testing”. You can also use tools to create checks to help programming (TDD). This is testing in the same way that me checking my bank balance makes me an accountant. Is it not the inspection of financial accounts? Isn’t that what accountants do? Well, yes, but nobody would seriously call me an accountant or say that I’m doing accounting because accountants are experts in the field who do many other things in the pursuit of different information to make different choices with reference to a different body of knowledge. Their intent is different to mine, at the very least in terms of scope.

Testing is bigger than checking and bigger than tools - and “automated” “tests” are artifacts of checking tools.

assuming the tests from TDD become the unit tests

Are they written to serve a test strategy that mitigates risks deemed valuable in context? Or are they written so that a programmer can make something properly? The first is a well-designed unit test, the latter is a TDD artifact to shorten feedback loops in programming. Maybe one check can be both things, but the intent of use is different so you may get different results.

So yeah, we could say that TDD checks are checks and therefore part of testing and are used in tools and therefore part of automation. We could also say they are written using English words and therefore a short story or poem. Technically true but pragmatically misleading. I would say that automated testing (or, as I’d call it in my own head, “tool-assisted testing using an automatic check suite”) is part of testing. TDD is part of an approach to programming that leverages some of the tenants of testing.

Here’s one article on the difference in intent between testing and TDD: TDD Is About Design, Not Testing | Dr Dobb's

Here’s one that separates unit tests from the TDD practice: http://blog.xebia.com/tdd-not-unit-tests/. I don’t agree with all of it but it makes the distinction.

Here’s one about TDD as a lifestyle, unit tests being only a part of that: My Version of "TDD is About Design, Not About Testing" | Codementor

3 Likes
7

The parallels between double entry bookkeeping and Test Driven Development are deep and plentiful.

By Uncle Bob (Martin)

blog.cleancoder.com/uncle-bob/2017/12/18/Excuses.html

2 Likes
8

@paulmaxwellwalters - Thanks for the clarification, I think I’ve been lucky in that I’ve only ever worked very “left” oriented teams, so I’ve never not been involved in the architecture, design, and development. I’d feel very claustrophobic if that changed!

The theme in most of these references seems to be that TDD is a design/development practice, but also that the benefits you get from it can be useful in testing - the artefacts from TDD can be good or useful tests, and it tends towards producing higher quality code. So maybe I can agree with “TDD is not testing” or “TDD is not part of testing” but not something like “TDD is not something testers should care about”.

@kinofrost - we’re on the same page about “test automation” being problematic but in as much as “test automation” is a term people like to use, if you’re using TDD artefacts as part of your testing (is there any reason they can’t be part of a test strategy?) then it certainly seems like it they would fall under that (unfortunate) umbrella. It doesn’t matter to me that the developer is the one writing them.

Anyway, I’m just poking at this idea of TDD not being testing to see if it really holds up, but it so far it doesn’t sound like there’s much wrong with TDD. Maybe that’s a question I need to ask on a a developers’ forum!

1 Like
9

is there any reason they can’t be part of a test strategy?

Sure, you can use them in testing. The problem is that you have to understand a lot about the limitations of what those checks are built to achieve, especially when they’re given names that are highly lossy abstractions of their actual behaviour. When you have that then you have to justify their cost. But TDD is not FOR testing. It contains some checks, and checks are sometimes used in testing, but checks written with the intent of aiding the discovery of important problems in the software will look different, and be understood by a tester (whoever that may be).

I’m going to answer the question “is there any reason that keeping them shouldn’t be part of a test strategy” instead. Yes. An automatic check suite is a development project. It has all the coding costs, maintenance costs, process costs and opportunity costs of any other development project. That means that if it doesn’t have value equal to or greater than that cost it’s a waste of time and money. Writing checks and throwing them away is a great way to get all the benefits of the tool (apart from pseudo-repeatability in an attempt to spot regression problems) without the cost.

But there’s a trap. If the cost is incredibly low then the checks need not have much value to be kept. Then you’re in the position where the check suite grows over time and begins to rot and cost cycle time to run, plus maintenance, etc. As time passes the idea that the checks are performing their described functions becomes normal in terms of culture and process. People begin to believe that “test valid login” really is testing for valid login, and a suite of disparate checks becomes a coverage map with more holes than the plot of the Disney movie “Holes”, protected by an illusory status of ceremony. It becomes a belief system surrounding a false idol like a series of pigeons in Skinner boxes pecking at the coloured circles to maintain their supply of food pellets. Try removing the checks and you’ll find the fear in their eyes. I’ve seen it. I’ve seen the fear.

it doesn’t sound like there’s much wrong with TDD

There isn’t. It’s just not testing. It’s good at what it’s for, which is writing well designed code. It’s not there to find important problems for stakeholders, just aid thoughtful and well-paced design in programming. If you want good code, TDD can be good. If you want good testing TDD is not (directly) important.

How about this: Any sense in which TDD is creating tests for testing is one tangential to the definition of TDD. If TDD ever creates testing then that’s not a part of it that’s TDD.

1 Like
10

As a software developer this is what I can add to this Topic.

TDD is a method of writing software where you first write an implementation via unit tests of your functions and subroutines.

There are a few advantages for developers here

  1. mental help in writing a usefull function since you first write an implemenation.
  2. make sure nothing else breaks when changing one piece of code while covering all of your business logic code (calculations, simple and complex if/else statements etc…)…

These unit tests can be set up to run automatically after nightly builds…

When working in team, a collection of unit tests will then guarantee working business logic accross the team.

1 Like
11

It almost sounds like you’re saying that it’s not worth having automated checks or regression check suites at all (I are not say “automated tests” in this conversation any more!) Or at least that there’s a very high bar to make the benefits worth it.

1 Like
12

what do testers think of TDD?
A tool or practice in the developers box, done by developers to help them build code. A choice for each developer to make whether they want to use it because they help them or not.

Is it wonderful when done right or completely overrated?
Can be wonderful, can be awful, can be completely overrated, can be completely underrated. It is not the idea itself that drives the success but how,why,where,when,what the developers are using it.

Maybe it’s not worth extra effort, or doesn’t add anything, or has some specific drawbacks even when done correctly?
Can go both ways…same as above.

In testing we’re using the same kind of judgements, when we have to chose, based on the context: choosing what tools to use, what to test, what scripts to write, what kind of visualization like mind mapping or tables would help, and so on.

2 Likes
13

The bar is variable. I guess I’m saying that if you’re going to spent time, money and skills on automation you must first know what automation can and cannot do and make informed decisions. Automation is a great way to, with reasonable reliability, check specific important facts about a product, provided one considers what is not checked and if that requires more testing, and provided the software lends itself to automation, and the checks are valuable, and they are well-written, and they serve a test strategy, and the results are interpreted by a human, and the results are acted upon sensibly, and they have a listed purpose so they can be removed or updated when required, and they are regularly reviewed or have an expiry date, and the technology doesn’t change frequently, and they’re not brittle, and they don’t misrepresent what they do or are interacted with by people who understand the misrepresentation now and in the future, and they’re written so that others can understand what they are doing and why.

There’s also TDD which is a whole other thing to do with writing code that may or may not care about some/all of the stuff I wrote above.

1 Like
14

Hi, out of interest what was the webinair you viewed? I’d be interested in giving it a watch!

1 Like
15

I’ll tell you what I think of TDD based on my limited experience with it. I tried a little bit of TDD by reading a book and taking a course. It sounds like TDD could be helpful if you want to develop utilities or frameworks for testing. But, I am not going to pursue it further.

Unfortunately, there are few resources which can teach TDD well with realistic examples. I suspect that the most effective resource would be a reputable practitioner, but they are rare and their hourly rates might be too expensive. I haven’t seen many people in the testing community using or promoting TDD. It appears that not too many people in the developer community practice it or get excited about it.

TDD sounds like its a magical tool which will revolutionize software development, even if its promoters did not intend to say that. It has been around since 2002 and is practiced by some famous people who also promote it. But, I find it odd that it has not become popular and widespread in the industry and learning centers (e.g. colleges) despite being around since 2002. Maybe its way ahead of its time, or the industry is too slow to change, or it has limited applications or maybe its just hype. I strongly suspect its just hype. I feel that its famous practitioners just haven’t made a strong case for it. Moreover, they haven’t provided any high quality training material to the public to try and judge it for themselves.

1 Like
16

Well said, and aptly put! I wonder if testers use TDD. Developers use it while developing code.

1 Like
17

@testervenkat - Do the developers at your company use it? Developers at famous companies and also obscure startups told me that they don’t use it their company. I know that this is just a few people and a few companies. But, I wonder if TDD has been adopted widely and if not, then why?

1 Like
18

The current team of developers I work with do get high coverage from unit tests. And this is great for me, because instead of trivial errors I can focus on system integration tests and acceptance criteria.

I don’t think we are using Red, Green, refactor. I think we are just mandating unit tests are introduced for new code before it is merged into main branch.

I have worked testing code that hasn’t been we unit tested, I definitely prefer with unit tests.

1 Like
19

I very often use TDD as a way of writing code in any situation where it could help me prevent the classic “bottom-up” programming. It’s almost my default now for the last 2 or 3 years. As @kinofrost points out, it’s not a brilliant way to be writing automated tests as a tester.

TDD forces you to architect code with the SOLID principles in mind, and since SOLID goes out of the window in 90% of test framework languages it is less helpful there. TDD coding is like having a second pair of eyes on your code while you write it. Much like automated tests are a second pair of hands when it comes to testing. That’s not to say your testing code can not use unit tests, I work in a homebaked framework, where any common Python modules that are generalized and shared across the entire system, have their own self-tests, which help us when bugfixing these modules. So these modules that have unit-tests, I have found easier to write in a TDD style because they still need to meet contracts. Additionally these TDD tests also become good sample code.

As for the original question, what’s wrong? Is that people think it makes your bugs go away, and that’s bollocks. “People” make bugs go away be using the right tool, at the right time.

2 Likes
20

As a tester I like to use TDD to increment test code in small steps. In thie following blog post I tell about using it in test automation:

2 Likes