Ask Me Anything: Whole Team Testing

This is a really great question! I think it’s essential that every member of the team is equally valued. On my last team, the developers referred to themselves as “engineers”. They did not consider testers, designers, POs or customer support people to be “engineers”. But they would use the term “engineers” like “Let’s have a meeting with all the engineers” when they meant everyone on the team. Or they would talk about something the “engineers” needed when the testers needed that thing too.

One way this team tried to be inclusive was with a “thing of the week”. Any non-inclusive behavior or language we wanted to make ourselves more aware of and try to change, we’d make a Thing of the Week. So we made one like this: "When you are talking about the team in general, or people on the team who are in different roles, just use the word ‘team’, not ‘engineers’. It had an effect.

Similarly, remembering to not say “you guys” but instead something like “y’all”, “humans”, “mortals”, “you folks” was an effective Thing of the Week.

Personally I don’t worry about semantics like “testing” vs “checking”, everyone on my teams understands the purpose of regression testing versus exploratory testing and other testing activities. But I’ve run into the “That story is done” quite a bit. I bring this up with the team and ask what we can do to make sure that we don’t say “done” when we mean “done with writing the code - we think”. One team put in big letters across our task board, “NO STORY IS DONE UNTIL IT IS TESTED”. Big visual charts are a great way to help people change their language and thought patterns.


In my experience, it’s the managers who are keen to have reports. It’s a good idea to find ways to make the team’s accomplishments and problems visible to management in some kind of concise report. I’ve used old-fashioned risk analysis for this in the past.

My teams have found it more effective to start by setting goals to address our biggest obstacles. For example, if we have too many bugs slipping out to production, we want to have a goal to reduce that. Maybe we want no more than 3 high bugs in production in the next 2 months. OK, what will be our experiment around that? Perhaps: “We believe that having a tester pair with a developer on stories will result in no more than 3 high bugs in production in the next 2 months”. Now we have a metric we want to track, and we want to make that as visible as possible. Maybe a big sign on our physical or virtual story/kanban/task board.

Metrics are so often abused and used to punish teams. And, they’re easy to game. The most useful “generic” metrics I’ve found are from Lean development and from the State of DevOps survey. Cycle time, from when we start building a new feature or story to when we release it on production, is a good one. It shows the length of our feedback loop. Ideally we are slicing that feature into end-to-end thin “learning releases” so we can get feedback from production use and keep shaping it - or killing it because it isn’t what customers wanted. Mean time between failures on production and mean time to recovery of failures on production are a couple of metrics which the State of DevOps found correlates with high-performing teams.

Check out Nicole Forsgren’s book Accelerate based on the State of DevOps survey for 5 years. She shares a lot of the science behind it.

1 Like

We’ve talked a lot about this on the broadcast and in here. My advice is, keep bringing this up in retrospectives, possibly in standups if you are experiencing a roadblock. Always when there is a problem affecting testing, make it a team problem. Ask the team to brainstorm experiments to make the problem smaller. Of course this depends on priorities. For example, if the team is pretty new and has committed to practice test-driven development, and they are really struggling to learn how to do that, I wouldn’t distract them with problems around UI test automation. I would see if I could help them get better at unit-level test cases, and I would look for some stopgap temporary solution for the UI testing until the TDD is going better and the UI testing is then a higher priority. I hope that makes sense.


Alan and Brent say that developers own “code correctness”. I agree with that. Kent Beck called this “internal quality” in his XP Explained book. Developers collaborate with business stakeholders and other delivery team members to gain a shared understanding of each feature and story. They are responsible for designing that code properly for ease of understanding and maintenance, as well as providing unit-level regression tests for it to ensure those small chunks of code continue to behave as expected. Ideally IMO they are using test-driven development. All of that is the internal code quality. It matters most to the delivery team, though it is a benefit to the business because it means we can change the code quickly and fearlessly. The developers who write the code must own this internal quality. That’s why it makes NO sense to have a tester write unit-level tests. I do know testers who get involved with static code analysis, which is another means of evaluating code correctness, and in some cases, this can make sense. But in general what I’ve seen work best is the developers are also responsible for running the static analysis to ensure their code is up to the standards agreed upon by the team, including quality attributes such as accessibility.

External quality is defined by the business stakeholders. They may compromise on lower external quality - perhaps the usability isn’t great, perhaps they know there are bugs and choose not to fix them now - for business reasons. Or they may be quite exacting about it, also for business reasons. It’s dependent on domain, maturity of the product and many other factors.


Janet and I wrote a three-part blog series about how we wrote More Agile Testing: Learning Journeys for the Whole Team. We used much the same process for our first book, though back then, it wasn’t as easy to do video chats and such. We did a lot of google chatting!

We started when we were both at a conference by doing a big mind map for the content for the whole book. This of course evolved over time but it was a good starting place. We divided the book into sections and sections into chapters. We created a release plan so that we “delivered” two draft chapters every two weeks. Each of us would take a chapter, work on it for as long as we could stand it, then trade out. We did a lot of interviews of people on teams around the world and got friends who were expert in different types of testing to contribute sidebars.

We actually had “user stories” for the book from our “customers” perspective. After we had all the draft chapters, we got feedback from our “customers”, a group of friends willing to give us feedback. I remember that we did a big reorganization of chapters and sections after that.

All that time we were mainly working remotely. We worked on the book every day, pretty much. We gave ourselves time off during the holiday season.

With the 2nd book, we took an idea from Ellen Gottesdiener and Mary Gorman that they used when writing their book Discover to Deliver. We got together at my house printed out the whole draft manuscript and put them on the walls. We actually cut. out sections and moved them around and taped them to different chapters or pages. It really helped us visualize the book and see how things would work better. We also printed out all 70 sidebars that people contributed and put them on the ping pong table, then decided where they fit in the rest of the book and taped them to appropriate locations. We also highlighted areas we wanted to update.

It was a fun process, I encourage you to read our blog series on it!

1 Like

Interesting thoughts, I’ve never thought about the idea of there being internal quality within the codebase and external quality, what the business values. I would assume that both combined is true excellence, while if there is too much focus on internal quality you risk overcomplicated solutions, that may be over engineered, while if you focus too much on external quality you risk code that isn’t easy to maintain or build upon.

Thanks, TJ, for asking this! :purple_heart: I have had horses all my life, I was quite the Dressage Queen for decades. Back in the early 00s I was leasing an upper level dressage horse from a friend, and her neighbor rescued some miniature donkeys from an auction. One had been abused and was terrified of people, and my friend agreed to adopt him. I rolled my eyes when I heard of her plan and said, “What will you do with a donkey???” Of course when I first saw Ernest, I fell in love. I spent hours earning his trust. We found out that donkeys need other donkeys, so some friends gave me baby Chester as a birthday gift. I adopted Ernest too because my friend wanted to rescue another donkey.

When my husband and I got our own horse property, and moved Chester and Ernest home with us, we realized that they were too small to defend themselves against coyotes - and we had a lot of big, bold coyotes. I contacted Longhopes Donkey Shelter in Bennett CO near where we lived then to ask their advice. “Come adopt a standard size jenny (female donkey) and she will protect the little boys”. We adopted Marsela and she is the best guard donkey.

All three of them are trained to drive, they pull carts and wagons individually, or in the case of the boys, sometimes as a team. They also do work around the farm like haul logs and brush. They love to work, they love people, and now they are happily settled in here in Vermont!

1 Like

Great question! This has been my experience over the years, too. Often on my teams, we testers ended up doing most of the UI test automation.

As with everything, I made this a team problem. On one team, we started having to add Ajaxy stuff to our UI, and our existing UI test tool couldn’t handle that. I got the whole team together to discuss choosing a new UI test driver library and framework. Our sys admin volunteered to do a proof of concept on Webdriver and found it worked well. Then, again as a team, we looked at various frameworks that we could ALL live with - devs and testers and sys admins. We had a couple rounds of “bake offs” where we automated UI tests for the stories in that iteration in both frameworks to see what worked better for us. We knew we’d be using the tool set for years and we wanted to make a good decision. This investment paid off. We chose a framework, brought in an expert to give us a week of training on it with our own code, and were up to speed right away writing valuable, maintainable tests. The devs were happy to pitch in anytime testers needed help.

On another team, the developers were simply required to automate tests at all levels including the UI for every story they worked on. They all paired full time. I found that developers working alone tend not to provide very good UI test coverage. Having testers pair with developers had better results. Also, we introduced example mapping . The business rules and examples helped the devs think beyond the happy path test cases. We also made changes like moving to using page object.


I love the sounds of this as it’d hopefully engage the whole team … and is light hearted that they’ll actually join in willingly.

I’ve been involved in too many projects where ‘the automation guy’ chooses. The downside is possible blindness to new options and a smaller supporting network.

Getting more people involved seems very healthy and should mean anyone working on automation can then draw on that wider network for support down the line.


I don’t think they are mutually exclusive. XP in particular promoted doing the simplest thing that could possibly work. The internal quality is about doing practices like TDD and refactoring that help ensure well-designed, maintainable code. The external quality does present more trade-offs, but it’s up to the business to make those decisions. Working for a startup, I’ve had to adjust to the idea that we have to get at least a thin slice of a new feature out to production for feedback before it is perfect, which means bugs get into prod. But, we have the means to respond to them within a matter of seconds or minutes, that makes all the difference.

1 Like

I hope there’s more shift to developer(programmer/coder)-tester(we are also developers!) collaboration. In my experience, having testers embedded in a cross-functional team helps product quality the most. I don’t have any science on what the actual trends are.

1 Like

I’m one of those people who follows Grace Hopper’s advice to ask forgiveness rather than permission. If you feel you need the blessing of the business stakeholders, I’d try to put it in terms they understand - money, including opportunity cost. If the team is spending time fixing bugs that weren’t found until the code got to production, they are not able to deliver as much new value. That is a real cost to the business. Not to mention the cost of task-switching, where programmers who wrote the buggy code maybe weeks or months ago have to remember how the code works so they can fix it. Preventing bugs makes more business sense than fixing them. At the same time, you need to bring risk and value into it. There are lots of “bugs” that customers can work around and they don’t care about those bugs if the things they value are working. I feel like zero bug tolerance in production can work, when you’re talking about new code, but if you have a lot of legacy code, it may note be good for the business to spend time fixing bugs that aren’t critical in it.

1 Like

In my experience, the best approach has been for the whole team to continually talk about and plan testing activities. Each new release, each new feature, each new story - how will we test it? How will we automate tests for it, if any? Do we have the test data we need? Do we have the subject matter expertise? A lot of this can come out at “three amigos” type planning meetings as well as team iteration and release planning. At retrospectives, look at the team’s biggest problems, are they related to quality or testing? The team can work together to set goals around solving the biggest problem and design experiments to address them.

That said, I do think most teams benefit from having at least one dedicated, professional tester. These are specialized skills, we can transfer them to others, but other team members have other deep skills. We need different people with different deep skills to solve all our problems and keep doing a better job of delivering value frequently.

1 Like

And, I think it’s not too hard to find the balance once the team has some experience, but starting out, there will probably be bumps in the road!