Ask Jenny a Question About: The Only Good Quality Metric is Morale

[heather_reid]

Our 9th talk during the TestBash Home conference is “The Only Good Quality Metric is Morale” with the fabulous @jennydoesthings.

We hear so much about metrics in software but are we measuring the right thing for our goals? Are those metrics gameable? What if we tried measuring something different, like morale? Jenny is going to talk to us about how to use morale as a metric to help create a place where quality can thrive and how we can measure morale

We’ll be adding all unanswered questions from the talk here so if we didn’t get to your question, don’t worry, we will find you an answer

If you’ve got a question you’d like to ask in advance, why not ask it now?

Unanswered Questions:

1. shabbir @ CDL - Selenium is o old; why you are still using it? its already been taking by lightening fast stuff (not to mention any here)
2. Kishor Mohanty - Though BS is a wonderful tool, why sometimes it shows different behaviour than the real devices? While issue appears in real devices, BS clears it without any issue…
3. Priyanka - Can this be integrated in AWS code pipeline and Slack for notification?
4. Mandi - What if I want to test an iPhone that on iOS 12, can I do that? or different versions of Android?
5. anagha - how to participate in 99 seconds talk
6. Varsandan Csaba - Do you make metrics for the found bugs as well? (not just labelling a high, medium, low etc.) Like, depending on their importance from 1 to 5?
7. Neil M. - What would be good, measurable metrics for external QA hires?
8. Neil M. - Is critical/bugs found pre-production vs client bugs post-production a good kind of metric?
9. Honey Chawla - How do we implement this new way of thinking in a large organization where there is a fixed mindset, fixed processes, fixed metrics delivering releases after releases?
10. Marissa - What metrics do you use to test customer satisfaction on an e-commerce website? Either with the customer being the product owner or the end-user?
11. Vincent Munier - What if there are people (maybe most people?) in the team who would be happier staying at home and twiddling their thumbs?
12. Alexandra H - How much can quality metrics be measured company-wide and how much should they just belong to the team?
13. Doug Ozdarski - When you are surveying several teams in an organization, what are your thoughts on comparing results between teams?
14. Lilla - I worked for a company which were as transparent as possible and tried something like this by creating a safe place to speak up and monitor the mood within the company. Yet sometimes they failed at this. What are the major mistakes one can do when introducing it to a company?
15. José Lisbona - Who should run this surveys so that it’s neutral for the team (to best avoid biases)? When and how should the results be shared?
16. gaviny - How do you “control” external factors that damage morale?
18. Ezra - Any techniques on how to find your own core values?
19. Elizabeth - What do you do if the reasons for low morale are outside the control of the team?
20. Mike Mc - It seems to be a big jump from good morale to focus on quality. It seems a focus on quality still takes quite a bit of work. Would you comment?
21. Christer Nilsson Ribbing - What difficulties can extroverts vs. introverts fall into?
22. Nicole - I would love to do surveys like this at work but am interested in the overall psychological health/morale of the company (we are small). Do you feel these same example questions would work?
23. Dave Duke - My workplace has got “Survey Fatigue”, have you ever seen this problem and what could be done to help solve this? How could you encourage people to still fill in a morale survey with enthusiasm if they’ve already had half a dozen surveys that month?
24. Ricardo - I really like the hypothesis that Good Team Morale => Good Quality. Is there evidence we can share with managers to support this?
25. Ruslan - What if one of the team members in the Agile team always decided not to vote in any metrics, surveys?
26. Alex Furch - What would be your initial morale boost as a PO or SM when taking over a team that been through a lot (management says they suck).
27. Ruslan - From your point, is Scrum Master any manager? If that is so considered in the company, is that right or wrong pattern?
28. Mike Mc - Is there a book to read on this for further info?

[jennydoesthings (Jenny Bramble)]

6. Varsandan Csaba - Do you make metrics for the found bugs as well? (not just labelling a high, medium, low etc.) Like, depending on their importance from 1 to 5?

In general, I don’t like to rely on metrics like that because I don’t think we can always trust them–and they can be used against people. Metrics that are game-able by their nature are always going to have these issues.

When I put a defect in, I do absolutely put a priority (often our internal priority) and a severity (generally representing the impact to the customer). But if I say ‘oh, we have 5 high impact defects which is too many’, then I find that people will start arguing bugs into lower impact categories to try and get the numbers they think will make the team look the best. This is never going to be a real representation of the quality of the product.

7. Neil M. - What would be good, measurable metrics for external QA hires?

I’m going to assume you mean when you’re interviewing a candidate for a position–please correct me if I’m wrong!

When I’m interviewing or thinking about a candidate, the last thing I want to ask them is about how many defects they’ve found or released to production. Those numbers aren’t going to mean anything to me with out the context of the application and ecosystem in which they were created. What I want to know is how they think. Do they follow scripts? Do exploratory testing? What have they learned recently? Do they read about testing methodology and concepts? How do they approach their job? How do they try to be a better person and tester?

These are the things that let me know if they are going to help foster the quality mindsets we need and advocate for quality practices.

8. Neil M. - Is critical/bugs found pre-production vs client bugs post-production a good kind of metric?

I really don’t think so. Not as a measure of your quality at least–it’s a great measure of your communication with the client or business and a great measure of your processes. It’s also a good indicator of trust between yourself and your client. But it’s not a good measure of your quality–in order for it to be a great measure of quality, you’d need perfect requirements from the client (ie: every case is accounted for, every data permutation is noted) and two to three times the timeline that most projects get.

This is never going to be something we have so we’ll never be in a place that we’re set up to catch all the defects.

9. Honey Chawla - How do we implement this new way of thinking in a large organization where there is a fixed mindset, fixed processes, fixed metrics delivering releases after releases?

The same way you build a bridge–one plank at a time.

It’s hard to change big companies and honestly in a lot of ways, that’s one of their strengths. Another strength that larger companies often have is the ability to experiment and iterate on a small scale. If you can find a pocket where you can have enough control to start trying something new, this can be something that spirals out in a really cool, really impactful way.

But, getting started can be a big thing. Start by looking at your metrics and asking who else is looking at them–why? What are they getting? Are they getting a really neat graph? Are they making team based decisions on that pretty graph? Are they actually not looking at it at all, but need you to turn something in? Is there another metric you could provide that might be interesting and not as direct as fixed metrics (look at something like the Zero Bug Bounce!)

10. Marissa - What metrics do you use to test customer satisfaction on an e-commerce website? Either with the customer being the product owner or the end-user?

I’m a huge fan of ‘ask them’. I like app reviews and net promoter scores. User acceptance testing is also glorious.

The best thing about app reviews is that you can weed out the ones that are content not code–“My food was cold!” “I didn’t like the series finale of Cop Who Is Also A Hairdresser And Also A Ghost!” “I don’t like the person who made this app.”

This lets you actually see where users are having issues and can be almost as good as a bug report–especially if you have analytics to back it up.

It gives you an idea of how the code is actually working for Real People in the Real World. Remembering, of course, that unhappy people tend to comment more than happy people!

11. Vincent Munier - What if there are people (maybe most people?) in the team who would be happier staying at home and twiddling their thumbs?

This is a rough one an really breaks my heart.

You have five types of people in a group or organization: actively engaged, engaged, neutral, disengaged, and actively disengaged. Think of it as a spectrum. On the actively engaged side, you have people who constantly advocate for the team and the company and encourage people to join. On the actively disengaged side, you have people who are working to harm the company. They are working against your organization’s best interests. Disengaged people are like you describe–twiddling thumbs, collecting paychecks, not doing work to support the team.

I find that a lot of times, disengagement comes when people are totally burned out, are seeing no return on their investment, or have lost their reason for doing the job. Sometimes this is because they’re not feeling supported by the team or company. Sometimes it’s because of trauma. Sometimes it’s because they’ve had too much asked of them. There are maybe a few cases where people are just disinclined to work, but I will maintain forever that there are reasons for that and laziness is not an inherent quality of most humans.

Okay, that was a lot of words–hope you’re still with me!

Here’s what I would say: find a way to make their work mean something. Can you offer them a way to make an impact? Can you offer them something new to try? What are their career goals? How can you help create an environment where those goals are real?

Every human has motivations. It can take a while to find them, but when you do, you can make a world of difference in someone.

12. Alexandra H - How much can quality metrics be measured company-wide and how much should they just belong to the team?
13. Doug Ozdarski - When you are surveying several teams in an organization, what are your thoughts on comparing results between teams?

Imma do these two at the same time because I think they’re related enough to work together well.

The results need the context of the team. The scores are all going to be relative at the end of the day–you want to look for trends up and down. There’s value in saying ‘this team seems happier than that team–are there things we can emulate?’ or ‘this team’s client interactions are rated higher than other teams–what can we take away from that?’ If you get into ‘team a has better morale and is thus better so team b needs to shape up’, that’s where it gets dangerous.

Morale among the company is a lot different than teams. The HR team has a different set of objectives than the test team which has a different set of objectives than the business development team. Comparing them might not make a lot of sense.

Comparing two development teams might make sense–but again, you need context. Is one working the legacy part of the app and one just doing API upgrades and another working on the new cool features? That’s going to change their base morale just as a result of their work. Comparing them to each other doesn’t tell you a whole lot. Instead, I’d look for takeaways to help teams that may feel they are struggling.

14. Lilla - I worked for a company which were as transparent as possible and tried something like this by creating a safe place to speak up and monitor the mood within the company. Yet some
times they failed at this. What are the major mistakes one can do when introducing it to a company?

I think a big mistake is expecting it to be effective on day one. Morale can be a trailing metric and you really do need a few sets of data to be able to get a baseline for a team. If you can’t give the data time to mature, you’re not going to have a good experience.

Also, if you don’t act on anything, then you’re going to have a bad time. Safe spaces are wonderful and shouting in to the void can be amazing, but there comes a point where it’s no longer satisfying. I can say “I hate my chair” a hundred time, but until someone says “you can request a new one; here’s the form”, it is going to feel unsatisfying and often leave me frustrated.

You need to have someone involved who can take concerns to the right place without betraying trust. Once you have the safe space, and a person trusted to do good with the information they discover, you can start to see real change.

Okay, I’m headed to sleep for a bit. I’ll answer more tomorrow!