6. Varsandan Csaba - Do you make metrics for the found bugs as well? (not just labelling a high, medium, low etc.) Like, depending on their importance from 1 to 5?
In general, I don’t like to rely on metrics like that because I don’t think we can always trust them–and they can be used against people. Metrics that are game-able by their nature are always going to have these issues.
When I put a defect in, I do absolutely put a priority (often our internal priority) and a severity (generally representing the impact to the customer). But if I say ‘oh, we have 5 high impact defects which is too many’, then I find that people will start arguing bugs into lower impact categories to try and get the numbers they think will make the team look the best. This is never going to be a real representation of the quality of the product.
7. Neil M. - What would be good, measurable metrics for external QA hires?
I’m going to assume you mean when you’re interviewing a candidate for a position–please correct me if I’m wrong!
When I’m interviewing or thinking about a candidate, the last thing I want to ask them is about how many defects they’ve found or released to production. Those numbers aren’t going to mean anything to me with out the context of the application and ecosystem in which they were created. What I want to know is how they think. Do they follow scripts? Do exploratory testing? What have they learned recently? Do they read about testing methodology and concepts? How do they approach their job? How do they try to be a better person and tester?
These are the things that let me know if they are going to help foster the quality mindsets we need and advocate for quality practices.
8. Neil M. - Is critical/bugs found pre-production vs client bugs post-production a good kind of metric?
I really don’t think so. Not as a measure of your quality at least–it’s a great measure of your communication with the client or business and a great measure of your processes. It’s also a good indicator of trust between yourself and your client. But it’s not a good measure of your quality–in order for it to be a great measure of quality, you’d need perfect requirements from the client (ie: every case is accounted for, every data permutation is noted) and two to three times the timeline that most projects get.
This is never going to be something we have so we’ll never be in a place that we’re set up to catch all the defects.
9. Honey Chawla - How do we implement this new way of thinking in a large organization where there is a fixed mindset, fixed processes, fixed metrics delivering releases after releases?
The same way you build a bridge–one plank at a time.
It’s hard to change big companies and honestly in a lot of ways, that’s one of their strengths. Another strength that larger companies often have is the ability to experiment and iterate on a small scale. If you can find a pocket where you can have enough control to start trying something new, this can be something that spirals out in a really cool, really impactful way.
But, getting started can be a big thing. Start by looking at your metrics and asking who else is looking at them–why? What are they getting? Are they getting a really neat graph? Are they making team based decisions on that pretty graph? Are they actually not looking at it at all, but need you to turn something in? Is there another metric you could provide that might be interesting and not as direct as fixed metrics (look at something like the Zero Bug Bounce!)
10. Marissa - What metrics do you use to test customer satisfaction on an e-commerce website? Either with the customer being the product owner or the end-user?
I’m a huge fan of ‘ask them’. I like app reviews and net promoter scores. User acceptance testing is also glorious.
The best thing about app reviews is that you can weed out the ones that are content not code–“My food was cold!” “I didn’t like the series finale of Cop Who Is Also A Hairdresser And Also A Ghost!” “I don’t like the person who made this app.”
This lets you actually see where users are having issues and can be almost as good as a bug report–especially if you have analytics to back it up.
It gives you an idea of how the code is actually working for Real People in the Real World. Remembering, of course, that unhappy people tend to comment more than happy people!
11. Vincent Munier - What if there are people (maybe most people?) in the team who would be happier staying at home and twiddling their thumbs?
This is a rough one an really breaks my heart.
You have five types of people in a group or organization: actively engaged, engaged, neutral, disengaged, and actively disengaged. Think of it as a spectrum. On the actively engaged side, you have people who constantly advocate for the team and the company and encourage people to join. On the actively disengaged side, you have people who are working to harm the company. They are working against your organization’s best interests. Disengaged people are like you describe–twiddling thumbs, collecting paychecks, not doing work to support the team.
I find that a lot of times, disengagement comes when people are totally burned out, are seeing no return on their investment, or have lost their reason for doing the job. Sometimes this is because they’re not feeling supported by the team or company. Sometimes it’s because of trauma. Sometimes it’s because they’ve had too much asked of them. There are maybe a few cases where people are just disinclined to work, but I will maintain forever that there are reasons for that and laziness is not an inherent quality of most humans.
Okay, that was a lot of words–hope you’re still with me!
Here’s what I would say: find a way to make their work mean something. Can you offer them a way to make an impact? Can you offer them something new to try? What are their career goals? How can you help create an environment where those goals are real?
Every human has motivations. It can take a while to find them, but when you do, you can make a world of difference in someone.
12. Alexandra H - How much can quality metrics be measured company-wide and how much should they just belong to the team?
13. Doug Ozdarski - When you are surveying several teams in an organization, what are your thoughts on comparing results between teams?
Imma do these two at the same time because I think they’re related enough to work together well.
The results need the context of the team. The scores are all going to be relative at the end of the day–you want to look for trends up and down. There’s value in saying ‘this team seems happier than that team–are there things we can emulate?’ or ‘this team’s client interactions are rated higher than other teams–what can we take away from that?’ If you get into ‘team a has better morale and is thus better so team b needs to shape up’, that’s where it gets dangerous.
Morale among the company is a lot different than teams. The HR team has a different set of objectives than the test team which has a different set of objectives than the business development team. Comparing them might not make a lot of sense.
Comparing two development teams might make sense–but again, you need context. Is one working the legacy part of the app and one just doing API upgrades and another working on the new cool features? That’s going to change their base morale just as a result of their work. Comparing them to each other doesn’t tell you a whole lot. Instead, I’d look for takeaways to help teams that may feel they are struggling.
14. Lilla - I worked for a company which were as transparent as possible and tried something like this by creating a safe place to speak up and monitor the mood within the company. Yet some
times they failed at this. What are the major mistakes one can do when introducing it to a company?
I think a big mistake is expecting it to be effective on day one. Morale can be a trailing metric and you really do need a few sets of data to be able to get a baseline for a team. If you can’t give the data time to mature, you’re not going to have a good experience.
Also, if you don’t act on anything, then you’re going to have a bad time. Safe spaces are wonderful and shouting in to the void can be amazing, but there comes a point where it’s no longer satisfying. I can say “I hate my chair” a hundred time, but until someone says “you can request a new one; here’s the form”, it is going to feel unsatisfying and often leave me frustrated.
You need to have someone involved who can take concerns to the right place without betraying trust. Once you have the safe space, and a person trusted to do good with the information they discover, you can start to see real change.
Okay, I’m headed to sleep for a bit. I’ll answer more tomorrow!