Can we talk about quality "engineering"?

After reviewing a lot of abstracts on the topic of “Quality Engineering” for TestBash Spring and TestBash UK, as well as a few past conference talks I’ve seen, I feel like there’s a conversation to have about how this topic is being presented and about the pros and cons of what it advocates.

I’ve done some searching but haven’t been able to find much online that seemed like authoritative resources on what this thing called QE is*, but from what I’ve seen the following are some pretty common precepts:

  • Focusing on “preventing” defects during development, instead of discovering them later.
  • In general, “shifting left” of testing effort/tester focus.
  • Having people in more of a “quality coach” role rather than having “testers”.
  • “Quality is the whole team’s responsibility”.
  • This is a more modern, superior, “evolutionary” approach to testing than other forms of testing.

The last bullet point, in particular, I find difficult to deal with when this topic comes up because it feels designed to shut down all debate or disagreement on the other points. In my personal opinion, many of the arguments, even by well-known speakers, seem to often compare QE approaches to what strike me as straw-man examples of poor testing or development practices.

To be clear, I’d love to have a constructive conversation about the underlying ideas, which is why I’m not calling out specific people. I’m also trying to get a better understanding of the arguments for QE, many of which I think are reasonable ideas if accompanied by an “and we still need responsible, deep testing because we understand that it’s impossible to truly prevent 100% of defects and there are many kinds of potential problems that don’t boil down to defects in code”.

I’d also be curious to learn from some proponents of QE what definition they use for quality.

*At least in a software context. The concept of Quality Engineering when it comes to manufacturing is well established, but not directly transferable from producing physical objects to writing software.

7 Likes

I was with you till this sentiment. I’m not sure I’ve seen or experienced this dichotomy you’re expressing, the idea that it’s QE or nothing?

I don’t think most folks who talk about modern testing/shifting left/whatever you want to call it are say it’s an end all/be all or that there’s a dichotomy between QE and other quality practices? In fact, I think most folks would say that there’s always going to be a role for manual testing (though primarily exploratory), and that it’s not an either/or.

Put another way, I think most folks do advocate for a Swiss-cheese model, so QE practices are just one layer to help deliver software with confidence, but not the only layer.

5 Likes

Not so much that it’s QE or nothing, more that you should evolve from whatever you’re currently doing called testing (though what I’ve seen tends to call it QA) and “transform” to do QE instead as opposed to “in addition”.

2 Likes

This is a more modern, superior, “evolutionary” approach to testing than other forms of testing.

you should evolve from whatever you’re currently doing called testing (though what I’ve seen tends to call it QA) and “transform” to do QE instead as opposed to “in addition”.

So I suppose the questions to test that are: Why isn’t it true? When isn’t it true? For whom is it not true?

I think it might be easier to break that down when we figure out what it is, because I don’t really know and every website disagrees on the definition. I looked up “quality coaches” and now I know where to take my vintage British car for repair in Minneapolis. I think software movements should be named like pharmaceuticals.

So I went and read a lot of coach/engineering abstracts, same as you.

Most of the bullet points you gave I agree with, or are just true. Catching problems on the left saves us the cost on the right. Quality just is the whole team responsibility - the toilet bowl of end-user software is filled with the lunch ingredients of a whole company from the CEO all the way up to the janitor. And I suppose if it didn’t consider itself better at least in some contexts then it would be unnecessary to come up with it.

I don’t know about quality coaches. It feels like something testers should just do, and if the talks reflect that idea (at least one does, from the abstract) I think a talk on how to get into it, how to do it, how to do it better is a good idea. More of that, please. I get the concept of quality coaches, it feels like the not-at-my-desk part of testing. Questions in the design meeting, an “um, actually” in the kickoff, the lunchtime talk you prepare on test’s involvement, someone to blame when nobody does pairing like they said they would, naming no names, Gary. How you’d then live without testers, though, is curious to me, because then the testers become everyone. I suppose in the new world order, where testers never became viewed as the highly-skilled individuals we hoped they would, “anyone can test” really did win. Or perhaps coaches work through their coachees like a tricksy puppet master. My prediction is that quality coaching is more to do with introducing and maintaining ideas of quality throughout the project, like an epistemology ninja with a minor in reducing cycle time. My second prediction is devopstestopstestdevops, the quality coach drilling all the glory holes in the silos. My third prediction is that they then have to test anyway, or you have to hire testers…

… or, my most cynical take to date. Strap in. The flaws in thinking about software testing moved from scripts to automation. We replace the test team with one automation engineer. Or, even better, make the devs do it, they don’t look busy and they’re expensive as it is. So here you are pretending your automation suite is looking after your quality because it says it does on the report, pretending you’re not suffering under the limitations and insufficiency of your test suite, but now you fired all your testers because they were terrible - great testers on this earth don’t seem to fill a Filofax - so where do all the great testers go? Well, if you can promise a deep understanding of testing and quality, which skilled and dedicated testers do, then you can sell the idea that automation is insufficient, because it is insufficient. So what’s the compromise between “automation isn’t enough” and “testers are expensive and bad at their job”? Quality coach. One person is cheaper than a full unskilled test team but not enough to manage the workload, but they can replace the good tester that advocated for kickoff checklists and gave lunch talks on testability. Catch problems early, all that, same as before, but a reduced number of testers swimming in the pool that the agile water falls into.

And my reasoning is that Quality Coaching and Quality Engineering do not need capital letters. Pretty much any team, especially any agile team, should be looking at the start of their pipeline for problems, introducing think twice code once ideas, promoting quality, supporting a project end-to-end and so on, not treating test like a blocker or gatekeeper or project manager. When a quality coach, which I’d say I was for many years, becomes a Quality Coach that feels like a political and economic concern, not a software one.

So, in short, no idea, sorry.

3 Likes

I’m a Quality Engineer, but at the same company was formerly known as (as was the entire department) as a Quality Analyst. This was a culture shift introduced about 18 months ago, by a fantastic leader we had at the time.

I can say that changing our titles from QA to QE did help change the mindsets of Developers and other parties… I don’t want to say ‘taken seriously’ because we were already, but it was more of an appreciation of what we, as QEs, actually do (which was, and still is, an education piece in itself)

I think you’re at risk of over analysing the topic, personally.

  • Analysts analyse… they look at stuff, and feedback.
  • Engineers engineer… they create and build.

So the perception of a Test Analyst is different to a Test Engineer already, just based on that change.

  • Testers test - Both adjectives.
  • Quality - is a noun. We can’t Quality - but Quality is ‘a thing’ that can be created, instilled, formed, spoken about, taught, shown. You get the idea.

@c32hedge I think you’re topic here is a good one to talk about, but its too focused on the title of Quality Engineer vs other titles in our industry. No one is better or worse, and the title is dictated to you by the company anyway.

What I would say is, that if a company is hiring for a Quality Engineer, it probably shows the company has a better understanding of what Quality and Testing is about over a company that is looking for a ‘Manual Tester’ for example - but I don’t think this is a given.

2 Likes

I see at least this points also in my understanding of a tester.
I don’t get why this should be exclusive to QEs.
Am I already a QE?:person_shrugging:

So far I never really understood what are the difference between QE and tester.
All definitions I have seen are very vague for me. Sometimes it even looks like the overlap with developers.

I understand your resistance here. I’m currently working as a Quality Engineer, and some of the good things that could be called Quality Engineering practices, I’ve done before when I’ve had other titles.

And I think this is where I inevitably run into the wall of Job Title vs the description of the craft we practice. Similar to the conversations of Testing vs Quality Assurance, as if it’s either or.

What I can tell you, is that I enjoy having the title Quality Engineer for two main reasons:

  1. It helps me frame those good practice things I was always doing, as a normal part of my role, instead of somehow extra.
  2. It encourages me to think differently about how I approach my work, how I set the expectations on myself and communicate them to others.

Could I do the same, and be called a Software Tester, or any other role? Yes, and I have to an extent. Do I find it easier now I’m titled Quality Engineer? also yes. :slight_smile:

Some more of my thinkings on this topic I poured into a blog post a while back:

It may also be interesting to you, that I see a handful of very different thoughts on what is Quality Engineering, and how to “do it properly”, from different leaders. And I find it very interesting to listen to multiple points of view, an then forge my own way.

Roughy there are two extremes, although there is overlap in both so they are not totally opposite:

  1. Quality Engineering is a practice, that done right is carried out by Software Development Engineering teams. These not only take ownership of quality, but also all testing activities. A single Engineering Manager/Leader of Quality Engineering supports all teams to go it alone.

  2. A variation of 1, is the approche still requires Software Development teams to own quality and do all their own testing, but adds multiple Quality Engineers, or Quality Coaches, to support them. They only coach, support, and ultimately try and make themselves redundant. One QE in this context is likely to support multiple teams.

  3. On the other end, there is the one or more QE per team model. In this model, QE’s are involved in testing activities more directly, while also supporting others in the team to own quality and join in with testing activities. Depending on team maturity, this role might look much more like a traditional Tester role, working towards whole team ownership.

1 Like

Thanks Sam. My intent is to focus the discussion more on the concept/methodology versus the titles.

What I would say is, that if a company is hiring for a Quality Engineer, it probably shows the company has a better understanding of what Quality and Testing is about over a company that is looking for a ‘Manual Tester’ for example - but I don’t think this is a given.

I’ll grant you that one since the “Manual Tester” one is so well-worn and unhelpful, but I’ll repeat the last question in my original post: what’s your definition of quality?

2 Likes

Sometimes it even looks like the overlap with developers.

That’s my impression as well. Having started as a developer, a lot of the stuff I hear about from the QE movement sounds like what competent developers would already be doing, and as a competent developer, I can’t imagine it going over very well to have some other team come in to tell me how to do my job.

Maybe some of that is a reaction of people who were testers working with bad developers and dealing with tons of obvious bugs that should have been caught before they even got to the test team. The nuance I feel like is often missing from these QE talks, though, is the “and”. Let’s prevent defects and continue doing testing to help find the nonobvious things that still slipped through. Let’s shift left and still have people whose primary focus is testing so that we don’t get surprised even further to the right.

4 Likes

I work is SaaS so I get farmed out to very different companies and projects, not only does my title change but often the role itself changes.

Often the CV title we pass to the company changes as does the some of the content to adapt to their language and their needs. As a tester I do not assure quality but sometimes the company looks for a QA so my title switches to QA, I’m not going to go to bat on a title I’m just going to go in and do a good job. If they want to call me a quality engineer that is also fine, mostly there is no real difference if its actually a testing role they need.

The role though does change and I’m fairly flexible with that, Quality consultant is often used with contractor based work and that level does often match a lot of your common precepts mentioned but I still do a lot of exploratory testing in addition to consulting on quality in general, on an other project I may be a Test engineer primarily doing basic automation. So I sort of switch from being a fairly senior consultant to an fairly average automation engineer which I’m fine with as it keeps the skills up and its sometimes a nice break to work on a more narrower scope.

I’ll add a separate comment on some of the things I am seeing in relation to this over the last couple of years.

2 Likes

This is where I’ve had the biggest change in mindset over my career. I used to think that, focusing on the testing itself was the ultimate goals. That if I could just do better at testing, it would add more and more value to the product over all.

Now my thinking has changed, I won’t say evolved because I don’t know if it’s better, but it is different. Now I see testing as a tool that is part of a whole, and not the only one available to me in order to improve product quality in meaningful ways. That is, unless of course we take testing to be an all encompassing term for the craft of improving product quality, and I’ve done that before as well.

So, as a Quality Engineer, I still carry out lots of testing. But now, instead of focusing on passing or failing, I focus on making sure the work I do is visible, useful and understandable by the team. Not to prove my personal value, but because my goal is no longer improving the testing, as a means to an ends in and of itself, but to use testing to improve the culture in the team, to provide information to support changes that will lead to improvements.

Should we also have dedicated testers? I see value in that, especially those who focus on integrations between parts built by different teams, with an even sharper customer focused vision. But it is a matter of scale, budget and value. It won’t be critical in all contexts, for all teams and all companies. And I’m OK with that.

2 Likes

Many companies if they outsource development often do not see the need for separate testers, they believe the developers are perfect and test their own work, they pay for working software so fail to grasp why they pay extra for someone else to test it.

Quality is very complex but the same company if asked if quality is important they say yes, so its a no to testers but a yes to quality. Now testing is a very important quality related activity so its often rebranded as a quality title, if they are interested in quality they need a quality guy or girl and oddly its actually more acceptable to those outsourcing development. This is likely the primary reason for the titles in my experience.

I am primarily a software tester but I have been a developer, project manager and been on old school QA process boards across the whole of development so I also do a level of quality coaching.

I also help people put together their quality plans, you mention “definition for quality”, each time we do quality planning we will also look to align on this, its complex and different for many companies and generally needs to include aspects like their specific business goals, what success looks like etc. I like I think its Cem Kaner’s basic “value to someone who matters” but that is only the starting point of the discussion with the customer and team, as we consider value and who we can end up with a rough definition of quality the team agrees on.

There are a lot of quality related activities considered when planning quality, testing is one of those, code reviews for example are another, the development process itself is another aspect and this bigger picture may be where they are going with this QE title.

I’d see it as very tough for a tester to transition into this broader role and where I’d seen it pushed they ended up being the quality police and go on fools errands like trying to tell the developers how to do code reviews that they do not know themselves, this model is dysfunctional and harmful.

Its my other experience in other roles across hundreds of projects and many companies that enable me to take on that bigger picture quality aspect in addition to my testing experience.

What you do get out of this is a more holistic approach to testing which is a good thing as you plan quality with the whole team but its not really an evolutionary change to testing in general.

2 Likes

This looks to me like a weak definition what testing is about.
A bad tester, someone with bad skills.

I assume you wanted to keep it short and have for yourself a more detailed definition.

I’m confused … this my understanding of being a good tester.

“but because my goal is no longer improving the testing, as a means to an ends”
I follow the philosophy that a not shared test result is a not executed test at all.
Communicating the test results to others is also part of the job. Testing is highly social.

No offense intended and I understand why some the the need for QEs:
To me it looks like a strategy to cope with bad coworkers, most prominent developers and tester.
I fear a bit the situation of solely responsibility, also I can understand if no one does something someone “has” to do.

I do not deny the need by which get some are driven to be/have QEs.
I’m worried about this solution for the situation.

2 Likes

Understand, I’m discussing here myself, as a tester earlier in my own career, compared to myself now. I’m not judging how anyone else tests. I don’t think of this has “bad” or “good”, but as maturing understanding and evolving engagement.

1 Like

So my company uses Quality Engineers and to be honest I was among the first to join the company under this title and let me tell you. They had no definition of what a QE is. So we took what the idea was based off of, SDETs, and modified for better or worse to what we as a community of QEs at my company believe it should be to add the most value for the business and for ourselves. So after meeting with all of our QEs at the company and getting data this is kind of how we expressed it…

QE’s are “Software Developers who specialize in Quality and Test Automation”
QE’s look at quality from the coding layer and work up to the client layer(UI) where QA’s should work from the client layer to the coding layer.
QE’s should focus on engineering, whether it’s automation(primary) or product test strategies.
Automation should be the focus. If we create a product, there should be automation. If there is code written there should be automation around it following the Test Pyramid Standard.
QE’s focus on Validation QAs can focus on Testing
Validation = do a thing 1000x or more and follow the exact same steps and validate pass or fail
Testing = more exploratory/ad-hoc.

Is this the right approach? Time will tell. Is that a QE should do. Not sure, but the business is on board and likes this idea. We’re altering trainings we offer here to try and follow that path. I do like these type of conversations since they’re very on brand for what we’re trying to solve at my current company.

3 Likes

I’m one of those people (YouTube), so I’ll pitch in.

I’d also be curious to learn from some proponents of QE what definition they use for quality.

Let’s start with that. I go by Jerry Weinberg’s “quality is value to some person”. Even though there are versions and alternatives that are more resilient to "yeah, but what if"s, this one succinctly covers the core of what I’m aiming for in a positively phrased way. All my recent experience has been with businesses that sell a product. To get customers to buy, subscribe, renew, upgrade, we need to be able to swiftly respond to their needs (improvements, features, bug fixes) and to the market (competitors, opportunities, risks).

This is the starting point for Quality Engineering, at least for how I see it. Not “given we have a product team with a tester and a “Ready for Testing” column, how can we test well?”, but rather: "considering we want to deliver maximum value to the customer and the business, what should our team, our communication, our processes, our pipelines, our test automation, our release strategies, our observability, etc. look like? You can also do those things without calling it Quality Engineering, but there are a few common consequences which might be a more difficult pill to swallow.

Let’s get to the concern you mentioned:

reasonable ideas if accompanied by an “and we still need responsible, deep testing because we understand that it’s impossible to truly prevent 100% of defects and there are many kinds of potential problems that don’t boil down to defects in code”.

Not everyone needs that.

In a Quality Engineering context there can still be room for skilled exploratory testing. A human is much more efficient at (or solely capable of) catching certain issues. However:

  1. It might not look the same. It might be more in the shape of pair testing and start earlier in the development process instead of with a fully integrated solution on an acceptance environment, but in some contexts there can still be significant return on investment for deep human testing as long as the feedback loops are extremely short.

  2. It might not be necessary. If you’re Facebook, for example, getting things out there quickly and learning from (and responding to) actual feature uptake and actual user behaviour is more important than getting it 100% right on the first try. Plus, you can use canary releases to significantly reduce risk and with your lightweight development lifecycle you’re able to fix issues before too many users even know it was there. And even if you don’t fix the issue at all and decide to roll it out to all users because you’d rather have people using the new feature, a social media platform’s revenue is not going to be impacted as much as a medical device manufacturer would be, if at all.

In many cases (especially for fast-moving product teams in web technology) I think the common precepts you listed can be essential to adopt. However, I’m open to counterpoints, since I’m on a perpetual journey to learn what works best for a given context.

4 Likes