A bit of a Friday question has arisen in my unique context. I find a fair chunk of the useful work when you are a solo software QA, is not in re-running regression tests. They uncover very little new information. I find I discover more actionable defects by manually installing and plugging in different wires and cables, something a script cannot do. Who else spends more time exploring than regression-testing?
I’m about to order some network switches and some power supplies this week, when it occurred to me that most of the testing I get asked to do involves setting the product up. Exploring, not regression-testing. A more relevant example : The other day I manually installed two of the plethora of products at the same time, something a test never does, and from different major releases, and the installer baulked. I had seen this happen once before so I managed to find the cause quickly. Our tests test each app standalone and clean, so they were never going to spot this problem either. You cannot realistically use 2 different versions of our products side-by-side. Fair enough limitation, but not something my automated tests would find, because there is very little customer use-case for this. However when a customer tries to, they get frustrated. So it’s a very definite defect. Anything that triggers an email to our helpdesk is a defect or hindrance. That’s just one example. I could automate that negative-test easily, but because a lot of the tests I set up involve hardware, I do end up exploring more than I do writing boring tests.
I first encountered this mindset in probably my 4th job as a QA where I did a lot of mobile app testing. The developers had just added some new features which were impossible to automate correctly and I had to step away from my coding comfort zone for a while and decide how to move forward. After a while we shipped a release and had a chance to talk about ways to get better testability for automation. Fast-moving products will do this to you.
There is always a need to regression-test every night, but in my current job I’m being required to set up things like a wireshark investigation one day, a 10gbit network the next day, VLANS are the next hot topic for me to try and automate. Part of my context being unique, is that there is only one other QA, who tests the hardware. The developers run unit tests and those tend to catch enough bugs before they get into master. Most relevant fact is history, there was no Software dedicated QA here full time before, so I’m playing catch-up a lot. So I feel I have had to find a balance, the regression tests run on TeamCity, but they are not my prime focus, they are an investment, but not the only one. Who else spends a lot of time exploring?
Yep, this becomes even more important now when AI can infer missed unit tests from the real implementation and automatically add regression tests to cover gaps.
Exploration gets even better when you meet the customers and see their setup, sometimes its eye opening what they have implemented on top of the products or services you work with.
I think you have triggered or somehow highlighted the big thing that I do differently @hvunen. I test systems, not units. I test end-to-end as far as possible, and so I am not sure how AI just can get it’s head around the big and the layers. Since I work in the embedded space, I have to test everything across at least 2 completely physical things, a computer of some sort, and the actual computer that does the work. Sometimes I emulate one of them, but often I don’t, but it’s that system view that breaks the google searches and the AI answers I often get.
Right now I’m measuring CPU load, noticed a new thing that Windows 11 (and new windows10 builds do) in that the cooked counters for cpu load are now giving a new counter that the task manager uses. It’s called “Processor Information” and it has a counter “% Processor Utility”, which is tracking the task manager while the old “Processor” counter called “% Processor Time” no longer tracks. I’ve taken this kind of diversion because unit testing and API testing are only one part of my quality picture. If our product cannot perform, then it fails, we get drop-outs while either the memory or the network or the hard disc catches up. And as such exploration is more valuable. But ultimately doing that exploration helps me build a performance regression tool. Something I can only do if I have a good grasp of the performance bottlenecks in network disc and memory. I better get cracking on it for now.
My context is still software rather than hardware though I test on real devices every day.
Regression risk is primary covered by automation mostly developer owned automation.
My role is focused on unknown risks, the things that the team may naturally miss, the things that because they have not been fully known automation does not cover yet and more recently focused on things agentic coverage will be less likely to pick up on.
I rarely run the exact same thing ever, variation is a strength when it comes to unknowns.
Ideally I want to be focused on exploration aka discovery, learning, investigation 80 percent of my time, once things get known very well they loop into the automated coverage. Some regression risk will naturally be a byproduct coverage of general exploration.
I see this as model of testing and development. When your development model leans more towards where everything is known very well some call it the mass production model testing becomes a verification activity and often exploration needs drop.
When you have a lot of unknowns though, say at the start of the product cycle, or when you add new features never built before or just the the high levels of integrations with other systems that are outside your control these unknowns are more evident, I also put users into this unknown category as even with guardrails there will always be unknowns when it comes to them. Because I usually work in this model I explore much more than verify.
A lot of teams will sway back and forth between models whilst a lot will have strong bias for one or the other.
Anyway I’m not sure it helps in your context but I have found the consideration of what’s already known very well compared to things we have just not discovered yet or are not so sure about and use that to guide how much exploration should be done.
I think the way I have years of low testing visibility before I joined definitely contributes. Having less than one QA per 5 developers is probably also going to shift the goalposts. People rotate jobs and over time even the hardware QA rotates ownership here I have noticed. Which in many ways is a good thing, but does mean that every time a QA comes in, they are resurrecting a lot of old work. I choose not to resurrect too much old work, as the bits I did find were old enough to be oracle, but not to be maintainable nor relevant.
I like how you step back and think of it as “a model” @andrewkelly2555 . Everything about software creation shouts for us to model the process so that it can be better explained and better maintained. High levels of integration, for example is a very definite way of describing where there needs to be more scrutiny of behaviour when we can control it less. I feel I need to re-structure my documentation to reflect better the pointing out of uncharted seas.