A bit of a Friday question has arisen in my unique context. I find a fair chunk of the useful work when you are a solo software QA, is not in re-running regression tests. They uncover very little new information. I find I discover more actionable defects by manually installing and plugging in different wires and cables, something a script cannot do. Who else spends more time exploring than regression-testing?
I’m about to order some network switches and some power supplies this week, when it occurred to me that most of the testing I get asked to do involves setting the product up. Exploring, not regression-testing. A more relevant example : The other day I manually installed two of the plethora of products at the same time, something a test never does, and from different major releases, and the installer baulked. I had seen this happen once before so I managed to find the cause quickly. Our tests test each app standalone and clean, so they were never going to spot this problem either. You cannot realistically use 2 different versions of our products side-by-side. Fair enough limitation, but not something my automated tests would find, because there is very little customer use-case for this. However when a customer tries to, they get frustrated. So it’s a very definite defect. Anything that triggers an email to our helpdesk is a defect or hindrance. That’s just one example. I could automate that negative-test easily, but because a lot of the tests I set up involve hardware, I do end up exploring more than I do writing boring tests.
I first encountered this mindset in probably my 4th job as a QA where I did a lot of mobile app testing. The developers had just added some new features which were impossible to automate correctly and I had to step away from my coding comfort zone for a while and decide how to move forward. After a while we shipped a release and had a chance to talk about ways to get better testability for automation. Fast-moving products will do this to you.
There is always a need to regression-test every night, but in my current job I’m being required to set up things like a wireshark investigation one day, a 10gbit network the next day, VLANS are the next hot topic for me to try and automate. Part of my context being unique, is that there is only one other QA, who tests the hardware. The developers run unit tests and those tend to catch enough bugs before they get into master. Most relevant fact is history, there was no Software dedicated QA here full time before, so I’m playing catch-up a lot. So I feel I have had to find a balance, the regression tests run on TeamCity, but they are not my prime focus, they are an investment, but not the only one. Who else spends a lot of time exploring?