Your feedback needed on a different testing approach


(Jeremias) #1

Hello all!

We are in the process of creating a testing tool with a very different approach to all else there is (or at least all else we know of).

The basic idea is that automated testing is basically not testing at all, but regression checking (following the testing vs. checking train of thought). And going one step further, regression checking is version control in disguise. I.e. it is about controlling changes to the dynamic properties of the software (the look and behavior). Because existing version control systems can’t do that directly, automated tests simply pin those dynamic properties down and transform them to a static artifact (e.g. a test script), which again can be governed by current version control systems.

So our test tool captures all dynamic properties and later compares them to the next version of the software. This approach has been called Golden Master testing. But we now add the capabilities to ignore individual changes permanently (like in any decent version control system) or let the user apply the changes easily.

We have a prototype that works for Java Swing applications, but are currently planning to create one for web also. We’d love to get some feedback on it. Please find it on http://retest.org.

Looking forward to any feedback you have!
Cheers,
Jeremias


Is Test Automation = Regression Testing?
(Jesper) #2

How about the automation activities that aren’t checking?

https://blog.gurock.com/6-technical-testing-skills-that-arent-automation


(Jeremias) #3

Sorry if I didn’t understand that. The blog post is great, but it lists technical skills that aren’t automation.

How do you derive automation that isn’t checking from that? Maybe I missed something there…


(Jesper) #4

In the RST namespace testing is exploring and challenging, while checking is confirmation. When I go there, testing can be both automated and not. Automation to me is various ways of tool support. Perhaps there is a difference between whether the tool does asserts or not.

To me automated testing in that context is very much more that regression checking. As we discussed in the other thread other forms of test automation happens when building the solutions.

That being said, it’s an interesting ability to make version control on looks and behavior. :slight_smile: Look into this one also


(Chris) #5

The basic idea is that automated testing is basically not testing at all, but regression checking

This has some abstraction leaks. The real practice of automated testing is manual testing with tools to help. The tools perform checks, but it’s the agency of testers that you’re ignoring here. That might be okay, but it’s worth noting.

And going one step further, regression checking is version control in disguise. I.e. it is about controlling changes to the dynamic properties of the software (the look and behavior).

This has abstraction leaks. It’s also about the same properties of the software in a changing context - changes to environmental concerns like platforms, time of year, third-party system updates and changes, and so on as well as operational concerns - how the users are using the system and how that changes, changes in styles and fashion, different kinds of users. Also business concerns - what the business wants out of this software may change - the target demographic, cost cutting initiatives, you get the idea.

Because existing version control systems can’t do that directly, automated tests simply pin those dynamic properties down and transform them to a static artifact (e.g. a test script), which again can be governed by current version control systems.

Yes, this is partly why choosing what to automate is difficult. It’s a complex cost-benefit decision beset by contextual problems, including business information and usage information many people have no knowledge of.

So our test tool captures all dynamic properties and later compares them to the next version of the software.
Aaaalllll of them? You need to expound the boundary you’ve drawn around the system. Whatever you’re doing you’re doing it programmatically, which means you’re working within technological limitations, which means you probably haven’t recorded the room temperature or the weather or maybe even the system date. To assume that those things don’t matter is to assume the context of all projects on which your tool is to be used. I’m sure you’ve seen the performance test outputs on that system where the guy shouts at the hard drives. The boundary you draw about what encompasses a software system and what the important dynamic properties are defines its usefulness in context. That in turn defines the useful difference between your tool and any other tool, which defines your USP.

It’ll be interesting to see how it develops, hope this feedback had some use.


(Jesper) #6

@kinofrost Good elaboration on the context around the software. Reminded me that one system in one context is perfect but in another is buggy, even though it has the same dynamic properties. Eg a standard patient system from one country is a disaster in another - due to context & culture.


(Jeremias) #7

@jesper: Thanks for the link. There are many tools that do a pixel-based comparison. Frankly I am not fond of that: it gives you a lot false positives and no safe way to accept the same difference in multiple tests.

@kinofrost: Thanks a lot for the feedback! :slight_smile:
The idea of test automation (as far as I understood) is that, as long as the test is “green”, you can be confident that what the test checks is still OK–no manual effort involved. So you can concentrate manual testing on new functionality, that has yet no automation, or existing functionality for which automation isn’t cost-effective. At least that is the theory, that also matches the “testing vs. checking” discussion.

It’s also about the same properties of the software in a changing context - changes to environmental concerns like platforms, time of year, third-party system updates and changes, and so on as well as operational concerns - how the users are using the system and how that changes, changes in styles and fashion, different kinds of users. Also business concerns - what the business wants out of this software may change - the target demographic, cost cutting initiatives, you get the idea.

I am sorry I was unclear there. What I meant is not that regression checking is controlling changes to dynamic properties due to context changes. What I meant is that regression checking is controlling changes to dynamic properties due to changes to the software itself. Like Wikipedia puts it:

Regression testing […] verifies that software which was previously developed and tested still performs the same way after it was changed.

Whether context changes are important for the software and should be regression checked is up to you and–as you say–choosing what to automate is difficult.

Do you agree on that clarification?


(Chris) #8

as long as the test is “green”, you can be confident that what the test checks is still OK–no manual effort involved
Ish. If we grant that a system displaying “green” does mean that whatever it’s checking is OK then the manual effort involved is writing the check, running the check, looking at the green, and evaluating what that means for the product and the project, including all the weaknesses of that check. This also rests on presumptions about the quality of that check, and whether it’s still valuable in the new light of a shifting context, which leads to the subject of maintenance of the code base. There’s a lot of person hours put into an automation suite.

There are many subtleties in the testing vs checking discussion. For example the subconcious processes that run in our minds while we explore are considered “scripts” under the ET 3.0 definition. While automation checks are explicit scripts, there are also tacit scripts, and it’s only with the agency remaining after what we cannot choose for ourselves that we test in the “test vs check” sense. This, I think, one reason why the proponents of ET 3.0 and the check vs test concept don’t like the terms “automated testing” and “manual testing”, because testing cannot be automated. It is one test effort, done by a human (always, by definition), aided to one degree or another by tools, which may include an automatic check execution tool. From this position we can see that we don’t “automate” the testing of any functionality at all. We check one very specific thing, at one time, in one environment, with one setup, with one set of data, and we do it in a way that no user ever would. That’s why we need a good reason to write a check - one that supports the purposes of a wider test strategy in the mind of a human - and that’s why “automation” is a “manual” process (or, if you prefer, there is no automation or manual, just tool-assisted testing).

What I meant is not that regression checking is controlling changes to dynamic properties due to context changes. What I meant is that regression checking is controlling changes to dynamic properties due to changes to the software itself

I disagree.

Regression testing […] verifies that software which was previously developed and tested still performs the same way after it was changed.

I disagree with this less. This definition doesn’t completely exclude the idea that changes that are not represented programmatically can influence regression testing. Luckily I don’t need to disagree with Wikipedia because Mr Bolton’s beaten me to it with characteristic speed and clarity: http://www.developsense.com/presentations/2012-09-KWSQA-Regression.pdf (page 5).

Whether context changes are important for the software and should be regression checked is up to you

What if I think they are, though? The problem seems to me to be the difference between a programmatic approach where the checks have some innate intrinsics and a human-centric approach where everything is a relationship and testing is a social activity performed for humans by humans. It’s the difference between those views that gives automation checks such limitation.

Do you agree on that clarification?

If I were to say that I did, or that we could assume it to be true for the purposes of this discussion, then we’d hopefully come to some conclusion - but that conclusion would have to be subject to the problems I’ve mentioned. That’s not to say it’d be without value, just that we are aware of its limitations, like we should be with all the tools and oracles and heuristics we apply.


(Jeremias) #9

OK, I can totally agree to that.

To that I also agree. But tests can be automated. Of course it would be better to have an army of fresh human testers that test all the time. But it wouldn’t be efficient or sensible. (“How much time should we spend looking behind us?” – great link btw.) So we use automation to help us. And that is exactly the point:
Test automation as regression testing as version control doesn’t tell you what should have changed (“quality of that check, and whether it’s still valuable in the new light of a shifting context”). But it does tell you if the software changed (if that change is covered by a check). Then you can have a (manual) look and see if this change was intended.

Maybe I should note that I do not think test automation is a silver bullet that solves all the problems. It’s a tool and as such has is benefits and limitations. I just think that establishing to understand automated regression testing as some form of version control (also a tool) instead of actual testing helps both testing and improves regression testing.


(conrad) #10

Yes, when a test can be automated completely, do so carefully. What we chose to automate is something often done by humans anyway, but in an attempt to cover more input data sets or environments, we use a bit of code itself, once again that becomes a cost. And thus the downfall of a lot of automation I have to date seen ,is greediness to automate-test for more environments and future needs.

Personally, I’d like to see more platform-agnostic tooling to guide testers toward the balance for their problem-spaces. For example one area of testing I do automate is at the API level - every product has a web APi, and testing it through a GUI is valuable, but not helpfull, since many customers consume the API, not the web gui, but these kinds of API’s are not only ideal for automation regression testing, they are also ripe ground for overkill automation tooling to test API return data remains the same (static). Yet, in other components, we rely more heavily on actual use-cases where test analysis is done on the GUI as a whole and as a guide to the workflow. Knowing how deeply to check results in the GUI test suite becomes a hang-up every time the devs change the underlying APIs for example. I’s thus like to see a way of everyone tagging tests as being regression tests we banged out years ago in an exploratory fashion, or as being a “check” - based on “written” or obvious requirements. Perhaps an agnostic tool can help us decide, something to manage tests , that actually knows the difference between test types.


(Chris) #11

Yeah, I’m basically with that. Tests can be automated only as very specific, deterministic checks (fuzzing aside), but if we’re okay with that then it can be useful. The problem comes when people make assumptions about what their automation suite is capable of doing - the same kind of assumptions that cause bugs in product code are the ones that cause bugs in test code, which in turn are the ones that allow magic tricks to work. We should be ever vigilant for these abstraction failures for any tool.

I just think that establishing to understand automated regression testing as some form of version control (also a tool) instead of actual testing helps both testing and improves regression testing.

I’ve mentioned some of the costs of abstracting the problem in this way, so I think the next step is establishing what the benefit is over, say, a typical Selenium suite. So how does a version control type test suite work, and what do we get with that over a separate version control system and unit/integration/UI automation framework?


(Jeremias) #12

The benefit is that with that mindset and requirements, we get a totally different tool.

When you design a regression test tool to be an extension of the version control system, you don’t want to have to encode and maintain individual checks. You get something more along the lines of Approvals, TextTest and ReTest. And since we are developing the latter, I wanted to have our most basic assumption discussed here… :slight_smile:


(Stefan) #13

These series of posts might come in handy when thinking about regression testing: