Guys, I have an idea to implement a testing tool to log and store the execution of the application methods, signature, inputs and outputs.
Generating a volume of data to extract probabilistic information about the behavior and evolution of the application, and possible regression errors.
Have you ever heard of anything like that?
2 Likes
Are you talking about the API requests or the frontend objects on the screen (buttons, text, etc.)? For APIs, there’s Swagger what basically does the same thing. For frontend objects, tools like Tricentis Tosca have this kind of functionality implemented. Don’t know about Open Source software, though.
The big problem with testing today is the time spent implementing it. This time matches the development itself. The idea here is to build a tool that will do everything on its own. Record the executions, inputs and outputs of the methods. And then when something changes in the application and “breaks”, those methods will return different.
You are wanting to look at security tools, and look in the areas about fuzz testing. Where this might be more directly used.
Record and playback apps are complex in nature, timing dependent and a source of bugs per se. I suspect, this is much harder to solve, and is empirical process. Often overlooked is the empirical nature of software testing, it’s not a “new” approach in terms of the tech to do “recording and comparing”, which is basically what all testing is, but curious to know how you would “check” correctness even if you could meaningfully record all api calls?
The idea is to record/log the ordered method calls, starting from a known state of the database. And then play those calls, checking if the outputs match. The big advantage is that you don’t have to write or maintain test scripts. Inferences are probabilistic, based on the volume of calls made. False positives can occur, but over time the application “learns” that they should be ignored.
Emanual, yes. I suspect learning what to ignore is hardest - we did a learning to ignore exercise in a prototype log analyser tool once, it took a while (data volume size) to learn what to ignore. Which means it’s really good for alerting you to changes (aka regressions), but needs help to train up on every version or build. The whole “semantically correct” data problem emerges, and any tool needs to still know what is and is not. In the end we used the tool as a second pair of eyes to highlight or guide us towards “possibles”.
I think it’s a very fun possibly ML project, and now that we see testers engaging with machine learning, this feels liek a good time to try and build these kinds of simplifying tools.