What do you think about AI features for software testers?

​Hi Community :wave:t2:,

lately, I have published a video about AI features for software testers on my channel:

In the video, I am presenting current AI features software testers can already use. I was wondering if you use any of the mention tools in your environment, and what are your experiences with them? Also, I would be interested if you use already other AI tools or features on a daily basis.

I am really curious to see your comments on this topic, since AI is (from my point of view) a topic that is disrupting many industries similar to the smartphones have done from 2007 onwards.

Have a great day!

Best,

Daniel

8 Likes

Hey Daniel!

Any chance you could list out the AI features here? I can’t quite watch the video just yet, but I’m curious!

3 Likes

Hi Daniel, thanks for the video. It’s clear that AI tools will significantly impact the industry, though I believe their influence extends beyond just testing. The entire development and delivery process, including testing, will be transformed. Your smartphone analogy is apt. It reminds me of how chess engines revolutionized players’ tournament preparations. AI tools seem poised to be a similar game-changer for our industry.

Moreover, AI tools themselves require rigorous testing, which opens up a vast and intriguing field of study. Far from becoming mundane, testing is evolving into an even more dynamic discipline.

Currently, many tools are available to streamline routine tasks and enhance daily activities:
Copilot is outstanding (especially with the new Copilot Chat feature) – it’s incredibly useful for code generation, migration, and other common tasks. Intelij IDEA AI assistant is also helpful tool.
AI-powered locators and test data generation are promising areas. I’m keen to explore tools like DevTools AI, although I currently don’t have a UI to test in my role, thus posponed the investigation.

There are many other potential applications. I’m curious about tools that assist with requirements analysis, test documentation, and test design. If these aren’t available yet, I’m confident they will be soon.

2 Likes

I have tried putting graphQL query definitions directly into Chat GPT 4 and prompting for test scenarios and automated checks based on this. The results were a bit mixed but better than I expected.

I have also found the Postbot feature for Postman quite useful.

3 Likes

Hi @rosie sure. Here is the list of features I cover in the video

  • AI Self healing tests
  • AI test code generation
  • AI test data generation
  • AI test case generation
  • AI code explanation
  • AI visual checks
  • AI natural language processing

For each of the features I added some vendors who are currently providing such features:

:link: https://katalon.com/
:link: https://www.testresults.io/
:link: https://testrigor.com/
:link: https://www.virtuoso.qa/
:link: https://www.functionize.com/
:link: https://www.testim.io/
:link: https://www.mabl.com/
:link: https://www.qyrus.com/
:link: https://mostly.ai/
:link: https://tonic.ai/
:link: https://www.synthesized.io/
:link: https://www.accelq.com/
:link: [aqua сloud] - Qualitätssicherung (QS) Management Software
:link: https://applitools.com/
:link: https://www.browserstack.com/
:link: https://www.lambdatest.com/
:link: https://kobiton.com/
:link: GitHub - newsuk/AyeSpy: A performant visual regression testing tool

Hope this helps :). I am currently checking for more AI tools that might be of interest to the community.
Have a great day!

2 Likes

Haven’t heard about Postbot. Thanks for the heads up Paul!

3 Likes

You are right. AI will change the whole industry. I really hope that companies will not do the same mistakes, when they tried to replace testers with automation…

I am really looking forward to all the AI stuff, it’s a bright future for testers, checking what the AI has done, to see if it’s true or just wrong.

Now is the perfect timing to get into this topic, to learn all about LLMs and the tools to be ready for the change :).

2 Likes

Hi Daniel. Very interesting. At 20:08 i propost the https://denigma.app/ tool that i have use it in VS Code IDE.

3 Likes

Thanks for the tool. Something new for me :slight_smile:

2 Likes

I might be unique on this but I feel like AI is almost always misused in a testing context. This is because plugging it into a use case requires a certain degree of fault tolerance and in general testing is about being fault intolerant.

The two exceptions where I thought “wow this really works well” were:

  • End to end testing LLM applications. This means throwing a creative (i.e. high temp) LLM at our LLM customer service app and telling it to be nasty, sarcastic, snide, etc.

  • A machine learning app by Mozilla for deciding the order of the ~48 hours worth of end to end tests based upon the change to get the MTTFF (mean time to first failure) down.

2 Likes

Is there anything that creates test cases/ideas/scenarios?

2 Likes

Generating test ideas and scenarios you can already use chatGPT. Take it with caution. The outcome is still to verify, something it’s just wrong :).

There are plugins from Katalon and others that can generate test cases from your requirements or even from your code comments.

2 Likes

I agree. Many companies doing the same mistake again. Remember the days, when they thought test automation can replace testing :D? Well, we all know that this is not correct. The same thing is now happening with testing!
I think that with the raise of AI, there is more work to do for testers than ever before. The results of an AI need verification. Testers need to get into the algorithms of LLMs.

I think the role of testing will change in the next years. We will use AI to test better and we need to have tools to test AI. Let’s start with our :brain:’s :slight_smile:

2 Likes

Testing is as much about the process as it is about the discovery (of problems).

Finding bugs/problems is only one part, the human and ‘project management’ part is much more complex.

2 Likes

100% Rosie! The sad but is that many companies don’t think that way. I see it on a daily basis. For them, it’s just about saving money in the first place. And if someone is telling them,
“Hey buy our fancy AI tool, and you can replace “resources”!”
They do believe them. To be fair not all companies think that way :smiley:
Many of them still need to learn the hard way that testing is so much more than clicking and tapping on a product…

1 Like

I saw this after asking on Twitter :smiley:

2 Likes

I discovered Postbot from Postman at their AgileTD booth from Postman. Looks awesome, but has some GDPR related issues which I want to investigate further. Am in contact with Postman about this since the AgileTD.

2 Likes

Hi Daniel, thank you for the interesting list. I am glad to see that many test automation tools are using AI. I would also like to mention Tailblu as an AI-powered test automation tool with following features:

  • AI Self healing tests
  • AI test case generation
  • AI visual checks

https://trailblu.com

2 Likes

Hi Alper,
thanks for the link. Will check it later.

1 Like

Take a look at my latest video about AI resources that might boost your testing activities:

:mechanical_arm: https://www.youtube.com/watch?v=CV3UVII3mg8

1 Like