🤖 Day 2: Read an introductory article on AI in testing and share it

30 Days of Testing
, ,
41

How to gradually incorporate AI in software testing.

AI in testing is an innovation that dev teams hesitate to make it part of their work. Misunderstandable results, different outcomes from different tools and other examples create the impression that something goes wrong with it. And QA engineers do not want tthings to break.

Introduction to AI in software testing is a gradual process and should have trust on it. There challenges to be faced like high costs, lack of human oversight and possible incompatibilities with the existing conventional systems. But with the right strategy and the correct evaluation of the toola the benefits overcome them.

Benefits to using AI in software testing include improvment in:

Accuracy: Faster executions of test cases with more precision and fewer errors than humans.

Coverage: No time and resources limits as humans. AI-based tools can generate a large number of test cases and run them in parallel.

Efficiency: Do the intense and repetitive tasks for humans. Offer insights into jobs requiring high focus like visual tests.

A good mindset to evaluate a tool is to answer the next questions:

How does the tool account for user feedback?
How does the tool handle changing needs as teams and systems mature?
Which time-consuming tasks can the tool take on?
What level of analysis/recommendations does the tool offer?
Which technical/analytical skills must the team possess to use the tool?
How does the tool help the team evolve its testing process while building quality engineering skills?

As I work in test automation in my project, I believe that a self-healing AI tool will be a good fit. I also believe that the challenges in such a use is how accurate will be, but also the most important, how efficient will be the provided solution. For example, is the tool reliable enough to provide not only a correct solution, but the best one, in case of an x-path expression that was changed?

Link: [How to gradually incorporate AI in software testing | TechTarget](https://How to gradually incorporate AI in software testing)

2 Likes
43

Good questions! We all know AI can do our job but how do we leverage and where do we integrate it in our current project is important to understand.

Ex:How do we feed in quality input for the AI to generate test cases

2 Likes
44

“AI in Software Testing in 2023: Benefits & Challenges”:

This article discusses the advantages and challenges of integrating AI into software testing. It covers topics such as smart test selection, machine learning, and computer vision bots.

Key Points:

  • Smart Test Selection: AI can determine necessary tests and eliminate redundant ones by collecting and analysing information automatically.
  • Machine Learning (ML): ML can generate test cases, audit existing tests, and run tests. Training ML models using relevant datasets is crucial.
  • Computer Vision Bots: These bots recognise elements in code (e.g., login boxes, shopping carts) and adapt tests even if the code changes.

I would add:

  • AI can help speed up tasks that rely on processing large amounts of data, which is useful.
  • For any AI to be really useful one should train a purpose built model. This could potentially also address some of the recurring concerns about bias.
  • AI will support humans in quality critical tasks, in areas where machines shine (repetitive tasks that have clear boundaries), but human input is and continues to be very much required.

Link to article.

4 Likes
45

The article on the Dunning-Kruger effect doesn’t really make sense. The correct explanation of first order behaviour is normally phrased in terms of regression to the mean, plus a general belief by everybody that on average they’re better than they actually are, which gives precisely the same graph. Of course random data shows a similar trend of regression to the mean, without the bias term; a random variable X has no idea what an independent random variable Y is scoring! Ignoring the positive bias, the whole point of the Dunning-Kruger effect is that people have a mediocre internal analysis of how good they are at anything.

2 Likes
46

Fuzzing seems to be one of the most direct AI applications in testing, especially since early fuzzers already used genetic algorithms, so it’s really a “let the computer figure it out” kind of field.

The DeepFuzz paper introduces a neural network (well, two RNNs, char-by-char) to generate C programs for fuzzing. They found 8 bugs in GCC including a segfault!

3 Likes
47

Google have gone a bit more heavy handed, linking up LLMs with their existing fuzzing framework. Less detail on what’s actually going on there, but it seems like an obvious application with people gluing ChatGPT to everything now…

The other way around, the TensorFuzz paper does proof of concept (non-AI powered…) fuzzing for neural networks, where the notion of “code coverage” is replaced by “graph coverage”. They find NaNs in a (badly?) trained model, and get what look like pretty good mis-classification results for simple classifier 32bit models quantized to 16bit.

2 Likes
48

Hi, everyone,

I read Testsigma blog Ai in Software Testing / Why it is important in Software Test? https://testsigma.com/blog/is-ai-really-important-in-software-test-automation/

Author provides detailed comparison between Manual Software Testing and AI Software Testing, also indicates various methods for AI-Based Software Test Automation, discuss, what task can AI Software Testing help with and which not, AI tools in the market and of course benefits of AI Software Testing.

What was interesting:

:small_blue_diamond:It is estimated that about 70% of the tasks a Manual QA tester performs can be automated via AI.
:small_blue_diamond: AI Software Testing can make testing faster, more accurate and cheaper.
:small_blue_diamond: It helps automate the testing process, such as creating test cases, running tests, analyzing results, and spotting errors.
:small_blue_diamond:But despite of all AI will not replace human involvement in software testing.

3 Likes
49

One of my greatest aspirations is to be able to expand a leg of my skillset in game testing area, so I did a bit of a research about how AI might impact testing processes in game development. There are several good articles about it, but a presentation by SEED that took place in GTC 2021 piqued my interest. Here’s a link of the summary of the event (presentation link can be found in it): GTC 2021: Towards Advanced Game Testing With AI (ea.com)

Aside from the standart software development processes for a game, the game and its components must be tested for elements such as gameplay, UI, physics, etc. Prolonged testing of these can take a very long time, and it also requires a large amount of testers depending on the scale of the project. The presentation describes that ML and RL in particular can be used and improved upon as a method for game testing and can be used to reinforce the development phase. The latest articles I found regarding content generation methods such as the Generative AI also supports the latter part. The presentation also states that, even three years ago, a well taught and reinforced RL process could find exploits even without being explicitly told to.

Reading this presentations and the more recent articles I found back to back shows how far the usage of AI came in the gaming industry, and how much more it has to go. Of course, what it means for game testing 'enthusiast’s like me remains to be seen, but I can safely infer from these articles that the manual testing aspect of video game development is slowly being replaced by a presence that is far more efficient at its job and profitable to keep around.

2 Likes
50

I found this article that gave an insight to AI in software Testing. This gave an introduction to AI in software Testing

Uses of AI in Software Testing

  • Automated Test Generation
  • Defect Prediction
  • Test Execution and monitoring
  • Test data generation
  • CI/CD
  • Performance Testing

There are many more uses that can be explored and used in daily activities of Software Testing.

4 Likes
51

The article I chose is from Maaret “How AI changes Software Testing”

In this article Maaret shares her experience with AI and such technologies.

This is one of my favourite sentences and I hope these next few days over March will be like this for me. :slight_smile:

I have laughed with AI, worked with tricky bugs making me feel sense of powerlessness like never before, learned tons with great people around AI and its use.

The initial thing that struck me about the article is that it states that AI in tech is not new. Early adopters have been using a form of AI tech for a while and the future is here now.

On reflection we use the term AI, but there are so many facets such as machine learning, large language models, and more that all go under the umbrella of AI.

We need to understand when we hear AI what it actually is relating to. For example the algorithms we are exposed to every day on social media are a form of AI, as it learns what you like and tries to give you more of this - for better or worse.

The next lightbulb moment for me comes from the statement:

Now that I understood that AI components are probabilistic and not hand-written, I also understood that
the problem is not testing of it, but fixing of it.

Maaret goes on to say that we have to change our mindset in that a “fix” for a bug does not exist the same way we might have been used to. We can only really use data and variables to see how it changes the outcome.

[…] the control we had would never again be the same

The other element the article touches on is helpful tooling such as Copilot in GitHub. But she highlights that you need clear intent when using such a tool that generates code for you to spot bugs and be able to use it correctly in your context.
Furthermore for using Copilot generated code in production, Maaret mentions ethical considerations which is something that I had not heard in that context yet.

I believe erasing attribution for the open source programmers may not be direct violation of copyright, but it is ethically shifting power balance in ways I don’t support

Building on her commentary about fixing bugs, she then provides her experience using ChatGPT.
Here we dive more into how AI is changing software testing.

I realized the big change was that testers would need to skill up in their thinking. Noticing it has gender bias is too low a bar. Knowing how you would fix gender bias in data used to teaching would now be required

I like her comments on how these tools could be used for “stupid testing”. For example being used to create a wealth of test cases that no-one actually needs.

There is some danger that these tools can send testing as a profession back some years, by creating a mountain of paperwork that is not useful or valuable.

Maaret ends with that AI is changing software testing like any new technology does, and that it is just another way to talk about automating things but at the same time trying to do it while acting in a human way.

  • natural language processing to communicate successfully in a human language.
  • knowledge representation to store what it knows or hears.
  • automated reasoning to answer questions and to draw new conclusions [like a human]
  • machine learning to adapt to new circumstances and to detect and extrapolate patterns [like the human experience]
  • computer vision and speech recognition to perceive the world.
  • robotics to manipulate objects and move about [like a human]
13 Likes
52

Insightful! It’s fascinating how quickly “everyone” has embraced ChatGPT and how it has become an ally in everything we do.

4 Likes
53

Summary: a good informative read on the benefits of AI that Browserstack provide. Very high level introduction which was just what i needed as a novice to AI

I definitely see potential use in implementing to automated testing, so I will use it to delve more deeply. Challenges are implementing to a company in very early stage of building a test team alomg with lots of orher priorities.
[/quote]

2 Likes
54

hi All,
here is the article I’ve found interesting
How To Learn AI From Scratch [2024 Guide]
It gives an overview of the subject as well as fundamental start points.
I would like to learn more how to use it in my mundane tasks and how I can use it to quickly generate test data.

2 Likes
55

I found a case study by an IT management company describing their experience of introducing AI into their processes. Turns out this is actually an ebook but my favourite chapter was this one on incident management. So not testing directly but definitely an area that quality-obsessed people often find themselves drawn into!

Much like automation, the idea is that AI is complementary to humans. It’s not here to do us out of a job but to make us more efficient. I really liked how this company used AI not to do the job of humans, but to help make humans more efficient. A theme I picked up was around using AI for data analysis which requires more intelligence than automation. As automation gets quicker and covers more, we generate more data than we can analyse. AI for this company helped to gather all the relevant data for incidents to help technicians deal with the problem. It could also capture data to point to the root cause by gathering all the events that lead to the incident.

4 Likes
56

I chose the video AI for Software Testing by Jason Arbon. IMO, Jason is one of the leading voices in AI testing, alongside his friend/colleague Tariq King. They’re both very passionate, confident and not afraid to ruffle feathers.

This video is particularly interesting as it is 5 years old and we all know how fast AI moves! I know a fair bit about Jason but hadn’t seen this video before and felt it would be useful to look back on it now to see how things have changed.
What is really interesting is Jason delves into how Machine Learning works, how you teach an AI to test and explains how it can be used to do all types of testing, including exploratory.
So far, I’ve yet to see the things promised by Jason to be available in the market even though he says in the video (2018) that they’re out there and being used. Still, I think this is a great video to demonstrate to anyone the possibilities of using AI in testing.

AI for Software Testing - AIxchange - Jason Arbon

5 Likes
57

The point on hallucinating is so important to remember. I was chatting to Tariq at a conference once and he put it like this “Remember there is no intelligence there (yet). The LLM will just supply a string of text that it thinks will be the most appropriate response to the string it has been sent”. When you remember that there’s no comprehension in the LLM in makes interacting with them safer and more productive.

5 Likes
58

The article that I read shows how AI testing enhaces Software testing efficiencies.
It explain some approches like, data sets generation, test cases generation, codeless development, etc.
It was very good to read this to have like a general view of what can be done and with approaches are available to use.
in my context it will be very helpful in the test cases creation, data set generation, and in the future some self healing code/ testing woule be important and added to my daily work

2 Likes
59

The article I liked was Top trends: 5 ways AI will change software testing | TechBeacon
This article listed 5 ways in which AI will change software testing. One of the ways I like is that our tools will change. It makes me think to keep growing and moving forward with change and learning.

2 Likes
60

Dear reader
There are 57 responses already on Day2.
While I am not as good as AI and scroll about 300 posts on day 1(which I did’nt),but I scrolled all 57 posts today. Among the responses, I definitely want to re visit a few ‘summary’ posts and read their the original article too, but overall I saw there are quite a few repetitions (and even claims that I think are not true yet, or maybe I need to catch up with fast gaining capabilities of AI.)

While looking for articles I saw almost every article is preaching AI’s capabilities / its basic structure and what we can expect to do with it. I like more of a hands-on approach. So, today I read an article which is about someone’s experiences in testing/working with AI.

Summary - Oh, nice. Before I could add summary, MoT has identified the article’s link and I see @punkmik already summarised it pretty well (Wow, I am glad that you did it already, as its 1AM here). I will make it a one line summary then -
“Take AI with a pinch of Salt. Remember, it’s not intelligent (yet !?).”

Thanks for reading!

8 Likes
61

I was reading other peoples posts and I stumbled upon the great talk from James Bach about “The future of the Testing Role”.

I liked the way he presented and explained things so I googled him and stumbled again on his review of chatGPT in testing which is related to to the task of the day. It is very interesting and for me it really tells how much we should take all this with the grain of salt and be really critic about it. It can be helpful but you need to understand what you are doing and always think twice before taking its advice. (I can even rhyme :sunglasses:)

I am curious what do you think about it :smiley_cat:

1 Like