đŸ€– Day 5: Identify a case study on AI in testing and share your findings

30 Days of Testing
, ,
42

Resource: https://blogs.infosupport.com/machine-learning-user-testing-case-study/

The article talks about making a smart system for a bank in The Netherlands using AI and machine learning. It helps bank employees create and share procedures easily. The system uses tags (keywords) a lot to work well. They made a tag recommendation model to make tagging faster and better.

At first, everything seemed great with high accuracy, but when they tested it with users, it didn’t go well. Even though more tags were there and they were better, users took more time to finish tagging. The issue was showing too many tags at once, making users slow down.

They fixed it by separating selected and suggested tags, showing fewer suggestions, and finding the right number of tags to display. After a few tries, they got it right. Now, users tag things faster and it’s easier.

The main point is, that even if a model works well technically, it’s essential to test it with users to make sure it fits their needs and makes things easier for them.

5 Likes
43

Ironically, ChatGPT itself seems aware of its own limitations regarding the use of AI in testing.
Anyone considering using AI to help with testing should also be aware of these limitations, as i think there is a bit too much cheer-leading and hype around AI in the tech industry generally.

Q: “Please outline ways in which AI tools are not helpful with testing software applications.”

Answer:

  1. Cannot Mimic Human Intuition: AI tools cannot capture the nuances of human intuition, which can be critical in testing.

  2. Lack of Context Understanding: AI tools may struggle to understand context, especially for domain-specific applications.

  3. Difficulty Identifying Visual Bugs: AI might not notice bugs related to a software’s visual layout or design.

  4. Struggle with Exploratory Testing: AI tools might find it hard to perform exploratory tests that don’t adhere to predefined cases.

  5. Need for Continuous Training and Upgrading: AI testing tools need to be continuously trained and upgraded to keep up with changes in software applications.

  6. May Overlook Complex Bugs: AI tools might overlook complex bugs that involve multiple elements or functions interacting in unanticipated ways.

  7. Limited Creativity: AI lacks the creativity to come up with tests that might break the software, something that a human tester may be good at.

  8. High Setup Costs and Maintenance: Setting up AI for test automation demands considerable upfront costs and maintenance.

4 Likes
44

In one of my machine learning projects, I used ChatGPT 4 to create test patients for end-to-end verification and validation instead of manually using public MDS risk calculators. I employed Generated Knowledge Prompting and Few-Shot Prompting techniques.

While ChatGPT significantly accelerated the generation of test patients with all desired test cases, the data required human review due to some inaccuracies.

Here are the specific issues encountered:

Generated Knowledge Prompting: ChatGPT’s knowledge of IPSS-M and R scores was limited, with a stronger focus on bone marrow data. This resulted in inconsistent scoring. For instance, an IPSS-M score of -0.38 was incorrectly generated as 1.

Few-Shot Prompting: To avoid replicating the original dataset exactly, we prompted ChatGPT to generate randomized data. However, the risk level data was systematically off by one step. For example, a “Low” risk patient was assigned a “Moderately Low” risk, and a “High” risk patient became “Very High.”

3 Likes
45

Hi,
Blog Post used: How AI Is Transforming the Future of Software Testing
Facebook has a tool called Sapienz that is helping their large engineering teams squash bugs quicker by automatically creating tests for their mobile applications.

  • AI tool named Sapienz was used to created automated test cases for mobile apps.
  • It is an andriod system automation testing tool, for test case creation, execution, loop and learning
  • Facebook is a widely used application, using this tool definitely helped them to provide better test coverage and quality
  • Adaptive testing feature of Sapienz is what I found interesting
  • It helps me to understand how large scale organisations are adopting AI to achieve better quality
2 Likes
46

Hi @sarahk,
here is a kinda feedback for you :wink:

firstly, unfortunately I have not received this task yesterday on my email igolkina.nata@gmail.com.
Am I the only one with such confusing issue?

and secondly, this task is very disputable by having only two options 
 I’ve joined the course as a unexperienced tester, so I do not know how to perform this task 
 used chatgpt but not in testing and very eager to know how I can implement it in my testing tasks

2 Likes
47

I found it really hard to find any warts-and-all case studies. There were plenty of advertising versions, but nothing really useful.

I came across this one that talked about using AI to generate test cases from requirements, which sounds helpful. We don’t have a lot of in-depth requirement documentation for much of what we do, so the usefulness could be limited for us, but I figure there’s no harm in trying with the limited documentation we do have.

I really need more though
 hoping reading other’s contributions will help!

2 Likes
48

Now you mention it, I also have not been receiving all the emails. Got the ones for Day 1, 2 & 4
 none since then. Weird.

I would suggest you catch up by skim-reading through the previous days’ chats, or just dive into looking today’s stuff.

Good luck!

49

One of the problems I find with this project so far is finding where AI will help me as I work with legacy windows desktop systems written in code as old as VB6 and Delphi, with some .Net Core 3.1 thrown in too, using old Windows Forms.

I am going to take a leaf out if @adrianjr book and talk more of experience than case studies, as I find they do not point to what i do,.

So perhaps this is more to do with what I have experienced in previous jobs and how I have evolved the code I work with to get a place where I can harness some of these new tools and concepts.

My first experience with an AI tool was some 8 years ago when I first saw Aplitools UI (it may have been Eyes) and it was definitely a ‘wow’ moment.
I had worked with CodedUI to drive some screens(not test!!), but this was something else. That a tool would take a template of a screen and tell me any changes, and ask if i wanted to replace the template with these changes, I though was exceptional.
As a cynic who thinks automation has no place in UI testing as it is a dark art, I still advised the company I worked for to adopt the tool and save time on UI to focus on Data.

Fast forward 6 years and my first encounter with Co-Pilot in Visual Studio was another ‘wow’ moment, albeit with a few ‘why are you prompting thats’ thrown in.
But I find this a fantastic ‘Tool’, it is still a tool. It is myself and colleagues who engineer the code but I love that Co-pilot takes all the mundane tasks and creating boiler plate code off my hands and lets me focus on the architecture and design of my test frameworks.

My next steps are to look at the database tools out there, and more into coding tools.
I have a feeling that AI will see the need to refresh what we just learned and thought we knew about these tools more often, as these will evolve at a rate of noughts.

2 Likes
50

@nigolkina and @teresa.weatherall.

Thanks for sharing, and we’re sorry you didn’t receive the email.

We’ve decided to set March 5th as International AI Glitch Day. Turns out we (or was it the machines :wink: ) set the email to send in 2026 instead of 2024.

Sorry for any confusion. The email should now be with everyone who is taking part.

Thanks for your understanding and for getting involved in 30 Days of AI in Testing. It’s very cool to see how much incredible collective learning is happening right now!

2 Likes
51

Thanks Hiral for the inputs

52

This is a case study from a few years back, but it shines as a fascinating example of how artificial intelligence can be innovatively applied in testing.

King, renowned for games like Candy Crush, faced the complex challenge of testing hundreds of levels, each with multiple solutions. To address this issue, they began employing artificial intelligence. Initially, they explored search trees and then advanced to training bots using reinforcement learning techniques and genetic algorithms.

One of the difficulties encountered was establishing effective criteria for the bots to test, such as determining if a level is feasible to complete and if its difficulty is balanced. This approach demanded deep expertise in artificial intelligence and machine learning, with a dedicated team of data scientists at King working diligently on this project.

Recently, there has been talk of incorporating data scientists into testing teams to drive the development of bot testing and automate routine tasks for testers. This could include automating activities such as defect identification and test prioritization, thereby freeing up testers to focus on more strategic and high-value tasks, such as exploratory testing.

2 Likes
53

I looked at Github Co-pilot for writing tests as it is used by some of my colleagues. I was watching a demo (and read the article) on using it.

My colleagues have often talked about it being useful to quickly create tests and through the article and demo I can definitely see that being true and quite powerful. When speaking to devs using it, I felt the weakness was that you’d need to check the tests as of course it will guess the logic. Overall though, it was definitely a time saver and made it easier to write tests and avoid that trap of “oh I don’t have time to write loads of unit tests”. It is an even quicker task.

However the demo raised an interesting thought. It is writing tests about the logic of the code, not the business logic or the behaviour. We’ve had some interesting conversations lately on how unit tests are best when they focus on testing the behaviour rather than the code, yet for people using this tool, are they going to be leaning more on testing the code not the behaviour.

2 Likes
54

Day 5
Over the past few days, we’ve built foundational knowledge about AI in testing. Today, we’ll take a look at how our discoveries play out in real-world settings by exploring case studies.

Task Steps

  1. Search for a real-world example of where AI has been used to tackle testing challenges. This could be a published case study or an example shared in an article or blog post.

https://static1.squarespace.com/static/61c565d1f94e5e0edb1471f3/t/61e861ff0ad2f853901e37a3/1642619392158/testai_CaseStudy_Software_v1.2.pdf

  1. Select and analyze a case study that seems relevant or interesting to you. Make a note of the company and context, how AI was applied in their testing process, the specific AI tools or techniques used and the impact on testing outcomes/efficiency.
  • The company had some issues as the requirements and new features are happening really fast so they hire an AI system to help in automation process
  • The AI helped them by updating and labeling the controls for the automation to interact with it.
  • The tool is test.ai.
  1. Share your discoveries by replying to this post. Here are some prompts to guide your post:
    o Brief background on the case study

There is a company that had a lot of changes in the software developing process so there is a lot of changes in the scripts and application so they cannot create a reliable testing regression set to reduce the manual work.

o How was AI used in their testing?

The AI helped the automation scripts by rescanning the webpages and update the controls identification id so the automation can keep running.

o What tool(s) or techniques did they leverage?

The case of study does not mention this.

o What results did they achieve?

They can reduce the manual effort and give this tasks to junior QA so senior QA can be involved in more critical tasks.

o What stood out or surprised you about this example?

The investment of the client in the AI, must of clients doubts a lot in AI investing.
o How does it relate to your own context or AI aspirations?

I guess we can use the AI to improve our delivery and meet the client success expectation.

55

Enhanced Test Environment Simulation:

  • AI enables realistic simulation of diverse user interactions and complex environments. By leveraging machine learning models, testing environments dynamically adjust to mimic real-world scenarios. This approach offers a more accurate representation of end-user interactions with the software.

Performance Testing Accuracy:

  • AI helps simulate real-world scenarios and load patterns, making performance testing more accurate and efficient. AI tools can identify vulnerabilities, analyze code for security flaws, and assist in penetration testing. Additionally, AI can determine which test cases need to be run based on code changes, reducing the effort required for regression testing.
    Test Automation:

  • AI plays a crucial role in test automation. It can generate test cases, optimize test suites, and even perform self-healing by identifying and fixing issues automatically. This streamlines the testing process and enhances overall efficiency.

1 Like
56

Case Study: AI in Software Testing at Tricentis

Company and Context:

  • Company: Tricentis, a leading provider of software testing solutions.
  • Context: The challenge was to enhance testing efficiency, coverage, and accuracy while dealing with complex software applications.

How AI Was Applied in Their Testing Process:

  1. Smart Test Selection:
  • AI Technique: Tricentis leveraged AI to determine necessary tests and eliminate redundant ones.
  • Impact: Reduced noise in test suites, ensuring that only relevant and critical tests were executed.
  1. Machine Learning (ML):
  • AI Technique: Tricentis used ML for:
    • Test Case Generation: Automatically generated test cases based on codebase, user requirements, and inputs.
    • Auditing Current Tests: Evaluated test coverage, performance, and completeness.
    • Test Execution: Automated test execution.
  • Impact: Accelerated test creation, improved coverage, and reduced manual effort.
  1. Computer Vision Bots:
  • AI Technique: Labeled elements (e.g., shopping cart, login box) in the code.
  • Impact: Computer vision bots detected changes in labeled elements, maintaining tests even when the code evolved.

Impact on Testing Outcomes/Efficiency:

  1. Faster Test Execution:
  • AI reduced test execution time by:
    • Removing duplicate tests.
    • Improving test data management flow.
  • Result: Faster feedback and quicker release cycles.
  1. Better Test Maintenance:
  • AI adapted tests to code changes automatically.
  • Result: Reduced effort and cost associated with test maintenance.
  1. Enhanced Coverage and Accuracy:
  • AI-driven test case generation and ML-based auditing improved overall test quality.
  • Result: Comprehensive coverage and reliable testing outcomes.

In summary, Tricentis successfully harnessed AI to optimize their testing process, resulting in more efficient, accurate, and adaptive software testing practices. :rocket::magnifying_glass_tilted_left:

1 Like
57

This reminds me of @lisacrispin and @janet_gregory’s Holistic Testing Model.

“Building in quality using the left side of the Holistic Testing Model” and “Checking what you built using the right side of the Holistic Testing Model”.

Makes me wonder if there’s an option to consider the “testing the code” or “testing the behaviour” with this model in mind. :thinking:

58

From a holistic testing perspective, I want to take advantage of the AI/LLM tools as we practice test-driven development and behavior-driven development. The tools help us think outside the box and inspire us to ask more questions and come up with the less-obvious scenarios. Since we automate tests at the same time as we are writing the code, I’m not sure how much need there is to analyze existing tests for things like duplicates. I suppose over time, though, that could happen. If AI tools can help us make sure we have the test coverage we want, that is quite useful!

The developer I work with uses an IDE coding assistant as he writes tests and code, and he is happy with the results. I think these are quickly becoming an ingrained part of many peoples’ software develoopment process.

2 Likes
59

Hey Testers,

Doing some catch up here but read this case study here:

Generative AI in Software Testing

(https://assets-global.website-files.com/5eb9845c1d37d7efad64e226/659702a0c73f95b685fb7d2d_Generative-AI-trends-2024.pdf)

Here are my thoughts:

Generative AI, powered by machine learning, crafts new data or content that resembles what humans create, which proves especially valuable in software testing. When it comes to testing software, generative AI takes charge of generating test cases, datasets, and even code, making the testing process faster and more precise.Unlike traditional AI models, generative AI is a self-starter, producing original outputs based on what it learns from data. This innovation is shaking up various industries by offering solutions to some complex problems.

Generative AI’s is good at understanding and using language boosts chatbots’ conversational skills, making customer service and communication smoother across sectors like healthcare, finance, and retail. In healthcare, generative AI is a big help in examining medical images, finding new drugs, and personalizing treatments, all of which make software testing crucial for patient safety. In finance, it can be used for generating realistic market data and helping with risk analysis, fraud detection, and decision-making, ensuring that financial software systems stay on point in ever-changing markets. Generative AI also works wonders in product design and manufacturing, dreaming up new designs, running production scenarios, and spotting potential issues early, making the manufacturing process faster and more efficient.

For us as software testers, it speeds up problem spotting, crafts diverse test cases, and automates testing, all leading to better software quality and reliability.

Generative AI in software testing is a game-changer, taking out the manual work, adapting to changes with minimal human input, and giving broader coverage, all of which overhaul software quality assurance. Exciting trends include the rise of autonomous test generation and predictive testing, both leveraging generative AI to make testing quicker and to spot bugs faster.

Things to look forward to when its comes to using generative AI:

  • Augmented Intelligent Testing: Generative AI will assist in decision-making, test case design, and result analysis, enhancing testing efficiency by allowing human testers to focus on complex tasks while AI handles routine ones.
  • Enhanced Predictive Testing: AI models will advance to predict potential issues before they occur, enabling proactive testing and improving software quality.
  • Customization for Industry-Specific Testing: Generative AI will be fine-tuned for specific industries like healthcare and finance, addressing their unique testing requirements.
  • Increased Use of Cloud-Based Generative AI: Cloud-based solutions will see a surge in adoption due to scalability, cost-effectiveness, and accessibility, enabling high-quality results with minimal human input.
  • Collaboration Between Testers & Developers: Automated test case generation fosters better collaboration, with developers fixing bugs quickly and testers focusing on essential tasks.
  • Enhanced Reporting and Real-Time Analytics: Generative AI will automate the creation of comprehensive reports with predictive analytics, empowering teams to make informed decisions quickly.
  • Autonomous Model Generation: AutoML and generative AI will combine to automatically generate and optimize test cases, ensuring thorough testing without overlooking critical paths.
  • Streamlined & Highly Accurate Defect Tracking: AI will identify and categorize defects, leading to faster resolution and improved software quality.
  • Improved Root Cause Analysis: Generative AI will accelerate root cause analysis by quickly identifying failure patterns and potential causes, facilitating faster bug resolution.
  • Generic Model Data vs. Agent Data Sets: A combination of broad, generic datasets and tailored agent datasets will be used for comprehensive and accurate testing.
  • Agent Data Sets as a Differentiator: Agent datasets, tailored to specific use cases, will enhance the accuracy and relevance of generative AI models, ensuring better performance in niche areas like healthcare and finance.
1 Like
60

Thank you for the information on PostBot! The YouTube video you included was super easy to follow. I imported a simple swagger json from https://petstore.swagger.io/ to work with. Then I was able to demo PostBot’s awesomeness to my team, and they LOVED it! I am almost to my limit on the BETA version already and I’m not sure how long it will be available for free . I am grateful to have been able to check it out. Cheers!

1 Like
61

Hi all,

Below are my findings for the day 5 challenge, see this link to mabl for more info an AI backed testing tools - https://www.mabl.com/blog/how-real-world-development-teams-are-using-ai-to-improve-software-testing

  • NetForum Cloud are leveraging AI-backed autohealing capabilities to mature their software testing for digital transformation. An increase in pipeline automation and ui upgrades demanded more efficient software testing even as experienced Selenium automators had extensive experience.
  • With autohealing tests, their team was able to increase automated testing by 40% and reduce manual testing by 20%. Less test maintenance had a powerful domino effect for the entire Netforum Cloud organization, leading to better development practices and less downtime for customers.

Best Regards,

Bill