🤖 Day 5: Identify a case study on AI in testing and share your findings

[kato (Kat Obring)]

I did a bit of Gemini assisted searching of the interwebs to find a relevant study, and as many before me here have noted, there’s little serious material available.

So I decided to do a bit of studying myself, and started a free class over on Codecademy (which, interestingly, they call a study): Debug Python Code with Generative AI Case Study

Unfortunately I didn’t get very far, as chatGPT soon told me I had exceeded the allowed number of requests …
But I got far enough to taste blood, so to speak, and since I already got the PythonGPT will continue to play with the possibilities of assisted programming.

The one thing that’s getting more and more clear: context is everything, and training data is a vastly underrated commodity.

[gwaterhouse85 (Gareth Waterhouse)]

New GitHub Copilot Research Finds 'Downward Pressure on Code Quality' --...

This was an interesting read, talkin gabout how the usage of copilot has lead to a decrease in the maintainability of code, and the lack of adherence to the DRY principle (Don’t Repeat Yourself).

It’s important to remember co-pilot probably isn’t the best tool at writing you something from scratch, but providing the right framework and context for it to work in, and the right structure, it can be extremely useful. Give it an example test framework and then ask for some more tests utilising that framework and it will be useful and powerful.

I’ve used it recently writing new code in typescript, and asking it to tidy up some code for me which was extremely useful as an example.

[gruja (Grujica Stanisic)]

I will go with real case.
Post Bot is really cool tool.
Very easy to use.
Code is pretty clean a so far i was receiving valid answers.
We can test each point separately or we can import all end points from swagger.
In other case we will have to make changes and to adopt data to our test data.
But it is definitely speed up my work.
Like with other AI tools, we must provide simple and clear task ( what to test and to verify).

[qagenai (Swapna QA)]

@sarahk `

In my personal experience using the using AI tools or techniques in your testing ,is

1. Leverage power of Generative AI in document processing for enhanced accuracy,
2. built-in data querying,
3. support for multiple languages and many such capabilities

• Brief background on the case study or personal experience

We are working on core doamins with multi level applications in Low Code No Code platform

• How was AI used in their/your testing?

We are using fori ntelligent Documen,t Processing and Unstructured Document purpose and which has wonderful results on the documentation process

• What tool(s) or techniques did they/you leverage?

1. Our AI-enabled data capture solution makes document processing a breeze, minimizing the need for manual data entry, exception management, and quality checks.

2. It works with unstructured or semi-structured documents to capture data with high accuracy and higher percentage of straight-through processing (STP).

3. The tool takes document processing to new heights by integrating [Generative AI]intelligent-automation/solutions/idp-and-generative-ai), for enhanced accuracy, automation, multilingual capabilities, and a host of other benefits.

• What results did they/you achieve?

1. Businesses get up and running faster thanks to our Template-Free approach.
2. Loaded with pre-processing and post-processing features, along with browser-based access and an intuitive GUI,
3. This tool along with IDP delivers a compelling user experience (UX) to knowledge workers.

• What stood out or surprised you about this example?

1. Experience template-free intelligent document processing
2. Speed up time to market significantly
3. Enjoy scalability and deployment flexibility
4. Experience high degree of accuracy

• How does it relate to your own context or AI aspirations?
  It was implemented in one of our domain apps in Health care , banking which was surprisgly result driver
  So AI-powered data capture is a result driven combination in my experience

[ash_winter]

Day 5

Option 1: Case Study Analysis

Search for a real-world example of where AI has been used to tackle testing challenges. This could be a published case study or an example shared in an article or blog post.

This one seemed interesting to me, as managing lower level tests and their quality is quite a challenge:

AI-Driven Java Unit Testing Boosts Developer Productivity for Financial Firm...

Select and analyse a case study that seems relevant or interesting to you. Make a note of the company and context, how AI was applied in their testing process, the specific AI tools or techniques used and the impact on testing outcomes/efficiency.

• This was a large financial services company, with a set of legacy services some of which had little code coverage and were seemingly buggy. Devs had to do a lot of rework on these services and spent a lot of time on creating unit tests for legacy code.
• In this context, the AI tooling was an IDE plugin to guide users through the process of creating unit tests and then by targeting tests at change hotspots.
• Apparently big gains in developer productivity through less time on unit testing and more on ‘innovation’, plus better delivery through reduced defects and rework required.

As this is a company selling their tool, its obviously biased and only gives you the positive side of the case study. The ideas are interesting though, as a lot of time is spent adding tests to legacy code (or not and watching it break) and running tests in areas that haven’t changed. These to me seem like great uses of generative AI, depending on how well it is done of course!

[flyagaric (Mike Stewart)]

Here’s the case study I found: Unleashing the Power of Generative AI in Software Testing - Case Study (Dynamics 365 Upgrade)

To be honest, the article lacked enough detail to be useful. It read more like an advert for a particular test management add-on for Jira. Nevertheless its key point was that AI technology “succeeded in identifying the critical Test Scenario that was missed by both the software development and experienced business testing teams.” It also claimed that the tool took 15 minutes compared to 3 days for a team of engineers.

I’m treating these claims with a healthy level of cynicism. From yesterday’s AMA video I learnt that effective AI tools need to be trained with the appropriate context but I doubt the 15 minute claim included this training. If they used the tool “off-the-shelf”, how was it clever enough to think of scenarios that trained humans couldn’t?

[hiroksarker]

I have chosen to analyze a case study from the article “Testing AI Applications: Best Practices and a Case Study” by MobiDev - Testing AI Applications: Best Practices and a Case Study

In this case study, a customer service AI application underwent testing to ensure it could deliver personalized responses based on individual interactions. AI played a pivotal role in the testing phase, employing machine learning algorithms trained on vast datasets of customer interactions. The results were promising, with the AI application successfully providing accurate and relevant responses, ultimately enhancing customer satisfaction. What’s notable is the profound impact AI has on reshaping customer service dynamics, offering tailored solutions in real-time. This example aligns with my professional interests in AI and machine learning, showcasing their potential to revolutionize customer experiences through advanced technology.

[pavlos (Pavlos Zotos)]

Hello all!

I found an interesting test case of a leading Telecommunication company and its problem to implement automated test suites. Their problem had to do with the speed that qa engineers created the test, but at most the difficulty faced to maintain them.

To solve their problem, they started to use testRigor, a Generative AI-based automation tool, which has the ability to create automated test cases written in plain language.

By using testRigor, they saw great improvement on test creation speed and ease of maintainance.

You can find this case study in the following link:
testRigor case study

[melissafisher]

The case study that stood out to me a year or more ago was by @theqaconnection My Experience Using ChatGPT to Perform Testing Activities – The QA Connection Blog
In this blog post Johanna talks through an example of using chat GPT to generate test scenarios, writing automation test scripts, writing defects and more. Johanna shares a lot of thoughts after each ‘experiment’. It makes me think about how we can generate our own experiments and simply have a go!

[indres (Indre Stasiuniene)]

Hi, everyone

my first idea was to present a real example of the use of AI in testing from my country, but surprisingly I didn’t find anything . In general, there are not much such information from other countries also.

I find a blog about the presentation of Estonian software testing company Teslio founder Kristel Kruustuk at the Agile Testing Days in German at the end of 2023 years. From her own experiens, company use generative AI tools to improve test creation, test refactoring, and issue accuracy.

Testlio’s team conducted two experiments using GenAI, that demonstrated a 40%+ reduction in errors and a 15%-30% increase in test refactoring speed. Testlio has also successfully implemented over 20 iterations of GenAI improvements since the original experiments.

Testlio experiments show that generative AI can make written instructions and information more understandable, leading to more efficient testing and better quality assurance results. Generative AI can reduce time spent writing, reviewing, and editing core instruction content.

What kind of challanges faced company:

importance of working in a secure technical environment to maintain data integrity;
dealing with unpredictable outcomes, response times, formatting puzzles, and prompt engineering;
there still need human intervention and control.

Link: Testlio Founder Inspires Audience with Keynote on the Use of AI in Software Testing - Testlio

[maaret (Maaret Pyhäjärvi)]

Since I already shared experiences on an earlier day, I decided to use today to create a new experience. That’s a case study, right?

I went to search for code in python as test target, finding specifically this:

Pretty little rabbit, right? Reminds me of reading some code at work, just less intentional. And really do not have the time or energy to read that.

I could test it as black box, learning that given a parameter of a number, it gives me a number.:

If I didn’t know what the rabbit implements or recognize the pattern in the output, I was thinking of just asking ChatGPT about it. However, I did not get that far.

Instead, I wrote def function(): on my IDE while GitHub copilot is on, thinking I would have wrapped the program into a function. It reformatted it to something a bit more readable.

Prompting some more in the context of code.
Comment line “#This function” proposes “is obfuscated”. Duh.
Comment line “#This function imp” proposes "lements the Fibonacci sequence using Binet’s formula.

At this point, I ask chatGPT how to test a function that implements the Fibonacci sequence using Binet’s formula. I get long text saying try values I already tried, but in code, and a hint to consider edge cases and performance. I try a few formats to ask for a value that would make a good performance benchmark, and lose patience.

I google for performance benchmark to learn that this Binet’s formula is much faster than the recursive algorithm, and find performance benchmarks comparing the two.

I conclude inserting the bunny code into chatGPT and asking “what algorithm does this use” to get second language model generate likely answer as the Binet’s formula. Given the risk and importance of this testing at this time, I conclude it’s time to close my case.

There are so many uses to figure out what it is I am testing (while being aware of what I can share with tool vendors) and a simulation of you could ask about the pull request was the case I wanted to add to the world today.

Edit: Put this in my blog, cleaning up a few weird sentences. A Seasoned Tester's Crystal Ball: A Bunnycode Case Study for AI in Testing

[neil_thompson]

My hasty Bing search found the same case study as @ash_winter has already reported. So…
As that was about unit testing, I thought: let’s ask specifically about integration testing (just a logical next step, not the end of the story, obviously). Unfortunately, it started “Certainly”. RED FLAG IMMEDIATELY.
Then it went, “Let’s delve”. OH, SNORT, CHAT-ITIS ONCOMING.
Then it said “Software testing is a crucial phase in the software development life cycle”.

I AM SORRY BUT I DO NOT HAVE ENOUGH PATIENCE TO CONTINUE WITH THIS NOW,

[christinepinto]

Thank you for sharing your insightful report. I must admit, the results on Ranorex and Katalon Studio were quite surprising to me, underscoring that AI, indeed, is still in its infancy within the testing domain. It’s fascinating to observe these early stages and envision how it will evolve. On a brighter note, I’m thrilled to hear about your positive experience with Applitools Eyes —it’s a tool I’ve used myself and found to be increasingly effective. It’s great to hear it performed admirably in your tests. Out of curiosity, which comparison level did you opt for in your test case? Your choice undoubtedly played a crucial role in the tool’s performance.

[sabine.k (Sabine Kolics)]

This blog post of testrigor tool illustrates how this could be applied healthcare domain. The tool focus on supporting test automation in the testing process for functional, end-to end and user acceptance testing. AI technologies like NLP, generative AI and ML are used to make automated testing as close as possible testing performed by a human tester. Efforts for auto test maintenance is reduced, self-healing capabilities. You can write test cases in plain English which is transformed into a test code, or you use the record tool.

Web applications can be tested across desktops, mobile devices, and native and hybrid applications. It supports testing across multiple operating systems, configurations, and device combinations. It supports with device farms and infrastructure frameworks like BrowserStack and LambdaTest. Test data or data import can be uniquely generated based on input type or format using RegEx. The tool complies with security standards like HIPA and SOC2.

Looking at the specific Case study https://testrigor.com/assets/pdf/medrio_case_study.pdf of company medrio. medrio is a leading eClinical company supporting clinical researchers and trial participants. In this case study it described they have UI intensive test cases. By using AI in testing, testrigor, they achieved 4x release a year, from 1-2 releases to 9 releases, saved time of 4 man-years per year and are able to automate screen capturing for compliance. Time saving was huge. They speeded up manual test process by automating. Auto tests they had available, did not scale. They found a solution where QA team can automate test cases in order to frequently release.

I think this a very interesting example, case study, showing that with a codeless automation tool enriched with AI capabilities can make the difference in an existing testing tool landscape.

[adrianjr (Adrian)]

Thank you for the kind words. I have just checked and for my tests within Applitools I used the strict eyes match level.

[crmueller (Claudia Mueller)]

I had ChatGPT come up with a Case Study - and I learned about RPA from it: Robotic process automation - Wikipedia

quote
Case Study: Robotics Process Automation in Banking
Background: Banks and financial institutions handle a multitude of repetitive and rule-based tasks, such as data entry, account reconciliation, and customer onboarding. These tasks are not only time-consuming but also prone to human errors, making them ideal candidates for automation.
Challenge: The challenge is to streamline banking operations, improve efficiency, reduce costs, and enhance accuracy by automating routine tasks while ensuring compliance with regulatory requirements and maintaining data security.
Solution: Robotic Process Automation (RPA) offers a solution to this challenge by automating repetitive tasks using software robots or “bots.” These bots mimic human actions by interacting with software applications, entering data, performing calculations, and making decisions based on predefined rules and logic.
Results: Several banks and financial institutions have successfully implemented RPA to automate a wide range of processes, including account opening, loan processing, fraud detection, and compliance reporting. By deploying RPA, these organizations have achieved significant improvements in operational efficiency, cost savings, and error reduction. For example, RPA can process large volumes of transactions accurately and quickly, enabling faster turnaround times and improved customer satisfaction.
Impact: The adoption of RPA in banking has transformed traditional banking operations, enabling organizations to reallocate human resources to more value-added tasks, such as customer service, risk management, and strategic decision-making. RPA also helps banks adapt to changing regulatory requirements more efficiently by ensuring compliance through automated audit trails and reporting.
Conclusion: The case study of RPA in banking illustrates the transformative impact of automation technologies on streamlining operations, reducing costs, and improving service quality in the financial sector. By leveraging RPA to automate routine tasks, banks can enhance their competitiveness, agility, and ability to innovate in an increasingly digital and competitive market landscape.
unquote

[blanchec (Blanche Carstens)]

Case study: Spotify

Background: Spotify, the popular music streaming service, turned to AI to make our music experiences better and their business smoother.

AI in Testing: Spotify used AI to figure out what music we like based on what we listen to. So, every time we play a song, AI kicks in to suggest more tunes we might dig.

Tools/Techniques: They used fancy machine learning to analyze how we listen to music and natural language processing to understand what we’re into.

Results: Thanks to AI, Spotify nailed the whole personalized recommendation thing, making our playlists even better. Plus, they made bank by using AI to serve up ads that actually interest us.

Standout Aspect: What’s cool is how AI seamlessly fits into Spotify, making our music choices feel tailor-made while also making Spotify’s business run smoother.

Relation to My Context/AI Aspirations: This shows how AI can make testing smarter and more effective. In my projects, I can see AI helping us understand user needs better and improving our products based on that. Plus, using AI for personalized experiences aligns with my goal of making our products more user-friendly and successful.

[markcole]

I test K-12 school administrator software, and I while back I had a conversation with chatgpt about it. The first question I asked was:

What is the best way to automatically test K-12 school administrator software?

It gave a fairly good answer with 13 action points. Its first action point was:

Identify Test Scenarios and Use Cases:

• Work closely with stakeholders to identify key scenarios and use cases that the software must support. This includes user authentication, student enrollment, grade management, communication features, and reporting functionalities.

So I decided to ask it a question about creating user cases for a specific part of our software:

prompt:
For Student Demographics for a High School what different user cases do I need to test the Student Demographics functionality?

It actually gave me a pretty good answer - 16 user cases with the first one being student enrollment:

Student Enrollment:

• Verify that the system allows for the successful enrollment of new students.
• Ensure that required demographic information (e.g., name, address, birthdate, gender) is captured during the enrollment process.

It really did give me some very good things to check such as gender identification:

Gender Identification:

• Confirm that the system supports various gender options and that the information is stored accurately.
• Test scenarios where students may identify as non-binary or prefer not to disclose their gender.

and address validation:

Address Validation:

• Ensure that the system validates and stores addresses correctly.
• Test for different address formats and handle scenarios where students have multiple addresses (e.g., home and mailing addresses).

I drilled down further.
Prompt:
What user cases are needed to test enrolling students?

It gave me 18 user cases! In seconds! Such as duplicate prevention, grade placement, sibling enrollment, and other things that I may not have thought about such as waitlist functionality etc.

I then asked it about user cases for student scheduling, quick student look up, modifying an enrollment record. All of the users cases were useful and generated in seconds.

Then I asked it given our company’s URL, to write cypress code to test it.

It did a pretty good job!

Answer:
I can provide you with a simple example of Cypress code to interact with the given screen. Note that you may need to adjust the code based on the specific elements and interactions available in your application. This example assumes that the page has a dropdown or list of classes that you want to interact with.

I then asked it for more user cases about other functionality in our app!

[mrukavina (Marijana Rukavina)]

I found this blog post with some really interesting examples.

How machine learning in software testing produces superior products

It starts by describing the Machine Learning term.
As Carlos said in his video from the Day 04, it is a bunch of algorithms and math trying to find patterns and looks like they predict things.

It is listing 3 core learning types for the machine learning engine:

• Supervised Learning – when the algorithm is given the correct training data to learn.
• Unsupervised Learning – when the algorithm is given a ton of data and tests to see what the machine has learned.
• Reinforced Learning – is based on the ideals of the rewards system. Reinforcing learning by training the algorithm to be rewarded for “good” behavior and corrected for “bad” behavior.

Examples of AI used in real-world problems:

• Airbnb - uses AI to categorize listing photos. They have hundreds of millions of photo listings and tagging them manually would take months, but with AI it was matter of days. They also use it to decipher what was in the picture. This way the app can detect the features and amenities in a home from photos versus relying on the owner to list them.
• Autify - uses AI to automatically detect changes in user interfaces in regression testing

One interesting point was the listing of the top sectors that will have the greatest benefit from AI:

• Healthcare Automotive
• Financial Services
• Retail
• Technology
• Manufacturing
• Energy
• Transportation

This is one is very interesting to me: healthcare can use data-based diagnosis to support decisions in treatment.

As an example, the physicians in Nairobi, Kenya used AI to assist in more accurate respiratory disease diagnoses. They trained algorithms in sound analysis by developing large data set of various patient breathing patterns. So machine was able to learn patterns of “crackles” and “wheezes” to aid in more accurate respiratory diagnoses.
Of course this implies a lot of challenges like privacy concerns and protection of sensitive health data.

[nao (Nao Deng)]

I recently read this article https://mp.weixin.qq.com/s/qxS6ty0tS1QDpIqPFNDseQ It’s a study and concrete demonstration of a ground-up solution for anomaly detection methods based on UI interaction intent understanding.

The article is in Chinese, so you can read it by translating it to English through software.

1. Brief background on the case study or personal experience:

• Meituan’s Store Platform Technology Department and Quality Engineering Department collaborated with Professor Zhou Yangfan’s team from Fudan University to develop a multimodal UI interaction intention recognition model and a corresponding UI interaction framework. As Meituan’s various business lines expanded and iterated, the task of UI testing became increasingly burdensome, leading to the development of this model.

2. How Artificial Intelligence was used in their testing:

• AI was utilized to fuse user-visible text, visual image content, and attributes in the UI component tree to accurately identify UI interaction intentions. This approach was taken to address the challenges of high manual costs in UI testing and the reliance on script testing for UI interaction functionality logic.

3. Tools or Technologies Used:

• The research used multimodal models that combine machine learning methods with image, text, and rendering tree information to understand and replicate the “cognition-operation-check” verification process that a tester would typically perform.

4. Results Achieved:

• The case study showed that test cases written based on UI interaction intentions demonstrated the ability to generalize across different platforms, apps, and technologies without the need for specific adaptations. The research has been accepted by ESEC/FSE 2023 (a top conference in the software field) and will be presented at their Industry track.

5. What Impressed or Surprised You in this Example:

• The article does not provide a personal impression or surprise factor; however, the innovative approach to UI interaction intention recognition and its application to create generalized test cases that can be reused across various apps and platforms is noteworthy.