🤖 Day 19: Experiment with AI for test prioritisation and evaluate the benefits and risks

📆 30 Days of Testing
, ,
1

Day 19 already! I hope you can all appreciate how much we have covered already - well done!

Today we want to turn our attention to whether AI can help us make decisions about test selection and prioritisation and evaluate some of the risks and benefits of this as an approach.

Using data to make decisions about what to test and how much has been around for a long time (most testers are familiar with the idea of Risk Based Testing) and it’s natural to think about automating these decisions to accelerate. The technical evolution for this process it to delegate to an AI model that learns from data in your context about the testing performed and the observable impact of the testing.

The critical question is…Should we?

Task Steps

You have two options for today’s task (you can do both if you want):

  • Option 1 - If your company already uses an AI powered tool for test prioritisation and selection, then write a short case study and share it with the community by responding to this post. Consider sharing:
    • The tool you are using
    • How does the tool select/prioritise tests? Is this understandable to you?
    • How does your team use the tool? For example, only for automated checks or only for Regression?
    • Has the performance of the tool improved over time?
    • What are the key benefits your team gains by using this tool?
    • Have there been any notable instances where the tool was wrong?
  • Option 2 - Consider and evaluate the idea of using AI to select and prioritise your testing.
    • Find and read a short article that discusses the use of AI in test prioritisation and selection.
      • Tip: if you are short on time, why not ask your favourite chatbot or copilot to summarise the current approaches and benefits of using AI in test prioritisation and selection?
    • Consider how you or your team currently perform this task. Some thinking prompts are:
      • To what extent do you need to select/prioritise tests in your context?
      • What factors do you use when selecting/prioritising tests? Are they qualitative or quantitative?
      • How do you make decisions when there is a lack of data?
      • What is the implication if you get this wrong?
    • In your context, would delegating this task to an AI be valuable? If so, how would your team benefit?
    • What are the risks of delegating test prioritisation and selection to an AI model? Some thinking prompts are:
      • How might test prioritisation and selection fail, and what would the impact be?
      • Do you need to understand and explain the decision made by the AI?
      • “How did test/qa miss this?” is an unjust but common complaint - how does this change if an AI is making the decisions about what to test?
      • How could you mitigate these?
      • If we mitigate risks using a Human in the loop, how does this impact the benefits of using AI?
    • How could you fairly evaluate the performance of an AI tool in this task?
    • Share your key insights by replying to this post. Consider sharing:
      • A brief overview of your context (e.g. what industry you work in or the type of applications you test).
      • Share your key insights about the benefits and risks of adopting AI for test prioritisation and selection.

Why Take Part

  • Understanding where AI can help: There is excitement/hype about using AI to improve and accelerate testing. For teams managing large numbers tests, complex systems or time-consuming tests being more data driven about selecting and prioritising tests might provide real benefits. By taking part in today’s task, you are critically evaluating whether it works for your context, you learn about specific risks of delegating responsibility to AI and are better prepared to make a considered decision about AI based Testing Selection and Prioritisaiton.

:diving_mask: Dive deeper into these topics and more - Go Pro!

3 Likes
2

Hello Bill and fellow testers,

I work in Financial Services area where I test the UI of the vendor website on a daily basis across different browsers & devices.

The response I’ve got is:

As an AI programming assistant, I can provide information on how AI is used in test prioritisation and selection:

1. Predictive Analysis: AI can predict which test cases are likely to detect a defect based on historical data. This helps in prioritising these test cases.

2. Risk-based Prioritisation: AI can identify the risk associated with each test case based on factors like complexity, business impact, and defect history. This helps in prioritising high-risk test cases.

3. Test Case Optimisation: AI can identify redundant or obsolete test cases, helping to maintain an optimal test suite.

4. Intelligent Test Generation: AI can generate test cases based on the application's behavior and usage patterns.

5. Real-time Updates: AI can dynamically update the test suite based on changes in the application, ensuring that the test suite is always up-to-date.

6. Flaky Test Identification: AI can identify and isolate flaky tests, improving the reliability of the test suite.

The benefits of using AI in test prioritisation and selection include increased efficiency, improved test coverage, reduced manual effort, and higher quality software.

We need to decide which tests to do first because we can’t test everything at once. We look at things like:

  • Browser/Device Usage Stats
  • Past defect data
  • Criticality of the UI component
  • Frequency of usage of specific UI component

The implication of getting this wrong would lead to user frustration, damage of the brand reputation etc.

Using AI to help us pick tests could be really helpful. It’s good at looking at lots of information quickly and figuring out what’s most important to test.

Sometimes AI might miss things that are important to people/that happen in our specific job. So, we need to have people check the AI’s choices to make sure it doesn’t miss anything important.

Overall, using AI can make testing faster & better, but we still need people to double check its work to make sure we’re doing the best job possible.

6 Likes
3

Option 2

Context: You have worked in the payment domain for over 6+ years, testing various payment systems.

Key Insights on the Benefits and Risks of AI for Test Prioritization and Selection in Payment Systems:

Benefits:

  • Improved security: AI can help prioritize tests that focus on identifying vulnerabilities and ensuring the security of payment systems. This is especially important in the payment domain, where security breaches can have severe consequences.

  • Reduced fraud: AI can analyze historical transaction data to identify patterns and anomalies, helping to prioritize tests that target fraud prevention mechanisms. This can help payment systems detect and prevent fraudulent transactions, protecting both customers and businesses.

  • Enhanced compliance: AI can ensure that payment systems comply with industry regulations and standards by prioritizing tests that cover specific compliance requirements. This can help businesses avoid fines and penalties, and maintain the trust of their customers.

Risks:

  • False positives: AI tools may prioritize tests that are not actually necessary, leading to wasted time and resources. This is especially concerning in payment systems, where false positives could disrupt critical transactions.

  • Lack of explainability: It is important to understand the reasoning behind AI’s decisions, especially in high-stakes domains like payment systems. Without explainability, it can be difficult to trust the AI’s recommendations.

  • Bias: AI models can be biased if they are trained on incomplete or biased data. This could lead to the prioritization of tests that are not representative of the actual risks faced by payment systems.

Mitigating Risks:

  • Use AI as a complement, not a replacement: Testers should still maintain oversight of the AI’s decisions and retain the ability to override them when necessary.

  • Implement rigorous testing: Use a variety of testing techniques, including manual and automated tests, to ensure comprehensive coverage and reduce the risk of false positives.

  • Foster a culture of collaboration: Encourage testers to work closely with data scientists and AI engineers to improve the AI’s performance and mitigate risks.

Evaluation of AI Performance:

  • Compare AI-selected tests to manually-selected tests: Track the number of defects found and the time saved using AI.

  • Monitor the AI’s accuracy: Assess the ratio of false positives to true positives and adjust the model accordingly.

  • Use feedback from testers: Gather feedback on the AI’s recommendations to identify areas for improvement and make necessary adjustments.

In the payment domain, delegating test prioritization and selection to an AI could be valuable. However, it is crucial to mitigate the risks associated with AI by implementing appropriate feedback and evaluation mechanisms, as well as maintaining human oversight of the AI’s decisions.

Conclusion:

AI has the potential to revolutionize the way we test payment systems. By leveraging AI for test prioritization and selection, we can improve security, reduce fraud, and enhance compliance. However, it is important to be aware of the risks associated with AI and to take steps to mitigate these risks. By using AI as a complement to manual testing, implementing rigorous testing practices, and fostering a culture of collaboration, we can harness the power of AI to improve the quality and efficiency of our testing efforts.

7 Likes
4

Hi my fellow testers,

I chose option 2 for today’s challenge.

Find and read a short article that discusses the use of AI in test prioritisation and selection.

I chose to read this article AI-Driven Test Prioritization

Consider how you or your team currently perform this task.

We have a group discussion around risk and any tests that are deemed a higher risk will be tested first

What factors do you use when selecting/prioritising tests? Are they qualitative or quantitative?

Again its based on risk, unless it’s seen as a very low risk feature then the test cases will ordered in a logical path through the software that tries to imitate the path a user may take

How do you make decisions when there is a lack of data?

I ask questions and make sure I have all the data I need before I start the testing

What is the implication if you get this wrong?

High risk test cases are missed, which have the potential consequence of missing severe bugs

In your context, would delegating this task to an AI be valuable? If so, how would your team benefit?

I would be hesitant to palm this task off to an AI if the consequence of that meant that the conversations around risk no longer took place. I really value these conversations and think they are highly effective and would be concerned that the high standard of quality in our software would start to slip without them

In an ideal world we would somehow feed these conversations into an AI model so that it could use them within its dataset as an input to where the risk in the feature is

What are the risks of delegating test prioritisation and selection to an AI model? Some thinking prompts are:

Some of the risks I see are a lack of context awareness and getting the AI to explain its decision making when it could be quite a black box

How could you mitigate these?

It would need to be fully knowledgeable of our context and it would need to be able to effectively explain its decision making

If we mitigate risks using a Human in the loop, how does this impact the benefits of using AI?

I worry that this would change us from being critically thinking people to babying an AI to give us the output we want - I don’t want that job

How could you fairly evaluate the performance of an AI tool in this task?

It would ultimately need at least to do as good as job as we do currently in this area, if not better.

5 Likes
5

Hi Everyone

:robot: Exploring the idea of using AI for test prioritization and selection was an intriguing journey. Here’s a peek into the insights I uncovered along the way:

  1. Industry Context Matters: Each industry presents unique challenges and requirements. Assessing whether AI can effectively aid test prioritization requires a deep understanding of the industry landscape and its testing demands.
  2. Data, Data, Data: AI thrives on data. Without comprehensive and relevant data, AI-driven test prioritisation may falter. Access to historical testing data and metrics is paramount for training AI models effectively.
  3. Balancing Act: Test prioritisation encompasses both quantitative metrics and qualitative considerations. AI models should be capable of weighing both aspects to make informed decisions that align with organisational goals.
  4. Consequences of Missteps: Incorrect test prioritisation can lead to detrimental outcomes, including delayed defect detection and compromised user experiences. Understanding the potential risks is crucial before fully embracing AI-driven approaches.
  5. Human Oversight: While AI brings speed and scalability, human oversight remains indispensable. Humans provide the critical domain expertise and context interpretation necessary to validate AI decisions and intervene when needed.
  6. Risk Mitigation Strategies: Mitigating risks associated with AI-driven test prioritisation demands a proactive approach, including continuous monitoring, feedback loops, and ensuring transparency in decision-making processes.
  7. Evaluating AI Performance: Fairly assessing the performance of AI tools requires defining clear metrics and benchmarks that encompass accuracy, efficiency, and adaptability.

In the end, the journey reaffirmed the importance of a balanced approach, leveraging AI’s capabilities while acknowledging the value of human judgment and oversight. With careful consideration and strategic implementation, AI could indeed revolutionise test prioritisation in the testing landscape.

Thank you

4 Likes
6

There’s some overlap between today’s question and my answer back in day 6.

In context of automation and regression runs, I see a benefit of a tool that would select only relevant tests in the context of the change being made by developer. It doesn’t really have to be AI - you can relatively easily script something close enough, assuming you have the right data.

The main benefit is faster PR regression testing and shortening the time from opening the PR to merging it. It’s not too bad in my team, as full regression run takes 30-35 minutes. But I’ve been in teams where it took 12 or more hours, and shortening that by even 25% would be great improvement.

The obvious risk is that tool might decide to skip the test that would actually encounter some issue. You can minimize that risk by keeping nightly full regression runs. Tool should also provide configuration option, so you can decide if you want it to be liberal or conservative when it comes to selecting tests.

In context of wider testing activities, I would not want a tool to decide what am I supposed to work on and in what order.

4 Likes
7

Thanks for your responses.

Do you see this risk as being more significant or likely to become an issue compared to your current approach (or approaches you’ve used on teams where selecting subsets of tests is more impactful)?

1 Like
8

Thanks for the thoughtful answers

In your team how do you decide on risk? Are there factors you consider in making that risk assignment? Are there factors you’d like to include but can’t due to the complexity or time to determine?

2 Likes
9

One of the main factors we use is an analysis of the code that’s changed, if the code change has affected a lot of different areas of the software then that will be deemed high risk. We’ve had missed bugs before with this sort of impact where the dev wasn’t aware of how wide spread their work was or weren’t aware that one part of the software touched another. I will also do some impact analysis with our product owner where we try and think of where the risk is.

1 Like
10

I work with Energy Trade and Risk Management.

so I asked Microsoft Co-Pilot
As a QA Engineer, understanding the current approaches and benefits of using Artificial Intelligence (AI) in test prioritization and selection is crucial. Let’s delve into this topic:

  1. Test Case Selection and Prioritization (TSP):
  • Problem: Regression testing ensures that software changes do not adversely impact existing functionality. However, running all tests can be time-consuming and resource-intensive.
  • Solution: TSP techniques aim to select and prioritize test cases, providing early feedback to developers.
  • AI Integration: Researchers have increasingly relied on Machine Learning (ML) techniques for effective TSP (ML-based TSP). These techniques combine information from various sources to create accurate prediction models12.
  • Benefits:
  1. AI-Driven Test Case Generation:
  • Problem: Creating comprehensive test cases manually is challenging.
  • Solution: AI algorithms generate diverse test cases based on code complexity, past bug history, and critical software features.
  • Benefits:
  1. Defect Analysis and Prediction:
  • Problem: Identifying critical defects efficiently.
  • Solution: AI analyzes historical defect data, code changes, and user behavior to predict potential issues.
  • Benefits:

I can attest to the Problem as we have a Regression test bed set that takes 14 hours to run. This is due to how the tests were built over 15 years, and also the complexity and volume of tests(almost 8000 iterations now).

Ideally I would love to get these times down to 4-5 hours. This would be possible but very expensive, as the tests could be split over multiple machines and old code refactored to async, a newer version of .Net etc…

I am currently working on moving as much of the tests as I can to .net 8, and of course new tests go to .net 8 immediately. However this is not going to be completed any time soon.

From what I have seen so far, I am not sure there is the maturity within AI modelling to resolve my issues. When you have 15 year old code, you have 15 year long problems.

For me, I need to get the tech right and better architected frameworks, then perhaps AI may be of use within this context.

3 Likes
11

I’ve been going through a bit of journey with AI. When I started looking at this in early January, I had great expectations. The more I’ve learnt, the less confident I’ve become. The key here is “bias”. The LMMs that we use at the moment have inherent biases that no matter how much “training” we provide, will always exist.

When it comes to prioritisation, the tests we select are context specific. Further, each context is different. We would need an LLM that can be trained for every context that could possibly arise before we need it. The LLM would also need to understand current market trends for your product, the financial needs for your company, the desires of your marketing department to future proof your product, the skills of your team and testers as well as knowing intrinsically all of the tests you have. These factors are all considered consciously or unconsciously whenever we assess what tests we need to run.

I would argue that for an activity such as “prioritisation”, using an LLM could at best provide some guidelines, but at worst, could mean that you use the wrong tests and set things further back. Personally, I would not use an LLM to prioritise any work at the moment.

6 Likes
12

Hello, @billmatthews and fellow learners,

I did a lot of studying as part of my today’s link and it helped me do a lot of learning.

I have summarized my learnings in the form of a mindmap here:

AI Test Prioritization
AI Test Prioritization1920×1179 123 KB

I have also explained my approach, various learnings, and reasoning behind each of these points in the form of a video.

Check it out here:

Do share your thoughts & feedback on this :slight_smile:

Thanks,
Rahul

5 Likes
13

When you compare AI vs any other tool in this specific context, I think the risk and impact is about the same. In my team we would reap all the benefits until it happens for the first time that tool omitted a test that was actually significant, at which point we would have a conversation - maybe we just accept this happening sometimes? Depending on the outcome we would look into adjusting the tool. If tool does not allow adjustments like that, we would look for alternatives. At the worst case we could decide this whole test selection shenanigans was not worth it and go back to full regression runs.

In general, I would put that in the same category as all other problems with pipeline and delivery. First and foremost team has to decide what they care about and what costs they are willing to pay. A tool employing AI is unlikely to impact the calculus in significant way.

1 Like
14

Janet and I are writing a blog post about flaky tests, and in my book, those are priority ones to FIX (or fix the code, or throw out!) Honestly, it’s never been hard to identify flaky tests, my teams always found them pretty easy to track. Still, I was curious.

So I asked Gemini if AI or LLM tools could be used to identify flaky tests in CI. It gave me a helpful answer. Here’s a part:

Many CI platforms now have built-in functionalities that leverage AI or statistical analysis to automatically detect tests with inconsistent results across multiple runs. These tools analyze historical test data, identify failure patterns, and calculate flakiness probability. This helps you prioritize tests that need investigation.

Looking around at tools, I found at least one that claims to use AI to help identify flaky tests: Use AI to find Flaky Tests

Looking further, it appears to me that CI tools already have data gathering and reporting built in to help teams identify flaky tests, and AI isn’t really required to do this. But I can see, it has potential. A machine could be trained to recognize flakiness.

The conclusion of Gemini’s answer to my question was surprisingly wise:
** * Custom Annotations: While AI can detect flakiness, you can also use custom annotations or markers within your tests to flag potential flaky behavior and provide context for further investigation.*
** Collaboration: Combine AI’s analytical power with human expertise. Developers and testers can use AI-generated reports to pinpoint flaky tests and work together to resolve the underlying issues.*

By using AI and LLM tools in conjunction with other best practices, you can significantly improve the efficiency and reliability of your CI process by identifying and eliminating flaky tests.*

3 Likes
15

Thanks for sharing the insights. Which tool have you envision on using or currently utilising for this kind of analysis. Please share your insights.
Thanks!

16

What I learned from ChatGPT today:
quote
ALM Octane, developed by Micro Focus, doesn’t have built-in support for integrating AI tools directly into its test cases module for test prioritization. However, you can integrate AI tools for test prioritization with ALM Octane through custom development or by using its API capabilities.
While integrating AI tools for test prioritization into ALM Octane may require some development effort, it can significantly enhance the efficiency and effectiveness of your testing process by ensuring that the most critical test cases are executed first.
unquote

3 Likes
17

Hello there :vulcan_salute:

I’ll go with the Option 2.

I asked Microsoft co-pilot summarize the current approaches and benefits of using AI in test prioritization and selection

AI in test prioritisation and selection is a growing field that offers several benefits:

1. **Efficiency**: AI can help prioritize tests based on their relevance and likelihood of catching bugs. This can reduce the time spent running irrelevant tests and increase the speed of the testing process.

2. **Accuracy**: AI algorithms can learn from past test results and use this information to predict which tests are most likely to fail. This can improve the accuracy of test selection and help catch more bugs.

3. **Adaptability**: AI can adapt to changes in the codebase and automatically adjust the test selection and prioritisation accordingly. This can be particularly useful in agile development environments where the codebase is frequently changing.

4. **Automation**: AI can automate the process of test selection and prioritisation, reducing the manual effort required and freeing up human testers to focus on more complex tasks.

Current approaches to using AI in test prioritisation and selection include machine learning algorithms, predictive modelling, and risk-based testing. These approaches can be used to analyze historical test data, identify patterns and trends, and make predictions about future test outcomes.

And then I asked What are some common challenges in implementing AI-based test prioritisation and selection?

Implementing AI-based test prioritisation and selection can present several challenges:

Data Quality and Availability: AI algorithms require large amounts of high-quality data to train effectively. If the available test data is limited, outdated, or inconsistent, the AI may not be able to accurately prioritize and select tests.

Complexity: AI algorithms can be complex and difficult to understand. This can make it challenging to interpret the results and to troubleshoot when things go wrong.

Integration: Integrating AI into existing testing frameworks and workflows can be challenging. It may require significant changes to the way tests are written and executed.

Maintenance: AI models need to be regularly updated and retrained as the codebase and testing requirements evolve. This can require significant time and resources.

Over-reliance on AI: While AI can be a powerful tool for test prioritisation and selection, it's important not to rely on it entirely. Human insight and expertise are still crucial for effective testing.

Cost: Developing, implementing, and maintaining an AI-based test prioritisation and selection system can be costly. It's important to weigh these costs against the potential benefits.

My impressions about it is that for big problems that take a lot of time to run the automation or to do the regression test the use of AI to prioritize the tests, but for small companies with only a few tests I feel that is faster to know the system and prioritize based in your knowledge of what is more used in the system.

That’s it for today :wink:

4 Likes
18

Day 19

Find and read a short article that discusses the use of AI in test prioritisation and selection.

This was a curious search as, it seems academia is interested in this problem. Mostly trying to answer the question:

If we have a test specification, which tests should be automated?

This is a must less interesting question than ‘what shall we test’ or even ‘at what level should we test.’ Also, its less of a challenge in my opinion. It depends on having a test specification which is not really how most teams work, they may have acceptance criteria or define behaviours in gherkin syntax or something. Maybe thats enough. There are some interesting ideas around codebase analysis and selecting tests based on what is connected to what has changed.

Consider how you or your team currently perform this task.

At the moment:

  • Elaboration of ticket with initial acceptance criteria - add test ideas and areas to check for regression.
  • Story Kick Off - 3 get together and elaborate further, more tests, where to tackle them (unit, widget, integration, exploratory).
  • Handover - developer and tester get together, go over low level tests and add any more test ideas.
  • Testing - further scenarios are generated during exploration.
  • Results - captured in a comment on the ticket. They disappear somewhat.

In your context, would delegating this task to an AI be valuable? If so, how would your team benefit?

I feel like with the right training data (code, model of the system, test results, bugs, tickets) you could have a model come up with the first pass for review by the team. As long as the collaboration continued as per my list above, and we could build the test selection iteratively. I think areas for regression would be powerful for us, as in our app there are lots of things that need testing manually (location tracking in particular).

What are the risks of delegating test prioritisation and selection to an AI model?

  • Diminishing the collaborative aspect of test selection and prioritisation. Between humans I mean.
  • I wonder how well the history of an application would be known as well. I have tested this app for 18 months and I know the areas that go wrong (or at least I feel I do).

How could you fairly evaluate the performance of an AI tool in this task?

I think, I would select 2 of the big four:

  • Lead time for changes - from first commit to deploy you would want to see an improvement.
  • Change failure rate - less remediation in production.

Always try and use the big four, rather than making them testing specific metrics or even related to the AI itself.

3 Likes
19

As I mentioned in my response in Day 6

In the blogpost AI Driven Test Development Cases Studies (URL: https://www.linkedin.com/pulse/ai-driven-test-development-cases-studies-ankur-chaudhry/) I saw the following statement (a testing question)

“If I’ve made a change in this piece of code, what’s the minimum number of tests I should be able to run in order to figure out whether or not this change is good or bad?”

Code which is modified, is modified in two distinct ways:

  • Existing code that has been changed
  • New code that is added

And of course there is also code that has not been changed

The situation ‘Existing code that has been changed’, you must be triggered by the fact that you must have some automated regression tests that always checked that part of the code that now has been changed. These automated regression test are becoming very important test case to run.
I would be great if A.I. can figure that out and provide an overview of these selected tests. As a responsible person for the testing part, I want to evaluate if the overview is correct. After my evaluation, I will proceed with the selected test set that cover the ‘Existing code that has been changed’. This selected test set has elevated in priority

The situation ’New code that is added’ is something completely different. There are “no tests”, they have to be created. These new tests can be created with AI based on the requirements, or these new tests can be created based on the code (or new UI created by the code).

I would be great if A.I. can figure that out and provide an overview of these new tests. As a responsible person for the testing part, I want to evaluate if these new tests are correct. After my evaluation, I will proceed with these new tests that cover the ‘New code that is added’. These new tests are now labeled as high priority.

Some of these new tests will be automated (can be done by A.I.) and will be added to the automated regression test set.

For sure I would delegate this test selection (and creation) and prioritization to an A.I., as I see the A.I. as an assistant. It helps me to quickly guide me to a possible outcome that can beneficial. As I don’t know how the A.I. comes with this possible outcome, it is for me to judge if it is correct and useful.

3 Likes
20

My Day 19 Task

Based on my current work status, for Day 19’s task, I chose Option 2

Regarding Reading an Article Discussing the Use of AI in Test Prioritization and Selection

The article I read is AI-Driven Test Prioritization

The main content of the article is as follows:

AI-driven test prioritization enhances efficiency and reduces costs in software testing.

  • :page_facing_up: Understanding Test Prioritization: Determining test case execution order for maximum coverage and defect detection.

  • :star: Significance of AI-Driven Test Prioritization: Enhanced efficiency, faster feedback, optimal resource utilization, improved test coverage, and reduced costs.

  • :hammer: Implementing AI-Driven Test Prioritization: Data collection, feature selection, machine learning model, prediction, and integration with testing tools.

  • :office: Real-Life Examples and Implementations: Google and Microsoft successfully use AI-driven test prioritization.

AI-driven test prioritization revolutionizes software testing by utilizing machine learning to rank test cases based on their likelihood to fail, enhancing efficiency and coverage while optimizing resources. This method relies on analyzing vast data, including historical test results and code changes. Implementations by giants like Google and Microsoft illustrate its value in reducing testing times and accelerating development. By integrating AI prioritization with testing tools, organizations can significantly improve their software quality and reduce costs, underscoring the transformative impact of AI in software development.

Regarding Summarizing the Methods and Benefits of Using AI Tools in Test Prioritization and Selection

I chose ChatGPT-4, and its response was:


AI in test prioritisation and selection, particularly within the context of software development and quality assurance, is a rapidly evolving area that leverages machine learning (ML), natural language processing (NLP), and other AI methodologies to enhance the efficiency and effectiveness of testing processes. Here's a summary of the current approaches and benefits:

### Approaches

1. **Machine Learning Models**: AI systems are trained on historical data, including past test results, code changes, and defect rates, to predict which tests are most likely to detect faults in new or modified code. Techniques such as regression, classification, and even deep learning models are employed.

2. **Natural Language Processing (NLP)**: Used to analyze requirements and change logs to identify relevant tests. This approach helps in understanding the context and semantics of changes, thereby selecting tests that are more likely to be affected by those changes.

3. **Test Case Embeddings**: Similar to word embeddings in NLP, test cases are represented in a high-dimensional space, where their similarities to code changes or features can be measured. This helps in identifying which tests are most relevant to the current changes.

4. **Prioritization Algorithms**: These algorithms rank tests based on various factors, such as historical failure rates, test execution time, code coverage, and impact of recent code changes. The aim is to run the most critical tests early in the cycle.

5. **Feedback Loops**: Incorporating continuous feedback from test outcomes to refine and adjust the models and algorithms used for test selection and prioritization, ensuring they become more accurate over time.

### Benefits

1. **Reduced Test Times**: By prioritizing and selecting the most relevant tests, AI can significantly reduce the overall time required for testing, enabling faster release cycles.

2. **Improved Test Coverage**: AI techniques can uncover gaps in test coverage by analyzing code paths and identifying untested or under-tested areas, leading to better software quality.

3. **Cost Efficiency**: Reducing the amount of testing without compromising on quality leads to lower computing resources usage and human effort, thereby cutting costs.

4. **Early Defect Detection**: Prioritizing tests that are more likely to fail can lead to early detection of defects, which is generally less costly to fix and leads to more stable releases.

5. **Adaptability**: AI models can adapt to changes in the software and testing environments, ensuring that the test prioritization and selection processes remain efficient over time.

6. **Risk Management**: By focusing on high-risk areas based on code changes and historical defect data, AI helps in better managing the risks associated with software releases.

These AI-driven approaches represent a significant shift from manual and static methods to dynamic, efficient, and intelligent test management strategies, aiming at improving software quality, reducing costs, and accelerating development cycles.

Regarding Reflecting on How You or Your Team Currently Execute This Task

Background and Available Information of My Current Project

  • The current project has tight delivery timelines, with significant pressure on task delivery and a risk of scope creep.

  • Existing project SOPs require compatibility with different browsers, resolutions, and devices.

  • The project is an e-commerce online shopping website, primarily for the web platform.

The Necessity of Test Priority Sorting

It is necessary. From the beginning of the project, based on the background and SOP, an initial testing strategy was developed. It included test priorities: business functionality testing first, followed by compatibility testing, then performance and network testing, and finally usability and ease-of-use testing.

Factors Relied Upon for Choosing/Prioritizing Tests

Mostly quantitative, with some qualitative aspects

  • Project team background

  • Project delivery pressure

  • Project SOPs

  • Team personnel configuration, more about the ratio of developers to testers

  • Results of negotiations and communications with the team

Decision-Making in the Absence of Data

Refer to useful information from past projects, negotiate and confirm with the team, and then make a decision.

Here, I must mention that the testing strategy and priorities are always iterative and updated, not set in stone. They can be adjusted based on the project situation and more information obtained.

Regarding The Value of Test Priority Selection to AI Models

There would definitely be value. AI could provide more reasonable and lower-risk outcomes based on known qualitative and quantitative historical data within the model.

Regarding The Risks of Test Priority Selection to AI Models

Risks are inevitable, mainly because of my concerns about data privacy and security with AI model tools. I would not transmit 100% of the project’s context and known information to AI. Therefore, the results generated by AI without full information might significantly differ from the team’s expectations. If the project is executed based on the outcomes provided by the AI model, it might not be possible to complete the project’s delivery on time.

Blog post link:30 Days of AI in Testing Challenge: Day 19: Experiment with AI for test prioritisation and evaluate the benefits and risks | Nao's Blog

1 Like