AI and Testing wiki

This wiki started at TestBash Spring 2023 during a panel discussion called “How AI Can Impact Our Testing Roles”. Feel free to edit and add new information and keep it up to date. Links to company/promotional material will be removed. For example, if adding a blog link, please only share a personal blog. Just look for the Edit button. It looks like this:
Screenshot 2023-03-29 at 12.18.59

Why should we care about AI in relation to the testing craft?

Provide compelling reasons why we need to care about AI and its impact on the testing craft

  1. Within several years most probably it will be everywhere and using AI will be an essential skill
  2. It will help you to be more productive

Examples of how AI can assist testing

Provide real-life examples of how AI has assisted your testing.

  • Generate test cases
  • Build end-to-end tests
  • Build unit tests
  • Helps to prioritize and select which tests to execute for a feature change
  • Generate some or all of the code of a test case
  • Significantly reduce test maintenance for end-to-end tests

Risks, worries and concerns about AI in relation to the testing craft

Capture any risks, worries and concerns about AI, the ethics of AI and its impact on the testing craft

  • There are many risks. Verified sources of data and unconscious bias in test data selection are major risks. These may result in lower quality if a system is tested using incorrect data and/or only a subset of possible data coverage. — Charmaine Short (source)
  • AI Models are sill in learning phase, at a big pic. In the linear algebraic\computational mathematical models we use, there may be delta error now, which later, at extreme level of calculations\optimizations which may end up in major error with impact to business. — Roopam Chopra (source)
  • Bias — Ville Rytinki (source)
  • add more

Ongoing conversations about AI and testing

Helpful tools that use AI to assist testing efforts

AI-assisted test tool vendors are welcome to add their tools to this list.

  • testRigor uses GPT-4 to build working end-to-end test based on plain English test case description
  • aqua generates test cases from a requirement or auto-creates test steps from a description
  • diffblue generates unit test cases from the given codebase
  • MagnifAI helps with visual testing
  • Codium “By analyzing your code, docstring, and comments, and by interacting with you, TestGPT suggests tests as you code.”
  • ScopeMaster uses AI to automate test generation from user stories. It also tests the user stories (like Sonarqube but for requirements)
  • Socket AI – Socket is using ChatGPT to examine every npm and PyPI package for security issues.
  • ChatGpt Test Case Creator by @guilhermevigneron
  • SofySense – advanced insights, analysis, and assistance for all your QA needs using the power of OpenAI.
  • SeaLights Helps to prioritize test execution
  • Katalon - provide a jira plugin that will read user stories and generate test cases.

Books on AI

Videos on AI

Articles on AI

Podcasts

Newsletters on AI

AI Industry news

18 Likes

Wow that’s an amazing collection of different sources to learn. Thanks for collating it.

4 Likes

In a research project with University of Texas Dallas we have developed a MVP with these key features:

  1. Prompt library to customize inputs for each testing need.
  2. Integrates 8 different LLMs to provide a range of capabilities.
  3. Optimized for generating software testing artifacts and content.
  4. Follows a client-server model for accessibility and scalability.
  5. Outputs can feed directly into testing and QA workflows.
  6. Enables iterative drafting and refining of test plans/reports.
  7. Designed to integrate with agile development environments.
  8. Improves velocity and quality of test documentation.

To recap, this software solution aims to enhance agile testing processes specifically by using its #promptdesign library and multi-LLM design to produce high-quality test plans, reports, and other QAtesting documentation more efficiently. The multi-LLM solution provides the flexibility and optimization to accelerate testing activity in agile environments.

Send me a note if you would like to learn more, I’m a new user and cannot attach the design documentation. The multi-LLM Harvey Ball analysis is available on LinkedIn here.

2 Likes

AI is a magician performing a trick. You don’t know how it works, you make assumptions, you’re amazed at the output. And the magician never reveals their secrets.

Always be careful about solutions you don’t understand, especially if you don’t understand the problem.

The responsibility stops with you. You’re the one who ends up in the news.