🤖 Day 6: Explore and share insights on AI testing tools

My take away from this is:

  • TestCraft: Looks promising. Generates some good test ideas, obviously doesn’t understands the context and business rules but you can pick what suits your app
  • Testim: Will have to try it out
  • PlaywrightGPT: Will try this and it should save me time researching playwright methods.

Why isn’t there anything for native mobile apps testing?

1 Like

@lisacrispin @restertest Was just about to mention Ioan’s video. :smile:

1 Like

Hey @hananurrehman,

Would you mind sharing your reasons for this?

I reckon it would very much help @natebosscher, who is the founder and developer.

1 Like

Ah yes, should’ve been more descriptive here :sweat_smile:
Well for one, I find the UI too cluttered and its hard to understand what it’s telling me about the issues/tests.
The autofill was expected to identify the kind of data to input based on the field label. E.g. If the field I’m filling in has the label or some attribute value that says “Description” then I’d expect a lengthy string. Also, it doesn’t seem to work for dropdowns and the fill all didn’t seem to work, at least for my web app.
Lastly, running any tests doesn’t seem like a good UX.

3 Likes

Focusing on test case generation, I tried to look for some AI MindMap tools to help assist in brainstorming test cases and testing aspects, for manual exploratory testing or functional testing.

Here are a few that I have tried, by giving the same prompt with 2 points of acceptance criteria, and ask them to design the test cases and draw a mindmap.

  1. Chatmind: https://chatmind.tech/
    It focus more on listing out cases for the numberic factors in my example, breaking down the cases in different values for fields like date, points, birthday. It is useful in reviewing if our test data has enough coverage.
    Context: ★★★
    User Experience: ★★★

  2. MyMap: https://www.mymap.ai/
    It provides more diversed and general aspects of testing, which has a higher overall coverage but less into the details or test data, such as suggestions on audit logging which my example originally did not mention as a criteria. It would be useful for brainstorming a higher-level test plan.
    Context: ★★★
    User Experience: ★★★

  3. GitMind: https://gitmind.com/
    Generates traditional functional test cases, list out postitive, negative, edge cases, performance test cases. Good level of details and comprehensive.
    Context: ★★★
    User Experience: ★★★

  4. Wondershare EdrawMind: https://www.edrawmind.com/
    Generates traditional functional test cases but less amount, list out only the main test cases for the criteria. Not as useful for brainstorming, but could be useful if one only wants a short list of acceptance test cases.
    Context: ★★☆
    User Experience: ★★☆

  5. Whimsical: https://whimsical.com/
    This one feels like a breakdown of the given lines into keywords and it takes them as factors to test. e.g. My given line has keywords like admin, points, reward, birthdate, and it add one node for each of these “factors” on the mindmap, describing a one-line test case to verify the corresponding validity.
    Context: ★☆☆
    User Experience: ★★☆

  6. Taskade: https://www.taskade.com/
    The UI looks fancy but the UX is not very clear and I failed to create a mindmap. Yet it gives me a list of test cases, which is similar to other GPT test cases.
    Context: ★☆☆
    User Experience: ☆☆☆

  7. Jeda: https://www.jeda.ai/
    It generates a memo-style visual which looks good in presentation, but not very useful for professional testing, as it is like listing a few examples for the criteria instead of test cases.
    Context: ☆☆☆
    User Experience: ★★☆

It is just a brief trial so please just take the stars as my first impressions only. Also the test above only looks into the performance of the direct result, pricing and other factors like mindmap editing, team collaboration are not reviewed.

8 Likes

Ooh I had no idea that AI mindmap tools existed or were even a thing, thanks for sharing.

5 Likes

Thank you for taking time to elaborate @hananurrehman!

I appreciate the UI concern - it’s really hard to squeeze the functionality into a small sidebar. I actually have adjusted the main screen in the next release to hopefully be a bit easier to understand.

If you wouldn’t mind, please DM me the site you were having issues filling. Before February 2, 2024 we weren’t able to detect some input-types for fields where the labels didn’t follow the <label for="" /> spec. If there’s other issues with Fill I’d like to fix those as well.

I’d be curious to know what you expected when you opened up Testing Taxi. And if you have specific suggestions on how we could do better. Happy to take this offline too if you prefer.

3 Likes

Thanks, I’ll take a look! It is very hard to keep up with all the posts here. Nice that so many people are diving deep into these tools.

2 Likes

:wave: Hi @lisacrispin, I’m the founder at Testing Taxi.

We use a very basic form of AI to help with our Fill, Edge Case and Suggestion prioritization features. You may have expected more of the chat-style interface or wordy outputs that we’ve become familiar with in the GPT (and similar) models.

In our testing we found that the long form GPT-style output was not as actionable or insightful as we would expect in an every-day testing tool. So, instead we opted for a issue-library approach for suggestions and use an AI to decide which suggestions to show based on the rendered page. We did a similar thing for edge cases. We created equivalence-classes library for fill edge cases and use AI to prioritize and generate within the classes based on historical fills and page content.

In short, we do use AI, but it’s not the LLM style that you might have expected. We took this approach because we believe it gives better, faster, more actionable results.

If you have suggestions on how we could use AI better, or where our approach doesn’t work as well as you would expect - I would love to hear your thoughts :slight_smile:

6 Likes

Day 6: Explore and share insights on AI testing tools

I chose to explore testsigma, this tool is designed to automate tests on Web, mobile, desktop apps and APIs in one platform.

Test maintenance is handled through auto-repair by AI as your test evolves over time. It also completes AI generated regression plans and identifies all subsequently affected test cases.

I find that a tool that can automate regression testing and auto update your tests would be very helpful!

2 Likes

I took a close look at the tools, I have started looking at in the task of day 3.

Katalon:

Success story in healthcare How Katalon helps Care Logistics | Healthcare Logistics Software Provider

Based on Selenium and Appium

A lot of integrations Katalon Platform | Integrations for End-to-End Software Testing

Limited Unit and API testing allows bugs to pass through and creates reliability issues. Prevent these issues with thorough testing.

on-demand environments in the cloud make it easy to run tests in parallel across browsers, devices and OS while we take care of the heavy lifting of setup and maintenance.

Easy no-code test recording + powerful full-code scripting.

Map automated tests to existing manual tests with one-click integrations

Optimize test coverage and run the right tests at the right time with dynamic test suites and smart

It could support all phases of testing process starting from test automation, codless tesing and integration Automated Script, Test Case Optimization, Automated Test Execution, Self-Healing Capabilities, Integrations to Reporting.

TestResults.io:

execute the automated test cases as regression, stability or performance tests

no unit tests, no UX testing (only layout)

TestResults.io combines a no-code GenAI prompt-to-automate designer and a low-code C# designer.

I am not sure how much this tool can impact the testing process, I have not found enough details for me here.

testRigor:

tests can be generated based on your own documented test cases

desktop testing, web testing, mobile testing, API testing

You can use plain English to build test automation. testRigor will understand and execute your instructions exactly as written.

There are a lot of features Features - testRigor Test Automation Tool

I see there are a lot of aspect which can support testing process and efficiency.

I think Katalon seems to be very interesting from these three given the information, I was able to find. Reading the provided information on the webpages this tool seems really powerful in regards of test automation, cordless testing and integration Automated Script, Test Case Optimization, Automated Test Execution, Self-Healing Capabilities, Integrations and Reporting.

It would be useful to give these tools a try, to see with which complexity of the system under test the tools can deal with.

2 Likes

Bonus task: Add a tool to the Software Testing Tools directory.

  1. Visit Ministry of Testing’s Software Testing Tools directory
  2. Search for tool you’d like to add
  3. If it doesn’t already exist, select + Add Tool and fill out the simple form

Wonderful! You’ve just helped a future community member with their tool investigations. :raised_hands:t2:

4 Likes

Added ‘Percy’ tool to the directory!

2 Likes

I took some time to install TestCraft as a Google extension using Playwright for the automation. Once I identified it as an extension, installing it was a breeze. From the beginning, it has been user-friendly; it even suggests both positive and negative test cases for automation. The “automate” step appears to be quick. We are learning Playwright, and I feel that this tool will assist us in leveling the ramp-up curve. In addition, the technologies in our software product vary, making automation testing a bit less generic (i.e., different grid configurations cannot use the same tests, or locators are challenging to nail down).

2 Likes

I’ve written up a review of the TestCraft web extension which I’m very impressed by. There are some limitations, however it is a free tool and the fact it can be added as a browser extension rather than downloaded is definitely a positive.

Its Accessibility Checker feature is particularly useful. I like how it can focus on particular elements (most Accessibilty Checker tools concentrate on entire pages). This is useful for web pages with lots of features on it (like dashboards) as it focuses on that one feature only. An accessibilty report for an entire page can be confusing, and misses issues that are limited to single features.

Main issues with it are:

  1. It only focusses on elements rather than user interactions and user journey through the page. This means the ideas can be limited to just the individual element, rather than groups of elements that interact with each other.
  2. User experience could be improved as you have to select an element for each action. When generating automated tests, if the settings needed changing, the element needs reselecting each time.
2 Likes

I like Mabl very much for teams that are not very technical or when their user journeys are too complicated for the developers to understand (e.g. in insurance sectors).

1 Like

Hi

I’ve spent some time on Testing Craft and Testing Taxi. Both tools have unique functions and can be helpful to testers, (especially juniors) to get things started. Tools can guide them on how and where to start

3 Likes

With my current ignorance of AI I looked into two tools mentioned so far on MoT, and one that I currently use.

Mabl : seems to follow the mantra of most of these tools, self healing, quick test generation, low maintenance, E2E . With respect to Agile increases velocity leading to better quality and coverage. There was nothing I saw to dispel these claims, so as with other similar tools, I will take that as read.
What impressed was the scope of interoperability offered with CI/CD tools, in particular Jira and MS Teams. I also liked the support for Data Driven Test Architecture.

Loadrunner : This one was of more interest in my current role as our applications deal with Trade and Risk Management so we have complex algorithms and calculations, and our client’s have high volume data sets.
While we don’t have performance monitoring(I prefer this saying to testing, as we aren’t testing as much as monitoring the behaviour with large data sets on the applications) to he degree we should, the idea of a toll that will assist with predictions with scalability of data volumes would have me looking further int o this.
It also comes with a free trial and ready made test database, as well as a community edition.

Co-Pilot for Visual Studio
This is something I currently use and overall have only positives with it.
To get the best of this it makes me think more about how I design my code, rather than just jumping in. It is a predictive tool so design classes etc… correctly and this is a big time saver on mundane and boilerplate type code.
It does have its moments when it tries to predict correctly but completes with total gibberish in terms of code.

I find UI test automation tools to be very alike and you should pick what suits as most will cover all needs in this sphere.
What interests me more are tools like Mabl and Loadrunner that will help me now and going forward in terms of taking the heavy lifting and allowing me to focus on design and architecting test strategies and frameworks.

As a natural cynic on where AI is heading, so far my cynicism is waning.
However, we should remember these are tools. Hopefully employers look at AI in QA the same.

2 Likes

This topic actually got repetitive.

1 Like

The following software testing tools pioneer the AI testing trend and incorporate AI technologies into their systems to bring software testing to the next level.

1. Katalon Platform
Katalon is an AI-powered testing platform with many AI features, including:

StudioAssist: Leverages ChatGPT to autonomously generate test scripts from a plain language input and quickly explains test scripts for all stakeholders to understand.

2.TestCraft
TestCraft is an AI-powered test automation platform that revolutionizes regression and constant testing by leveraging the power of Selenium and offering comprehensive web application monitoring capabilities.

3. Applitools
Applitools is a software that manages visual applications and employs visual AI for AI-powered visual UI testing and monitoring. The incorporated AI and machine learning algorithms are fully adaptive, enabling it to scan and analyze app screens like the human eye and brain, but with the capabilities of a machine.

4. Testim Automate
Testim Automate is a test automation platform that uses machine learning to address two recurring challenges in software testing: slow test creation and extensive test maintenance. With Testim, individuals without coding skills can swiftly generate end-to-end tests using its recording functions. Engineers can also utilize code to expand on these capabilities, combining the best of both approaches.

1 Like