Codeless test automation tool

Hi,
This week I am attending some training sessions with a codeless automation tool called TestSigma.

I’m wondering if anyone out there is also using a codeless automation tool - how are you finding it? Pros/cons?

I’m making an assumption that coded automation is more robust and flexible - but I could be completely wrong!

5 Likes

A year ago I explored Selenium IDE. It is Selenium with low code flavour.

Pros

  • ít has record and playback.
  • the code can easily be modified using the autofill option. It is also readable for newbie programmers.
  • the code can be exported to another programming language like Java. If the tool will not supported in the future, then tests will still be available. There is no vendor lock in.
  • the tool has several suggestions for locating for Web UI elements.
  • it is open source.
  • there is a free course on Test Automation University available.
  • it had options to make functions for repeating steps.

Cons

  • it had some bad suggestions to locate web elements.
  • it has another language than the programming language which the developers are using. So you need other people for support.
  • the testers with a programming background prefer Selenium with known programming language.
  • the programmers still need to use proper web element ids.

Other points of attention:

  • what are the results of a proof of concept? In the past one company did not sell licenses, because simple processes could not be scripted in the time they promised. BTW this was a live code session.
  • is your stack supported? E.g. Angular.
  • how does it handle time delays?
  • is it data driven? Here is a table with combinations of valid and invalid usernames and passwords. Please use this table for the login test.
2 Likes

TL;DR - I avoid codeless tools if I can. But, I might be open to using them if my needs will never change significantly.

I have used two codeless automation tools, one for non-testing work & another for testing work. I have seen short demos of some more. I noticed that such tools are essentially a UI layer over code/frameworks. Ex. Functions can become drag & drop widgets with text fields for inputs & output fields. Custom code can be another widget which opens up a code editor etc. Ex. SoapUI is one (free) such tool for API automation (REST & SOAP). Try a few realistic use cases on it and quickly see how frustrating it can be.

Coded automation is hard to develop & maintain. It is hard to find & retain good employees to do coded automation. But, it also has flexibility as you mentioned. Code tools have benefits like searching, refactoring, debugger, integration with linting & code analysis tools and more. These features are unlikely to be available in codeless tools or might be broken/rudimentary.

Some points to think about -

  • How will you refactor repeated stuff in codeless tools with few clicks?
  • How will you automate checking for bad practices in your codeless tests?
  • If codeless tool produces code/tests in proprietary/hard to work with format, then how will you migrate away from that tool?
  • What if tool company sinks or takes several missteps that have material impact on you?

My uneducated opinion on AI based tools - Use them to assist you in your work, but don’t let them become the mainstay of your testing needs/work. AI is not very intelligent and it might be a while before we can be confident in it.

Go to stack overflow or the company’s tool forum to see the kind of problems (if any) that users face. See the resolution. Look at any feature request boards of their tools. You might find plenty of red flags there, enough to convince you to avoid the tool.

If the tool seems ok to try, then do a project with a small team, i.e. a proof of concept, to see how it works before you invest in it.

2 Likes

This is an excellent topic.

First of all, I am a developer, not a tester.

A few years ago, I would have avoided codeless test automation tools.
The situation is different now, there are a few good options out there.

Some of them seem to offer the same flexibility as writing code, since you can have variables, if statements, loops and reusable functions.

My colleagues who are in charge of testing are using Endtest.

Looking at the pricing page, it seems their standard PRO Plan seems to be more affordable than TestSigma.

Even Microsoft is using that tool, you can read the interview here.

I would do POCs with a few tools and avoid the Sales Reps.

Good luck!

2 Likes

Hi Melissa,

Yes, yes I am.

And I have been following the trend for a while now. I’m using Leapwork, but also looking into Testim and Mabl, and Test Project.@bethtesterleeds has an intro to test project here:

But it is part of a larger trend. I mockingly compare it to Zapier og If-This-Then-That (IFTTT) and similar “home automation” that visually allow you to tie things together… with the moniker “no code” / “low-code”. there’s a discussion of the trend here: https://community.wardleymaps.com/t/low-code-evolves-and-outmaneuvers-high-code/697/8

wrt testing and in reply to @han_toan_lim and @anon68517856 : Lowcode makes most sense to me, when you don’t have access to the source code. This is especially true for standard applications in the enterprise, they only have the SAP or Dynamics GUI to interact with to test the specific customizations.

If most of the testing is currently done by SME’s, then lowcode is a more user friendly etc. In the end though, low-code is still code, and needs to be maintained as such.

Hope it helps, else reach out. Also there’s more on my blog :wink:

/Jesper

2 Likes

Thankyou for the mention and the follow @jesper :ninja_navy:

1 Like

Thanks everyone. Still digesting your comments! I’m going to start having more of a play from tomorrow, so I’ll report back!

1 Like

I’m curious about this too.

Currently, I’m testing a React app using Mocha Chai w/ Webdriver to do E2E UI test and experiencing the single selector fragility inherent in Selenium/WebdriverIO testing.

Tools like Mabl & Functionize promise to give me recorded repeatable tests that - through multi-selector capture and machine learning - won’t break. However, the general response I get to suggesting POCing these tools is derision (example response “It sounds like GameSalad”). ¯_(ツ)_/¯

The implication is that serious E2E testing can’t be achieved through recorded low/no-code tests. While it was certainly true once, with AI computing advances and these SaaS products finding traction with major companies: Is it still true? Or is the perception a couple steps behind a new way of getting the monotony of UI testing done?

1 Like

Hi @fletcher

yeah. I get that too. So options: either improve the testability of the system under test and continue with Mocha / Webdriver, OR solve it some other way. Or don’t solve it at all, if it’s not important enough for a small spike / POC.

It does require courage and imagination to consider that maybe the assumptions of the past doesn’t hold because we now have better tools. I can name 4-5 tools in this space already, so there must be something too it.

(oh, and btw - lowcode tooling goes beyond test automation). Are integration tools like mulesoft equally bad?

/Jesper

2 Likes

Last week a special section has been opened for annual partners of Minsitry of Testing. Mabl is one of them. You could pose questions over there:

Let me rephrase your problem in my own words.
You tried to automate GUI test of a React application. Uptil now you ran twice into a single selector problem. What you are looking for, is the right tool(s) which can solve this problem. At the moment you are looking at an AI solution.

My gut feeling is to consider Cypress. I once had a small introduction and it looked good. In the JavaScript front end automation it is frequently used.

You could even follow the free introduction course on Test Automation University.

2 Likes

Hello,

This is a very wast topic to discuss here,

But As a Software Tester, I would like to suggest to you below listed codeless Test Automation Tools, So you can go through them and shortlist them as per your needs and requirements.

  1. Katalon Studio
  2. TestCraft
  3. Perfecto
  4. CloudQA
  5. Sikuli
  6. Mabl
  7. Applitools
1 Like

Greetings!

In my experience, codeless is implemented in an abstraction layer as detailed by @anon68517856. In my opinion, it was still necessary to have an understanding of programming. That is, I still need to understand decision statements, assignment, iteration, and other programming concepts (concepts that are integral to any coding language) to be effective with these tools.

@jesper mentioned the MuleSoft tool (called Anypoint) which uses a drag and drop interface to construct workflows. Despite the drag and drop method of constructing the workflow, there was still a need to design them which relied on programming concepts. Recovering from errors was particularly painful. One redeeming quality is its built-in unit testing framework.

Lastly, another consideration may be the actual implementation layer. If I recall correctly, the Test Sigma test resides in the cloud.

  • Are there security or access concerns around artifacts residing in the cloud?
  • Is it possible to get copies of cloud artifacts that could run locally if the vendor goes out of business?
  • Can tests be scheduled and test results sent securely to your team?

Joe

2 Likes