Hey all!
Wanted to understand what’s the best way to go about automating Web testing.
Do you see value in paid codeless automation tools like MABL, Eggplant, Testim etc? Or is Selenium good enough?
Have you used any of these paid tools? Which one is the best?
Well, I haven’t personally used any of them but I’m guessing they offer a layer of abstraction over selenium, which I’m sure they use in the background.
Basically, if your testers don’t code, it might be an useful resource. If they do, there are also open source frameworks like webdriver.io so you don’t need to set one up from zero.
Hello @adityasg13 and Welcome!
I am also considering paid codeless automation tools. In my context, these tools offer a method to have testers start testing sooner, and I would not have the overhead of maintaining automation that I build.
If I were to build a similar solution, I may not have a product ready in time to support the project, and the maintenance would distract me from exploring other opportunities to help
testers and testing.
I’m also interested to hear experiences with these tools.
Joe
I wonder if you would still have the same levels of flexibility? Or at least, enough flexibility that you won’t feel like you need multiple paid suites to achieve what you want.
I do appreciate how much time it takes to maintain selenium scripts though. You can easily fall behind if you have other responsibilities.
Great question, @froberts!
In my opinion, it depends on the application under test.
If the application is one built and maintained by an organization (such as a website), more flexibility may be required. I assume testers have access to all parts of the application. In that case, they can decide the types of testing and automation to apply. I might follow the testing pyramid an have lots of unit tests that explore business behavior.
Moving up the pyramid, some manual or automated tests can explore the UI as well as security, configuration, and connectivity. If the UI is complex, codeless automation products would make less sense to me.
If the application is provided by a vendor (as in my case), I may not need much flexibility. In this case, no one has access to the code of the application. The application has defined workflows that multiple clients use; we have only to configure the application for our specific use.
I would welcome a codeless automation product that allows our testing team to start “testing” immediately. There is not behavior to evaluate (behavior is already tested by the vendor) rather we need to confirm the configurations. Hopefully, those configurations exhibit some evidence through the UI which testers can confirm.
I’m at the start of this journey and today, I believe I will use a combination of codeless automation and other techniques for evaluation in my context.
The more I wrote about this, the more dimensions I started to consider (a good thing!). I hope this provides a peek into how I’m viewing this build vs buy decision.
Joe
Hi @adityasg13,
Firstly, welcome to the Club!
There are no best tools, just the right tool for you and your team. @devtotest has touched on this a bit regarding the context of your product will contribute to which tools to use (although I would raise caution about the testing pyramid - a conversation for another post perhaps @devtotest?).
To identify your tooling you need to look at the testability of your application. How it’s built, what technologies it uses, where it is deployed, etc. all make a difference to the tools you can use. For example if you have well-formed Web APIs, perhaps look at some HTTP testing tools. You can also look at your teams context as well. Automation is a team ownership and not an individuals, so who has experience of automation on your team? What languages do they know? Do you work in a team where individuals are willing to share or pair on problems? All of these questions and more will have an effect as well.
So spend a little time learning more about your context. You may find opportunities you wouldn’t have found if you dive straight into the tools.
In regards to the whole paid vs. open source thing ask yourself. How much is it going to cost me and my team to learn and implement open source tool X? Or how much is it going to cost to hire someone who has the skills we require? I’m not suggesting that money is more important than spending time learning new tools and approaches. But if you see that the cost of a license for a paid tool that is designed to have an easy learning curve and has professional support is less than the cost to your team of going it alone. Then perhaps a paid tool is the right thing for you. You can still use professional tools and learn other open source tools on the side.
Remember, your project and product context will give you the answers you need. Not the features or pricing of a tool.
I agree with most of the comments here. “Context” is king. I know you mentioned you wanted to do web testing, so some things to consider when choosing a good tool/platform would be-
Once you have answered some of the above questions, then you will get a better picture of what you are looking for.
Selenium has been around for a while and is one of the most popular open source testing tools out there. I myself have used Selenium for 8 years and it does have some great features. The reason people are still using Selenium is-
That being said I think it does have its own limitations such as
NOTE: Selenium was built by people who care about testing and they actually help to maintain/update it outside their actual jobs. So, we cannot complain if Selenium has problems as people are doing their best to maintain it. In fact, a lot of codeless platforms have some components of Selenium built into it.
To avoid some of the limitations of selenium, people may look into codeless automation platforms. And again, they do have their own advantages and disadvantages just like selenium.
The biggest issue with record and playback tools is, it is not extensible like selenium and you are confined by the features provided by the tool and if you want to do something extra the tool does not provide; your hands are tied. At the same time it does help to get everyone involved in automation including non-technical people who have really good testing and domain knowledge.
I personally work at Testim and have been helping to make our codeless testing platform better to overcome the above limitations. You can read about how we make the platform extensible here - https://blog.testim.io/why-testim/.
I have also personally used MABL, Functionize and they are good in their own ways.
In summary, it all boils down to what you want to do? what kind of application you are testing? and how much you are ready to invest in UI automation? 
Just a note:
Free, in software, generally means the same as in free speech, not as in free beer.
Hi Aditya, there are many comments and I will be repeating a lot of what has already been said. So in short my advice is you’ll need to do an evaluation exercise and Proof of Concept (POC) exercise first.
Not every project, product, application, team or situation is the same. So you will not be able to get an answer this question in regards to ‘Which one is the best?’ or ‘Do you see value in paid codeless automation tools’ you will need to answer this yourself though a period of evaluation.
As someone on here could say ‘oh yes XX tool was brilliant!’ yet you may find out that it doesn’t support Mobile web, cross browser testing, or you’re unable to run tests in parallel, the reporting isn’t clear, it’s only available on Mac OS and everyone is on Windows in your team or visa versa, as all possible examples. Then you need to consider budget, ongoing costs, maintainability, support, life of the product (how long do you intend to use it for, 1 year, 2 years?) what is the return on the investment you make.
Have a clear goals e.g, ‘I want to prove that we can automate our product using this tool’.
With many paid tools you should be able to use it for free for a set time in order to be able to reach conclusions on where you go.
Others have mentioned the pain of maintaining selenium, it maybe how you are working as team rather than the tools that is the issue. Are you sizing and estimating correctly? Is your definition of done factoring in automation? etc… (although if you have a framework that is difficult to maintain then that needs addressing also).
Hope that helps.
Georgina
I agree with a lot of what has already been said that “context is key,” but I also think it’s important to understand the team that will be managing your web test automation efforts. At TestCraft (the company where I work), we have customers who invested in codeless automation specifically because the team they wanted to manage their testing didn’t have enough coding skills to use Selenium.
Understanding your use case will obviously help you out as well. If you’re already using Selenium, and you see that you are spending too much time on test maintenance or modification, then a paid codeless automation tool might save you a lot of time and resources. It’s a matter of knowing your team’s pain points, determining if a paid tool will resolve those issues, and finding the right tools that will give you the most benefit for cost.
A lot has been said already, but I feel selenium has a lot of potential to automate anything and hence it is widely used for web application, mobile application and even for the api performance testing.
Selenium has a wide reach and users worldwide, hence there is always easy support available.
Unless you have an application on a different platform which selenium cannot automate, then we have no option but to switch to other applications. But if we talk on the common ground, then in general there is nothing that selenium won’t do that other paid apps can.
However, there can be a few drawbacks of selenium :
I can explain somewhat how we went about procuring our tool Leapwork some 3 years ago. We had a problem around the maintenance cost of the Selenium scripting that we have, the same problem is happening with our scripting in ServiceNow where releases and changes break automation.
We looked to solve the former problem by looking at codeless tools. Now 3 years ago the market was different, it wasn’t around self healing and AI, it was about being ‘better’ than Selenium. I’m not buying in to that as a reason to go for a tool, as said, it’s about context.
We drew up our criteria and looked at what the market offered and what we needed. Then we invited a number of vendors to tender. Mainly the criteria asked for was:
Most importantly easy to use. The intent was to push automation out to users to accomplish tasks.
The tool we have is a Selenium abstraction and used in the right way is very powerful and if our guidance is followed you don’t duplicate functionality. It was purchased to automate the regression for a HR system and we produced about 170 flows for their regression pack.
We’ve used it to great success and it allows us to pivot and produce automation quickly, for example we had to change some data due to Covid, there were some 2000 records and manually it would take 5 days at 8 hours a day. As a side effect of it being easy to use we started to use it for RPA and remove manual processing. The automation was laid out and constructed in 30 minutes, within an hour I was processing information. All told it took 10 hours to run, and took a day and a half to develop and test.
We mostly use it to automate corporate systems such as Oracle, but we do also use it for RPA and smaller automation projects. I use it to fill in my time sheet on our terrible slow system using an excel sheet.
The flows use blocks of functionality such as Find Web Element and Click Web Element. You then ‘wrap’ these within sub-flows that are either reusable, or as in the screen shot ‘blocks of functionality’, this allows you to read the automation like a flow chart.
Within the first block ‘Check if Timesheet exists’ the logic looks like this. Remember the system is slow. I first look to see if I’ve already created the timesheet by looking to see if the modal is in the viewport. If it is because it positioned at 0,0 then it’s on screen and I need to make a new timesheet.
If I didn’t find it at all then I look to see if I can see the timesheet entry page. Again, if it isn’t at 0,0 then it’s on screen and I stop looking.
I’ve been running this same automation for a couple of years now and I’ve had to change it twice, once because I wasn’t accounting for Bank Holidays correctly and another when some of the elements changed in the new version. All in all it’s maybe 3 hours of work that saves me 30 minutes a week - it really is bad, there’s a reason why the loop lasts for up to 60 seconds.
