Hey all!
Wanted to understand what’s the best way to go about automating Web testing.
Do you see value in paid codeless automation tools like MABL, Eggplant, Testim etc? Or is Selenium good enough?
Have you used any of these paid tools? Which one is the best?
Well, I haven’t personally used any of them but I’m guessing they offer a layer of abstraction over selenium, which I’m sure they use in the background.
Basically, if your testers don’t code, it might be an useful resource. If they do, there are also open source frameworks like webdriver.io so you don’t need to set one up from zero.
Hello @adityasg13 and Welcome!
I am also considering paid codeless automation tools. In my context, these tools offer a method to have testers start testing sooner, and I would not have the overhead of maintaining automation that I build.
If I were to build a similar solution, I may not have a product ready in time to support the project, and the maintenance would distract me from exploring other opportunities to help
testers and testing.
I’m also interested to hear experiences with these tools.
Joe
I wonder if you would still have the same levels of flexibility? Or at least, enough flexibility that you won’t feel like you need multiple paid suites to achieve what you want.
I do appreciate how much time it takes to maintain selenium scripts though. You can easily fall behind if you have other responsibilities.
Great question, @froberts!
In my opinion, it depends on the application under test.
If the application is one built and maintained by an organization (such as a website), more flexibility may be required. I assume testers have access to all parts of the application. In that case, they can decide the types of testing and automation to apply. I might follow the testing pyramid an have lots of unit tests that explore business behavior.
Moving up the pyramid, some manual or automated tests can explore the UI as well as security, configuration, and connectivity. If the UI is complex, codeless automation products would make less sense to me.
If the application is provided by a vendor (as in my case), I may not need much flexibility. In this case, no one has access to the code of the application. The application has defined workflows that multiple clients use; we have only to configure the application for our specific use.
I would welcome a codeless automation product that allows our testing team to start “testing” immediately. There is not behavior to evaluate (behavior is already tested by the vendor) rather we need to confirm the configurations. Hopefully, those configurations exhibit some evidence through the UI which testers can confirm.
I’m at the start of this journey and today, I believe I will use a combination of codeless automation and other techniques for evaluation in my context.
The more I wrote about this, the more dimensions I started to consider (a good thing!). I hope this provides a peek into how I’m viewing this build vs buy decision.
Joe
Hi @adityasg13,
Firstly, welcome to the Club!
There are no best tools, just the right tool for you and your team. @devtotest has touched on this a bit regarding the context of your product will contribute to which tools to use (although I would raise caution about the testing pyramid - a conversation for another post perhaps @devtotest?).
To identify your tooling you need to look at the testability of your application. How it’s built, what technologies it uses, where it is deployed, etc. all make a difference to the tools you can use. For example if you have well-formed Web APIs, perhaps look at some HTTP testing tools. You can also look at your teams context as well. Automation is a team ownership and not an individuals, so who has experience of automation on your team? What languages do they know? Do you work in a team where individuals are willing to share or pair on problems? All of these questions and more will have an effect as well.
So spend a little time learning more about your context. You may find opportunities you wouldn’t have found if you dive straight into the tools.
In regards to the whole paid vs. open source thing ask yourself. How much is it going to cost me and my team to learn and implement open source tool X? Or how much is it going to cost to hire someone who has the skills we require? I’m not suggesting that money is more important than spending time learning new tools and approaches. But if you see that the cost of a license for a paid tool that is designed to have an easy learning curve and has professional support is less than the cost to your team of going it alone. Then perhaps a paid tool is the right thing for you. You can still use professional tools and learn other open source tools on the side.
Remember, your project and product context will give you the answers you need. Not the features or pricing of a tool.
I agree with most of the comments here. “Context” is king. I know you mentioned you wanted to do web testing, so some things to consider when choosing a good tool/platform would be-
Once you have answered some of the above questions, then you will get a better picture of what you are looking for.
Selenium has been around for a while and is one of the most popular open source testing tools out there. I myself have used Selenium for 8 years and it does have some great features. The reason people are still using Selenium is-
That being said I think it does have its own limitations such as
NOTE: Selenium was built by people who care about testing and they actually help to maintain/update it outside their actual jobs. So, we cannot complain if Selenium has problems as people are doing their best to maintain it. In fact, a lot of codeless platforms have some components of Selenium built into it.
To avoid some of the limitations of selenium, people may look into codeless automation platforms. And again, they do have their own advantages and disadvantages just like selenium.
The biggest issue with record and playback tools is, it is not extensible like selenium and you are confined by the features provided by the tool and if you want to do something extra the tool does not provide; your hands are tied. At the same time it does help to get everyone involved in automation including non-technical people who have really good testing and domain knowledge.
I personally work at Testim and have been helping to make our codeless testing platform better to overcome the above limitations. You can read about how we make the platform extensible here - https://blog.testim.io/why-testim/.
I have also personally used MABL, Functionize and they are good in their own ways.
In summary, it all boils down to what you want to do? what kind of application you are testing? and how much you are ready to invest in UI automation? 
Just a note:
Free, in software, generally means the same as in free speech, not as in free beer.
Hi Aditya, there are many comments and I will be repeating a lot of what has already been said. So in short my advice is you’ll need to do an evaluation exercise and Proof of Concept (POC) exercise first.
Not every project, product, application, team or situation is the same. So you will not be able to get an answer this question in regards to ‘Which one is the best?’ or ‘Do you see value in paid codeless automation tools’ you will need to answer this yourself though a period of evaluation.
As someone on here could say ‘oh yes XX tool was brilliant!’ yet you may find out that it doesn’t support Mobile web, cross browser testing, or you’re unable to run tests in parallel, the reporting isn’t clear, it’s only available on Mac OS and everyone is on Windows in your team or visa versa, as all possible examples. Then you need to consider budget, ongoing costs, maintainability, support, life of the product (how long do you intend to use it for, 1 year, 2 years?) what is the return on the investment you make.
Have a clear goals e.g, ‘I want to prove that we can automate our product using this tool’.
With many paid tools you should be able to use it for free for a set time in order to be able to reach conclusions on where you go.
Others have mentioned the pain of maintaining selenium, it maybe how you are working as team rather than the tools that is the issue. Are you sizing and estimating correctly? Is your definition of done factoring in automation? etc… (although if you have a framework that is difficult to maintain then that needs addressing also).
Hope that helps.
Georgina
I agree with a lot of what has already been said that “context is key,” but I also think it’s important to understand the team that will be managing your web test automation efforts. At TestCraft (the company where I work), we have customers who invested in codeless automation specifically because the team they wanted to manage their testing didn’t have enough coding skills to use Selenium.
Understanding your use case will obviously help you out as well. If you’re already using Selenium, and you see that you are spending too much time on test maintenance or modification, then a paid codeless automation tool might save you a lot of time and resources. It’s a matter of knowing your team’s pain points, determining if a paid tool will resolve those issues, and finding the right tools that will give you the most benefit for cost.