Anyone Getting Value From Visual Test Tools?

šŸ™‹ Questions
,
1

Been using Applitools for 2 years for ecomm site testing. Iā€™m just not seeing the value. Itā€™s expensive. And itā€™s not finding near the bugs Selenium is finding. Maybe my siteā€™s bugs are more related to user journeys. Iā€™m not sure.

Just wondering if visual testing is more hype than practical, in your experience?

2 Likes
2

The main problem I find with it is its sensitivity to flakiness is very high. If there is nondeterminism somewhere in your stack it will give you tons of false positives.

Making the test environment entirely hermetic is a prerequisite to making snapshot testing work but even that sometimes isnt enough.

1 Like
3

I felt the same way when I looked at model-based testing tools in the past, for example. Itā€™s no secret that Iā€™ve been using Sahi Pro for many years (and I donā€™t get paid for it, I like it) and Tyto, the manufacturer, has now added flowcharts as an additional feature to v11. You should have an unbiased look at it, because you can visually display many variants that you canā€™t create or see so quickly at a glance with the scripted or keyword approach. It is easier to implement nagative test scenarios in the visual part without having to repeat the duplicate part. Very convenient. Just have a look.

1 Like
5

Thanks for sharing this. It looks very interesting. I wonder if it works with WinUI3 with Uno Platform.

More questions:

  • What type of application do you test with Sahi Pro?
  • How fast and precise is it when running the test?
  • Any problems with running repetitive test and how easy to solve them?
6
  • Web-applications ā€˜old fashionedā€™ and modern with ā€˜web componentsā€™-technology and Angular ā€˜flavoursā€™ as well as mobile hybrid application on Android/iOS
  • Sahi makes no difference between the browsers. You can develop the tests under a Chromium-based browser and execute it with Firefox-based browsers. If the developers have made differences, you will find them :slight_smile: We have running some Runners to spread the executions through many runtimes as needed. In the pipeline a normal single-run 30 min. release test needs <8 minutes with 3 Runners.
  • Through the business-driven test automation concept we have three layers: The implementation layer, the accessor repository layer and the business layer. If there are controls changed, mostly you need only to change one entry in the accessor repository layer, except if the behaviour is also been changed - but then you want to really a change. We have exchanged in a project the date component which is used in each workflow and there was only one function to change. It depends on your structure of your test framework. If you follow the common tips to do test automation, you have less maintanance costs than I have seen with other tools.

Hope that answers your questions? You can do the principles also with other test tools but I havenā€™t seen so much which have understood the needs of a testerā€™s test automation approach.

We started with visual tools :slight_smile:: I use Suite Analysis very often because it visualizes me which part of test function I would need to fix, to fix additionally other test steps or test cases,

And we have just started to use the visual component ā€˜Flowchartsā€™ as an additional possibility to design test casesā€¦

7

What about Browser stacks Percy? Have you tried this. We use applit tools too.

8

@mahatheed, what are your experiences with Applitools? How often is it catching bugs you didnā€™t find elsewhere? What is your process, do testers check it daily? How many different mobile platforms do you render the checkpoints with?