I saw someone recently asking for advice on tooling for visual automation testing and one of the follow-up replies sparked an interesting question that I thought should be asked here on The Club.
Why do you do visual automation testing? What problem are you trying to solve or what goal are you trying to achieve?
For me visual automation testing is a fast way to check whether an application works by comparing a baseline with a snapshot.
My major problem is to reduce the actions to execute the tests. For visual testing you do not need to determine the xpath of id of a web element and find the proper name of the attribute. I can just make a picture and use this as a baseline.
An advantage is that one screen picture can capture everything on the screen with one command. The scope is bigger than a check of a text attribute of a web element. The scope of visual testing is bigger than a standard single value attribute check.
There is a tool which provides human interaction to determine the final test result. Also filtering or leaving out parts of the screen is an option. E.g. date and time.
My goal with visual automation testing is to provide faster feedback than non visual automation testing.
The short answer whether I replace E2E testing (End to End testing) with visual testing is no.
Let me use the imaginary VIP movie app, which lets the user order a movie ticket, snack, and drank in advance. A bar code provides me the ordered items.
The happy path is simple: at one end I am ordering and at the other end I am enjoying the movie while consuming my snack and drink.
If I only use visual testing, then I can only test the ordering. So I could call this end to midpoint testing. There are still interactions of me with the bar code reader at the door of the cinema and the bar code reader of the cinema shop to be tested.
Let me complicate the happy path. Suppose I order potato chips, but they are not in stock. So they have to be ordered. There is small detour via the supplier. Maybe I could automate the inventory and ordering system of the cinema. But the order processing system of the supplier is not likely to be automated.
Time for a unhappy path. So the potato chips cannot be provided at the time of my cinema visit. Then I will have to test manual procedures. What will be offered to me? How will cinema employees help me? Do they help me the same way?
I’d tell you why we’re not doing any UI automation checks(visual or functional - we tried both).
We came to the conclusion(me, PO, and lead-senior devs) that it’s too expensive for what we get as a development team, and also what the business gets in return.
We decided we could get for the same budget an extra developer which would code extra features that would bring more user satisfaction and money to the company.
The UI is not stable enough for non-flaky checks, too many rules/exceptions to build plus several API mocks were required, viewport scenario exceptions to maintain, scenarios needed constant maintenance as feature change quite often, the testing that we’re doing covers our risky stuff and lots of problems which we can’t even get to fix, the impact of a possible problem is not significant.
I liked the idea of visual checks - I wish I could manage such a strategy & framework sometimes. But you need the right context to implement it.
For the moment we stick with unit checks & logging & monitoring for external services, Middleware, and soon for frontend http requests