Preamble
First off - apologies for the wall-o-text!
I have started a new job and been tasked with writing UI based automated tests for one of our products. Unfortunately, due to the nature of the work, I cannot discuss any specifics about what the application is or what it does.
Here is a mock application that helps give you an idea of the structure of the app:
Challenges
Insane amounts of windows
The AUT is Java Swing, running on Windows
It consists of a main window, With a drop-down menu (File, MenuA, MenuB etc) along the top. Via this menu, you can launch other windows, which are completely separate windows (Not constrained within the main window, have own taskbar icons). These child windows also often have child windows of their own and this pattern repeats - I have seen 6 levels of depth of children of children.
Some of these windows are also modal.
Many of these windows are very guicontrol rich (often 100+ guicontrols in a window)
So at a guess, we are talking 500+ windows and 10,000+ guicontrols in the application.
Therefore, with this level of complexity, I am leaning very much towards using a “Page Object Model” - ie a class which handles each window, which derives from a base class.
That way, given a window C which is a child of B which is a child of A, I can just call Open() on C, and it knows that if B is not open, it needs to open it before opening C, and if A is not open, it needs to open it before opening B. Otherwise, tests are going to have to be very verbose in detailing how to get to each window in order to test it.
Distributed testing
If that wasn’t enough complication, this is also a networked application
There are clients and servers, and we need to do tests such as performing an action on client A, and verifying that the action permeates through server A, then server B, then finally something happens on client B.
ie we could be having 4 machines involved in one test.
Other complicating factors
- Time is also a factor in the tests. It has a map, things move on the map
- As mentioned there is a visual (Symbols on a map) element which needs to be asserted.
At the very least, pixel comparison will be needed, maybe even some form of AI or machine vision for a little fuzziness in the matching. - Red tape. Lots of it. Any software by any company even remotely connected to Russia, for example, is likely to be verboten.
Requirements
-
Full Code
Simplistic frameworks (No-code / Low-code) are most likely not going to cut it
They simply will not be able to handle the complexity of the task at hand. -
Appropriate programming language
Intellisense is going to be a must - when referencing Page Objects we are going to want Autocomplete etc
Preferences are Java or Python
Due to the Page Object Model requirements, full support for Classes and Inheritance is required -
Appropriate IDE
Solutions using bespoke IDEs are most likely not going to cut it - it’s almost certain that we will run into problems and need to refactor, so unless the bespoke IDE has refactoring tools on a par with the likes of Visual Studio, it’s not gonna cut it. -
UI Spy
We will need a Spy type application for finding the selectors for UI controls.
Bonus points if I, on my dev machine, can Spy the UI on a remote machine via its Test Runner -
Distributed testing
We want there to be one machine that orchestrates the test, and that connects to a Test Runner on each of the 4 other machines (Server A/B, Client A/B) in order to control the GUI of the AUTs running on them. Should be able to execute operations on multiple machines from within one single test. -
Test Runner switching
When developing tests, I may be developing the steps for Client A, (So Client A would be running on my local machine, and Client B and Server A/B would be on remote machines), but then once I have done that bit, I would need to swap it around so that say, Server A was running on my dev machine and all the others were remote.
Therefore, we need to be able to easily switch between local and remote runners -
BDD
We would like to write tests using Gherkin. Preferably Cucumber.io
Reviewed frameworks
I had a sales call / demo with Froglogic about Squish, and did a trial of the other 3 products listed below.
I did quite an extensive search, but discounted most other products that I came across. The industry trend seems very much towards no-code/low-code and/or AI/Machine vision these days. Machine vision is simply not gonna cut it methinks (For the GuiControl / Window manipulation side of it), as it simply will not be able to deal with obscured windows.
Froglogic Squish
- Failed on (3) - Bespoke IDE
- The only solution I know of that passes with flying colours on (4) though (Can Ui Spy remote machine)
Smartbear TestComplete
- Failed on (2) - Does not support fields in classes
- Failed on (3) - Bespoke IDE
- No autocomplete for user-defined objects
- Folder structure in IDE is fake
OpenText UFT Developer
- Non optimal on (6) - Has Grid runners for targeting remote machines, and local runners for targeting local machines. Cannot mix both within one test, so Grid mode the only viable option. However, this causes a fail on (4) as the UI spy does not work when in Grid mode.
Soft fail on (7) - Local runners and grid runners do not share the same interface, so you cannot switch a machine from being local to remote easily. Can be worked around to a certain degree with a wrapper tho.
Smartbear TestLeft
The only thing so far that I have found with zero failures
However, it is somewhat “discontinued” (They launched it, gained no traction, so sort of shelved it - it’s not even listed under products on their web site)
The ask
And so to the point of the post - can anyone recommend anything that I may have missed?
As I mentioned, I did do quite an extensive search, but to be honest, it was pretty hard going - lots of marketing speak but often very few details on how the software worked.