Does anyone know of the best approach or strategy to automate a highly configurable product. The product has been designed to be configurable, which clients then configure and the number of combinations results in different UI layouts and workflows.
I would like to automate the most important flows, but even that will result in different scripts that will become difficult to maintain. What would be the best approach to creation of automation strategy
This is one of the best examples to automate in the API level. Can you get any sort of coverage there? See what you can automate before you go to the UI layer, make a gap analysis and then ad a few UI test.
I would with
Hello @oorupabo!
For complex products like what you describe, I recommend isolating behaviors to improve coverage and testability. In this case, at a high level I recommend separating the ability to configure the product from the presentation and behavior on the screen.
Configuration
Explore the implementation to learn how the configuration is implemented. While there may be a UI to select configurations, those configurations must be maintained somewhere.
One method is to store the configurations in a database and expose the ability to change them through an API. With the API, you would be able to verify that configurations can be stored, changed, and deleted. Additionally, you could evaluate multiple combinations of configurations and their validity.
I recommend investing the majority of your automation here.
UI
I recommend learning about the business purpose of the configurations and how it influences the UI content and workflow. This information could be the basis of what you might evaluate. Most importantly, in my opinion, is to demonstrate that the content and workflow are driven from the configurations. I recommend starting with something simple that shows this connection and move towards something more complex. You may also have to demonstrate the UI can be used to change configurations.
Once that is demonstrated, you might add a few more test cases to explore diversity of configurations.
This phase of the testing may not benefit from automation; it will depend on risk and pace of change of the product.
Joe
A strategy we used for testing in a highly configurable product was similar to “Personas”.
This is assuming that the configuration dramatically changes the the flows and views in your UI to the point where any automation for one will not be similar to another configuration.
Describe different client cases and setup a configuration that match that Persona. Then create automation for each of these personas. I.e.
“Super-agile-tech-savvy-client”: It is all about minimizing the number of clicks for the user to reach the desired action, they regularly changes their flows to match their new business need and KISS (Keep it simple stupid) rules everything else.
“Highly customized dinosaur”: Every view is full of customized field that only they use and understand. Any change in the product will go through 10 iterations of decision-making before they will hit the product. This also means that the majority of the users still uses IE 11 on Windows XP.
“Don’t touch what is not broken”: They are still running on the default initial setup that was done for them 3 years ago. This means that unless forced to change the product they will not see or use new features.
In terms of coverage they should complement each other as one is using default configuration, and one is never using default configuration. One is highly optimized and removes everything they don’t want and one want’s it all… and so on.
By doing this we drastically reduced the number of variant while still having a good enough coverage and also having a very known state on the system under test. With this we added targeted manual tests for very specific customization and every issue reported from customers had to go through the question. Do we change any of the configurations with this issue? (Most often we did not need to)
Good Luck!
Yes, I’m currently considering this approach. It will avoid the overhead of maintenance.