Due to this fact, we have too many variations to cover with test scenarios. With 15+ customers, it grows up to 100+ variations of tests sets.
Actually, you have infinity number of tests. A test done 1 second later is a different test, although it might be the same in all ways that matter to your testing. The point is that you need to have a good think about why you’re doing testing and what matters to you. All testing is sampling. Good testing is good sampling. So look at how you’re doing your sampling. That will help you control your testing based on the time and money you have available (the logistics of your test strategy)
Q1: Any ideas about complexity estimation?
Projects tend to change and warp depending on what happens during the project. You are trying to measure testability, and I have to recommend this to you: Heuristics of Software Testability. This will give you lots to think about in terms of how to deal with the difference between what you know and what you need to know (the epistemic risk gap).
If you want to look at testability risks in a structured way, you can find a set of approaches here Heuristic Risk-Based Testing.
Q2: Is there a reasonable way how to manage such big range of various configuration and therefore reduce nunber of tests?
What would you consider “reasonable”? The easiest way to reduce your testing is to do no testing. I’m guessing you’d consider that unreasonable because you’d be lacking the information you need. You could change nothing, but I’m guessing you’d consider that unreasonable because your testing is too expensive.
So you could reduce the problem by reducing the information you need. This is improving the epistemic testability of the product by stating that you do not need to know certain things. Make a cost-benefit decision about dropping tests.
You could reduce the problem by improving the way you test. If you have sets of test cases, for example, you could change to a list of the things you need to know about and allow your testers to explore those things freely. They are sometimes called scenarios or test charters, if you’re searching for them. This has many advantages in that you’re more likely to find problems because your test tasks are less prescriptive, testers are more engaged with the system and learn and explore better, and testing tends to be done faster because the paperwork is much lighter. If you need that paperwork then I’d point you towards session-based test management.
You could test based on better information. Knowing the nature of bugs that tend to escape to production, which parts of the product are business-critical, what customers actually want from the product, what parts of the product are are more complex, where your interfaces are, etc. This will lead you to perform better sampling on your product.
Fixing bugs helps testing go faster. Finding bugs is time-expensive for a tester, because they have to invest time in investigating and reporting it that they could be spending testing. It also affects testability because bugs hide other bugs and create unknowns. Because finding bugs is time-expensive moving testing up front is a great way to improve this problem and has massive impact in other ways by reducing feedback loops. Finding and fixing a problem during programming in a pairing session is MUCH cheaper than finding it during a testing session later on.
Look at issues in your test project. A bug is anything that threatens the value of the product. An issue is anything that threatens the value of the testing (or project). “Cancel does not close dialog box” is a bug, while “I can’t access the test environment” is an issue. A tester needing information or resources is an issue, disruptions to the pipeline is an issue, poor availability of test builds is an issue. Ensure testers feel that they can (and do) improve these problems, including making requests from others to do so.
When you’ve done all of that there are ways to reduce time faster than reducing coverage using combinatorics. All combinations of OS, LCD, Printer are too many, so you create tests that pair each value in one list with each value in all others at least once. This is called “all-pairs” or “pairwise” testing, and you’ll need a tool for it. There’s a free one called allpairs.
I’m trying to sum up, here, how I manage testing and develop testing strategies which has been a lifetime of learning for me, but hopefully it’s of some help anyway.