This is a case study from quite a few years ago but it stood out for me as a really interesting application of AI in Testing.
What attracted me to this and prompted me to share with you all today is that the approach solves a real testing problem “How do we test 100s of levels of a game for each release where each level may have multiple solutions”. The presentation is by a Testing Lead at King, who produces games such as CandyCrush which has thousands of levels.
The other interesting aspect of this talk is that the type of AI techniques being used is far removed from the current hype cycle of using Generative AI to solve all our testing problems.
They started using Search Trees -which many people don’t even realize is considered part of AI – and while successful found that the time taken to play levels was considerable. They evolved this into bots that were trained using a combination of reinforcement learning and genetic algorithms to create more intelligent bots that “play” the levels.
A challenge with these types of approaches is that we can rarely rely upon explicit expected/anticipated outcomes to determine if a test has “passed”. So as testers, we need to consider how we express our expectations of the outcomes to determine if there is a problem worth exploring. One of the use cases within King seems to be training different types of bots (e.g. a bot to finish levels quickly or maximize score) to explore how the application (or a level) behaves under different types of play and whether it crashes. This is a fairly clear signal that something unexpected has happened but how would you tell if a level is solvable and the level of difficulty is appropriate? Deciding on criteria for the bots for such test ideas is more complex and nuanced…but an interesting challenge for testers!
Another reason i chose this was that in my career, I’ve had many similar challenges around difficult testing challenges (although not in the games industry) and opted for more interesting approaches like this to solve the problem at scale. Often, I start by looking at how we can use search algorithms to explore some problem state efficiently but shifting towards integrating learning bots into the process is interesting in my opinion, and opens up some interesting ways to explore complex applications.
I guess a challenge for test teams is that building these types of bots is not trivial and requires a fair amount of specialist knowledge about AI/ML. At King, they had a team of data scientists working on the problem.
As an aside, last year while working with a client the question of a Data Scientist in Test role was discussed to bring data scientists into the testing teams to support their testing development of bots, synthetic data generation and inject autonomous decision making into testing processes processing (such as automatic triaging of defects or prioritization of tests) to reduce the level of toil for testers (toil being fairly routine work that detracts from more value-adding work such as exploratory testing). I for one, would find such work interesting 