Michael is starting by explaining the things that all roleplaying games have in common: they’re weird. Adults sit around a table and invent a story?!
But how do they get tested? The material can be creatively interpreted. People can abandon rules. It’s not a fixed experience like a book. And unlike a book or a film, the people potentially experiencing the fun are also the people responsible for creating it. Socially, it can be great: if you enjoyed it and the people you played it with – then you will think the game was good. Or, if you didn’t like the social context, you might think the game is bad.
It’s key to know who the audience is (although we do need to ask whether the audience know what they want… obviously this is never a problem!!). People’s interpretation is so wide. Some people are happy to be immersed in a world (and don’t mind not talking to others), some prefer to follow rules, some prefer to play more freely. And, in the spirit of humans being bad at self reporting – how can we know what people want, when these people are basically lying to us?
And then we get to the point of fun. All things can be correct, but is the game fun?
Michael is now describing how we deal with ideas. We write it down and decide it’s shit. We adapt it and decide it’s still shit (his words, not mine!). After a few iterations, you show it to a friend, who either tells you it’s shit (especially in some cultures!) or they lie to you to save your feelings. And at some point you take the game to a group to test play. And you realise… ok, you get the point.
Basically, it’s hard to test art. Trying out chase rules, combat system – it’s difficult to know when we’ve got it right. It’s also frustrating because there are no deadlines. People will give you feedback at times when you don’t expect it. In an adventure, you can’t simply make people start to fight. What most often happens is that people get together for a game. If they had fun, that’s a good sign. But as feedback loops go, it’s not a particularly good one.
One approach was to give newbies a role playing game and let them play. They had loads of fun – but played it completely wrong. As a game designer, it’s hard to know what to take from that and where to improve.
In this context, it’s understandable that things can go wrong. Michael’s company released a beta version of a role playing game. It hadn’t been tested. It was “complete”, and was going to be premiered at a conference. Michael took it home, did some testing, and found out that the combat system didn’t work. They’d introduced a skill which basically meant you couldn’t get hit. A giant release had only been seen by a couple of people. And the consequences were great: it worked as a fantastic marketing campaign…for their competitors.
Since then, they’ve changed their approach. They have groups of people who test rules. They test them for consistency within the game, for potential mathematical problems and also for their “feeling”. After all, role playing is a cooperative game. If mages are imba (imbalanced), then everyone will only want to play a mage.
They also use crowd-funding to test the idea of the game as well as initial artefacts. Crowd funders help the quality because they are emotionally and financially invested.
In summary, it’s a tough thing to test since it’s so incredibly variable. Michael says though that the quality over the years has improved – thanks to starting to actually test and because of the crowd-funding approach.