As a former recruiter and someone who has recently been through a number of interviews, I have some opinions about this…
- The role / seniority of the person you’re looking to hire should definitely be a factor in the kind of interview you conduct - generally, we expect senior colleagues to have more experience and solid knowledge than junior ones
- Understanding how the person thinks and approaches testing is more important than the “results” of the task, or if they can put a name to the techniques they use, in my opinion
Having said that, I also think it’s important to ensure that candidates in senior roles have a good foundation of knowledge and understanding to be able to guide and lead junior colleagues, even if you decide that the senior person doesn’t need to be a hard-core tester (perhaps in more of a management or consulting role).
In a recent interview I attended, they didn’t ask me to complete any tasks at all, but I’d viewed their site and tested the service beforehand and found several bugs / issues that I discussed with them in the interview. I also downloaded the app on mobile and read many reviews on the Play Store. They shared that they were unaware of at least two of the bugs I raised, which I hoped would show them what I would add to the team in a practical situation; what risks I might uncover that others hadn’t. An advantage of not asking me to complete any tasks is that I could show them how I took the initiative and what steps I took during my own, self-created, task.
Another interview I attended covered a lot more. They asked me to test their application on site (it involved exchanging money, so I didn’t look at it beforehand) and I made sure to “think out loud” throughout - something I’m really big on when it comes to understanding how someone tests. Again, I identified a number of issues they had not (including a potential security issue) and they also got to see me work out what different things were on the actual system I’d be testing, as opposed to something made up and potentially less relevant. They also asked me to name a few testers I admire, which I really liked. This lets you know if a candidate is interested in testing outside of their immediate working environment, in my opinion, and potentially some of the ideas they subscribe to.
To try and answer your questions, Steve, I would suggest that you still have testers test a user story (maybe do this as a three amigos exercise, which I did at another interview (I get around :P)) as well as actual software, as these are different skills and some testers don’t even think of testing beyond the SUT (shift left is still unheard of to some people). I would always recommend the “think out loud” technique, and so would discourage tasks that candidates complete at home or by themselves / in silence.
In the user story test, it sounds like you’re doing the right thing in terms of amount of information and “holes”. In the software test, I would give them as little information as possible; probably just the context of the page they’re on / part of the workflow they’re starting in, or to let them know not to click a particular button that goes to the next stage so as to keep it to a specific page / area / feature. For me, personally, I’d look for someone who doesn’t need requirements to start testing, which is why I’d keep information to minimum. Of course, they can still ask questions, but they might not realise that, so I’d look out for how they react when they don’t know about something. Chris has mentioned some great questions he asks during this kind of task.
As to how long they get, I wouldn’t leave them in a room alone to “complete” the task. For me, it’s not about “finishing” it (when is testing ever finished?), so I’d just block out time in the interview for the exercise and get a feeling of when I have enough information to move on to something else.
This response is longer than I intended… I think you’ve inspired me to write a blog about various testing interview techniques I hope this was useful to you in some way. Feel free to follow up with questions.