What surprised you?
Just how bad the overall quality of the candidate pool is. Particularly for SDET roles. I would understand if there were many candidates that were either great programmers but with a poor testing mindset, or great testers with poor coding skills, but I’ve interviewed an awful lot of candidates who absolutely stunk at both.
How did you become better at interviewing?
I actually started out as an interviewer with interviewing intern candidates, not necessarily just for testing roles. We had a set script of questions, intended to objectively measure candidates equally, but the more interviews I did the more I realized some candidates did poorly because of our script rather than because of their actual ability. My biggest “aha!” moment was interviewing a candidate with an impressive resume who did quite poorly on pretty much every question I asked. Since at that point I had gotten pretty skeptical of our script, I flipped the script and told him something like “look, I see these various accomplishments on your resume and I can tell you’re smart, but your answers to my scripted questions aren’t very good. Is there a particular CS topic you feel like you’re stronger on that we should talk about?” Sure enough, he suggested an area that we didn’t cover in our script, but I knew enough about to come up with some questions off the cuff, and he did great. Unfortunately, not all stories have a happy ending–I advanced him to the next round but this was in 2020, and all our intern positions were cancelled shortly afterwards due to the pandemic.
But I’ve kept that lesson with me–having an initial script can be helpful, and I constantly look for ways to improve it, but I’m not afraid to go off script particularly if I’m trying to find a candidate’s strengths.
What were your biggest challenges when you first started interviewing testers?
Asking questions that didn’t look for creativity or critical thinking. Plenty of candidates can give you a stock answer they read in a book (or <cough> the ISTQB syllabus), but disappointingly few can think on their feet and go deeper when you push them on context. Which leads into…
What advice would you give to someone new to interviewing testers?
Focus on creativity and critical thinking. I ask several questions that are intentionally badly worded, because I want to see if the candidate will think deeper and challenge me, or just go along with answering the wrong question. For example, in one question I describe a “simple” program and then ask “how many test cases would you need to test this program?”
Good candidates will ignore that specific request for a number and just start brainstorming, quickly realizing that the “simple” program isn’t so simple. Great candidates will start challenging, asking questions, pointing out flaws in the program design/requirements (and I throw some softballs there–it’s amazing how few candidates point out that using a single text field to accept three distinct input values is a really bad idea).
What are some common mistakes people make when interviewing testers?
Well, if the “top 100 interview question” cheat sheets on sites like LinkedIn are any indication, a huge mistake would be focusing too much on arbitrary definitions rather than, as I keep saying, looking for creativity and critical thinking.
Another I would postulate is not digging deeper. I get a lot of candidates that might provide a reasonable initial response, but if I try going deeper or asking the question a different way to see if they think about another aspect, they basically just keep repeating what they said the first time. Even if I specifically ask something like “what else would you consider” or “is there any situation where that wouldn’t be a good idea?”