Hiring processes and emphasis placed towards puzzles and reasoning tests

(To Be Archived) Questions
1

Today in the testers.io slack channel a discussion kicked off regarding the hiring process of many orgs and the inclination towards using some form of logical or reasoning test to determine a candidates viability for a role.

In particular, I raised as a potential candidate, one particular aspect of job hunting I have found frustrating is the amount of orgs which insist on some form of logical reasoning tests or equivalent.

These tend to come in the shape of logic puzzles, riddles or some other variant in my experience, these are “out of the blue” and kept somewhat mysterious until the actual day of arrival for onsite interview - making it incredibly difficult to prepare for. To me, these are activities which must be practiced in advance and in all honestly, I feel that they are of little relevance to the job in question and a poor assessment of ones ability to perform the job in question.

Personally, I am an advocate of assessing ones attitude towards learning and current technical ability within a specific context as opposed to generic tests like I have mentioned above.

In the event you are going for a role in which you know one of these tests are coming - what advise would you give? For example, a logic puzzle or any other type of reasoning test - i.e logical, numeric, verbal etc.

In addition to the above question in particular been keen to hear:

  • How do you feel about hiring processes which involve these types of tests? Do you feel you get better candidates as a results of running candidates through these types of tests?

  • If not, what type of interview process do you believe gets the best candidates?

1 Like
2

I am a self-confessed puzzle nut, and I absolutely loathe these. They’re not in any way a reasonable test of a person’s ability to perform as a tester: all they test is whether someone can solve logic puzzles or riddles under what is one of the highest stress situations a person will face in a lifetime (If I remember correctly, job interviews are somewhere in the top ten biggest stressors - after the usual death, marriage, moving house and so forth, but still high up there).

In addition to that, logic puzzles and the like have a nasty “gotcha” factor like the old series of children’s riddles:

How do you fit an elephant in the fridge? You open the door and push hard.
How do you fit a giraffe in the fridge? You open the door, take the elephant out, and push the giraffe in.

They’re not used to evaluate candidates. They’re used to eliminate people.

I don’t have enough experience to say what type of interviewing process gets the best result, but I strongly prefer interview processes that give the applicants a chance to actually do the job they’re applying for: tests of coding ability for programming positions, tests of testing ability for testers, and evaluate the process that’s used as much as the outcomes.

3 Likes
3

In one job interview, once upon a time, I was asked how I would test a certain functionality of the system I would be testing. This was a good question, even though the answer I gave was dead wrong. But, what I didn’t know at the time, was that I wasn’t being judge on the quality of my answer, but instead how I arrived at the answer. To reach the answer, I asked how the data was set up and input into the system. I asked what users needed from the system. I asked how strict the regulations were that we were testing against. Finally, I wanted to know how they tested in previous releases, as it was existing functionality. After about 10-15 minutes of Q&A, I came up with my incorrect solutions.

This was (according to me) a good interview question since I wasn’t being judged by my answer, but by how I thought about the testing. I only learned later that my then-manager was an experienced tester, and wanted someone who could think critically over someone who knew standard testing processes.

In another job, a portion of the interview process was a set of personality and IQ tests, required for all prospective employees. Both my recruiter and the company’s HR person told me that they didn’t think that the test results said much about who I am, and how I work.

But what the company’s HR person did next was, again, in my opinion, brilliant. She asked about my opinions about the tests and what they meant to me. She explained that the tests were required by the company. She gave me the opportunity to use real life examples about negative results to demonstrate that some of the results weren’t as negative as they appeared in a 1-10 scale. In other words, she didn’t use the test as a metric, but as a tool to steer the conversation. And through this tool, I learned something valuable about the company as well. (That being that the company is focused too much on processes and not enough about people. But the people can adapt the processes to create something worth-while).

But then you have other companies who ask questions without knowing the motivation behind the questions. They would ask how you would test this pen and expect answers about regression testing or usability testing or something else. So the expected answers says to me “I don’t want to work here.”

So, to the point. The questions interviewers ask tell me as much about my prospective employers as my answers tell them about me. The puzzle questions usually tell me that they are not prepared to interview people and gives me the opportunity to use my vast collection of rehearsed answers, hopefully steering the interview back on track.

In the better interviews, they never even have the chance to ask the questions.

6 Likes
4
5

I totally agree with you on this. I actually think these questions are a bit of a fad inherited from undergraduate interview techniques at elite universities. They used to be a staple of Oxbridge interviews (I had a similar question asked at my Cambridge interview in 1998. Also my boss at my first IT job asked me “how many opticians are there in London?” in my interview. I asked him months later about it and he said he got it from his interview at Oxford. He must have answered it well as he completed his degree there!).

I have also heard that they used to be popular with interviews at Google and Microsoft, both companies that used to be (now not so much) very biased towards grads from the elite US universities. I think other companies saw the above doing them and decided that if teaser interview questions were good enough for Google and Microsoft, they should try to copy them.

1 Like
6

Inherited from the “cool kids” actually makes a perverse kind of sense. It’s something of a psychological thing, wanting others to pass the same sort of tests you had to, so I could see elite university graduates using it to psych out candidates when they applied for jobs.

They’re a useless measure at those interviews, too - all they really test for is whether or not the person knows how to answer the kind of trick question being asked.

7

I’ve not used them, nor plan to.

If I had to use them, I’d be more interested in how people respond and cope with the challenge rather than getting it right or wrong.

Sure, knowledge/skills matter to a certain extent, but what matters more to me is the drive/passion/desire to be a positive and effective team member.

8

Not a fan of this approach. Microsoft was famous (or infamous) about using these years ago. While they were really neat brain-twisters, I imagined that working with co-workers would consists of constant “one-upsmanship” - where each co-worker would be trying to out-do the others, all while ignoring the real work that had to be done.

Look at the bottom line: If great puzzle solvers write or test great, then Microsoft’s products should be nearly defect-free. Well, we know the answer to that!

I’ve always thought a good idea would be for the candidate to test an app during the interview. Yes, they would probably need an hour or two. Ideally an app with different types of flaws would exist. UI flaws, usability flaws, incorrect calculation flaws, etc. You could ask the candidate to log bugs in Word or Excel. I specifically call out these because it doesn’t let the candidate know what types of information is good to capture. Will they write a Summary and a desc? Will they include Repro Steps? Severity? Priority? Screen prints, etc. This would also give you a good way to judge their communication skills.

1 Like
9

I’ve been on both sides of these questions. They were asked of me when I interviewed for my current company and those questions remained on the “interview checklist” up through when I started doing the interviews for my company.

Oddly enough, I kept asking the questions even though during my own interview I found them extremely frustrating. I told myself that the questions were “fun” and “showed how the interviewee thought through a problem”. The reality was that most people didn’t really talk through their thought process - they either got the answer immediately or had to be talked through to the answer by the interviewers.

I’ve dropped the questions from the interview process for more recent interviews in favor of questions like “How would you test ______?” and “Tell me about a time when you learned something new.” I’ve found that I care a lot less for the interviewee’s “thought process” and more about how and why they try to learn. I want to know if they ask questions, and if they do anything to improve their testing skills.

2 Likes
10

As to the question "how many opticians are there in London?”, my answer would be “Sufficient to satisfy the needs of people living and/or working in London.”

And that should tell an interviewer what they would need to know about me.

4 Likes
11

As a hiring manager, I think logic puzzles in interviews are quite frankly shit (and yes I love logic puzzles). There are much better things to spend time on. If you want to see how someone thinks, then realistic testing scenarios are much, much better.

I’ve heard folks say they like to use these puzzles in interviews to see how the candidate performs under pressure which begs the question: What kind of horrible workplace have you got that they’re going to be under this type of pressure? Maybe you ought to fix that instead.

2 Likes
12

I have been to an interview that asked me a couple of logical puzzles and didn’t find them a useless measure at all.

For this part of the interview I was paired with a developer, so I deduced that the idea was to see

  • my approach to how I tackle a problem
  • what questions I ask
  • if I am able to learn and apply new knowledge
  • how I relay information verbally and on paper.

He wanted me to talk through what I was thinking, and I felt that it was more of a discussion about the problem that needed solving rather than just getting to the correct answer.

There was no pressure on me, and I quite enjoyed it, feeling that this is similar to the way that teams interact. I’m not a fan of one way interviews that are just questions and answers, so this collaborative approach worked well for me personally.

2 Likes
13

In that case, it’s looking at how your approach a new problem and how you communicate your findings - the puzzle questions I’ve had were “Here’s a list of questions, give us the answers, you have this much time.” which is pretty useless.

So - you got logical puzzles done right. Good on the people doing that interview - I wouldn’t mind seeing more of that.

2 Likes
14

I think these things are tools and it depends on how they are used. A big problem is that people use these tests like exams… but they don’t know how to write good exam questions, and maybe don’t know how hard it is to evaluate someone. Writing a simple methodology for a science experiment is hard enough, let alone evaluating a human.

Secondly people don’t actually know what they want to hear from a test. I’ve sometimes asked questions in an interview about the test and afterwards received feedback that I was aggressive and shouldn’t have questioned the interviewer. This suggests to me a couple of things - that they didn’t know what they wanted from the exercise, that they couldn’t answer questions about their own exercise because they didn’t understand it themselves, and that they have a culture where asking for answers from authority is frowned upon. In a way, it worked - they didn’t want me and I didn’t want them. My biggest piece of advice in using anything for an interview is to understand it, understand the context around it, and know what you’re trying to learn by doing it. If your plan is “give them a puzzle, watch them, then judge them” then you’re wasting your time and theirs. If your plan is “give them a puzzle and add up the marks based on their performance” you’re either a professional assessor or you’re wasting your time and theirs (or both).

Do you know what’s a puzzle and reasoning test at the same time? Software testing. Get candidates to test something and rather than judge them by what they find (after all, who is ever, in real life, expected to find all the interesting problems with a brand new test item in a brand new context in an hour?) ask them to explain their process and thinking. Make it obvious that saying “I don’t know what I’ll find out” is okay. The problem with testing is that an amateur and a skilled professional both look like people sitting in front of a computer with a blank expression on their face. One is inner thought and structure and careful application of heuristics and evaluation of methods and so on, the other is someone banging on it randomly until it does something. So to make that tacit information available you have to get them to test something and try to express their inner process to you. Oh, and you’re not seeing how well they do on their own, but trying to get them to do their best to see what they could be capable of. If your interview is “how well does this person perform when they’re very nervous around new people and in need of a job with no encouragement from me” then you haven’t seen what they might do if you put them in your company. Get them to communicate their findings - their testing story - including what might make it more testable or what they’d do with more time and resources. Make it clear that questions are fantastic and keep reminding them to ask questions - you don’t want to say “well, they didn’t ask any questions” - you want them to ask questions and then question the questions (“Does the answer to your question matter? Why does it matter?”).

2 Likes
15

I have the view that puzzles and reasoning tests aren’t a bad idea, if they’re used as a small part of the overall process e.g. to get an idea of the candidate’s logic powers ‘on-the-spot’.

If hiring a tester I would definitely include some form of practical test, preferably based on a real-life situation in the hiring company’s product e.g. if the company have a website, use the login form as part of the test or if they company have an app, hand them a phone with the app running on a particular screen such as the login screen.

And talking of tests, I remember a job I applied for years ago at some consultancy place, where part of the lengthy recruitment process was to hand me this little device which had a psychometric test on it, I was then left in the room alone to answer all the multiple choice questions which went on and on and on, all about as useful as a chocolate teapot on a very hot day. After about ten of these questions appeared on the screen, I got bored and started pressing random answers. Unsurprisingly, I didn’t get (or want) the job and happened to see a news item about the same company being rescued by a rival a few years later, after they get burnt by a disastrous takeover. I’m not sure what the moral of this story is, but maybe it’s that if someone supposedly trying to hire you hands you a small device and then proceeds to leave the room, there’s something wrong somewhere with them and their company.

1 Like
16

Pat,

You said, “if someone supposedly trying to hire you hands you a small device and then proceeds to leave the room, there’s something wrong somewhere with them and their company.”

My first thought was of the practice of the British Army back in the imperial days, when officers (particularly those of noble birth) accused of heinous crimes would be given the option of a pistol with one shot, and a bottle of whisky, their subsequent “suicide” being less damaging to their family than a trial.

On that basis, I for one would be straight out of there!

1 Like
17

A logic test or puzzle is just a tool to help you decide if this s the right company for you. Give an answer, any answer, and when told it’s wrong ask why.
You could, if clever enough, come up with an answer that is technically correct, but not the answer they were expecting. (You probably need to have come across the question before though to accomplish that)
I have, in the past, employed a person that told me, in no uncertain terms, the test was crap and a waste of time. That started a great discussion. So, if you’re looking for that ‘magic’ person , and won’t settle for second best, my guess is that this sort of interview question is essential for sparking the great conversation that will follow. There are other ways of sparking that conversation though!