What kind of test can you give the candidates as part of the interview?

[zed0011 (Mo Z)]

Hey all,
We currently do not have any automation testers in our company but have been on the look out for a while. We recruited a couple but they left after a couple of months.
We are still on the look out, however I wanted to know whether it’s common to give the candidates some sort of test as part of the interview?
There has been quite a few interviews where the candidates have reached the 2nd stage and when it got techy they struggled.
I think this can be avoided if a test scenario of some sort was given to the candidate who is experienced in automation testing and passing that they can be called for a 2nd interview but also it gives you a feel as to where they are at?

Anyone agrees or disagrees on this? can anyone elaborate on this if this is common?

We have done a few interviews for automation testers and it hasn’t worked out, however when the company interviews for developers they do give them tests as part of the interview.

Thanks all for reading

[kinofrost (Chris)]

Do whatever you need to do in order to not waste their time/emotions and your time/money. I’ve never led a first-stage interview for a tester without seeing them test and explain their actions and choices. I don’t know what your first interview is for, but it seems like it’d save you a lot of hassle to include some hurdles that are important to you - even if just to ask them if they can do what you expect them to do in the second interview.

Also consider what your current interviews look like. Maybe your second interview is very difficult to pass because it is flawed, or the person evaluating it is rejecting newcomers out of fear they’ll lose their job, there’s a lot of reasons that interviews fail. Make sure that your hypothesis that it’s to do with candidates’ technical ability is correct, and not to do with the way you evaluate it, otherwise you’ll be pouring effort into something already doomed to fail. Humility, strategy, success.

1. Consider what you can and cannot expect of a candidate. Maybe you expect them to know how to write C#, but you can’t expect them to know the context around your domain. This will depend on the scope of the role, how you work, what business you are in; everything I don’t know but you and your teams do. Are you hiring someone to just write code, or will they be proper testers who need to know how to formulate a test mission, do test framing, strategize, risk assess, all of the associated skills? Do they need to work in a team, and talk to people, or work alone?
2. Make some simulacrum of something you expect them to do. If you hire a chef you watch them cook. If you hire a coder you watch them code. If you hire a tester you watch them test.
3. Your test should not be pass/fail. Firstly it would be arrogant to think you’re perfect at making the test, second it’s not what you’re looking to see. Ask questions of them, why they’re doing what they are doing, make their invisible brain processes more visible to you - good coders and bad coders writing code look exactly the same. Sometimes a candidate has a good reason why they’re not excelling at your test, and you should have the humility to listen if you want your test to be useful. Know what you’re looking for.
4. Encourage them to ask questions, give them hints, make them feel calm, anything you can do to see them at their best. Imagine if I interviewed for a chef but they failed because they couldn’t find where we kept our frying pans and I didn’t help them.
5. Involve other people if that person will be working with other people or those people can better evaluate candidate answers and behaviour.

It’s a big subject, but hopefully some of that is helpful.

[zed0011 (Mo Z)]

Really appreciate your response thanks. I’ve never been involved in the 2nd stage interview as it’s the dev manager and devs are involved in that, not sure what I could bring to the table. It’s definitely not the case of someone evaluating and is rejecting newcomers out of fear as we have a big gap and need an automation tester to automate and also help us qa’s improve in the automation testing field.

I have sat in the first interview with the dev manager and asked a few basic questions in their current day to day job and a couple of questions to do with automation . They can comfortably talk about selenium /specflow/c# but talking about it is one thing and doing it is another and that’s the problem we have. 1st stage is pretty much getting to know the person and whether they can fit into the team. Maybe this needs to change?

I do agree they won’t have the domain knowledge but they will be working as part of the QA team. They wouldn’t be setting up test frameworks from scratch as it’s already setup .

Having some sort of test I think is good maybe a scenario where they can write the test in gherkin? and maybe code using c#. I think it gives us a good idea what level they are at (no disrespect to them).

I’m not sure where to start and how to achieve this as I’m not an automation tester myself. I have some basic experience, watched tutorials in the past

[lmontgo (Landon Montgomery)]

I have a couple of technical tests that I require as part of my hiring process for testers, depending on the role that they are applying for. There are various public testing sites available for automation practice. I picked two. One that is focused on UI testing, and another on API testing. I require that candidates provide, in any framework or language they are comfortable with to perform at least one positive and one negative test for the API site, and if they are applying for a position that has UI testing, also for the UI site as well. This has to be done before I will interview them at all. I’ll review their submission, and determine whether to go forward from there.

Some folks will claim years of test automation experience, but will then submit code that is completely disorganized, halfway commented out, no noticeable structure or standards followed at all. I am not at all interested in those folks. It’s a waste of both of our time to interview. This doesn’t have to be perfect, or great code. What I am looking for is some form of organization, some form of understanding of the automation process, and development principles in practice. Assuming there are no major issues here, I will proceed with the interview.

During a technical portion, we will review their submission. I will take notes about their choices and ask them questions about why something was implemented one way versus another. Most of the times, the implementations the candidates do are nearly identical. Most candidates will try to use some form of gherkin approach, and in most cases, it’s nearly the same code, with nearly identical assertions, with minor variances, and usually the same junior level of implementation, even when they say they have 5-10 years of automation experience.

I will ask them about making some change, asking what would happen if we did some trivial thing. In many cases, the change is very very simple. There is usually some form of unnecessary duplication in the code, and when I spot that in my review, I will ask them how we can reduce it, in the form of, “If we needed to change this for some reason, how can we take this section and that section and consolidated it, and only need to change it once?” All I tend to look for in this case is a response about using a method. Sometimes there might be variables involved. Simple object oriented approach, and most candidates can’t answer it. When that is the case, I know that they don’t understand some of the basic principles that I need from them.

I may also, especially for back end testing positions, give them a small SQL example to implement. There is a site that one of my people found: https://www.db-fiddle.com. I have used that to setup a couple of small tables. The info on the tables is all there for them to see, from creation to insertions. There is a relation between one table and the other two. I ask them to write a query that requires joining some information on the tables. I want them to walk me through their understanding the tables, ask any questions they have about their relation, and then show me they can write an SQL query. I gave this test to several folks, with just some text so they didn’t have the tables built on their own at first, including people that don’t use SQL regularly, before I started using it, and every one of them solved it on their own in about thirty minutes, so I assumed that other folks that have back end testing experience and say they use SQL every day should be able to breeze through it. Most of them can’t. I don’t mind so much that they can’t generate code on the spot. Interviews are stressful, and it can be hard to do that. I do mind when they can’t talk through how to solve it. Some will not ask questions about the relations, or even make an attempt with pseudo code. I would find those responses acceptable. Not doing that though, tells me that they will struggle with asking for help, or verbalizing when they are struggling with something.

I used to be against take home tests and I absolutely abhor any organization that tries to ask data structure questions, manipulating strings or arrays like LeetCode style things. Those are impractical and focused on the wrong things. I could not care less if a test is 5% faster because of this implementation vs that. I care very much that someone who has worked in test automation can actually write some automated tests, in an organized, maintainable fashion, and be able to talk about it a little. I don’t have to agree with their choices, I just want to know that they thought about what they were doing, and didn’t just follow some guide without understanding it.

[kristof (Kristof Van Kriekingen)]

You could ask them how to test an elevator?

• How would test an elevator?

Just kidding! Ask them whatever seems fit for your organization and something based on his experience.
Let’s say he knows Selenium, you could ask whats the difference between asserts & verify. But the real questions is: How does Selenium work? You want him to know what he’s using and not just someone who uses any framework without knowing what it does.

It really totally depends on the person who you get in front of you.

@zed0011

There has been quite a few interviews where the candidates have reached the 2nd stage and when it got techy they struggled.
I think this can be avoided if a test scenario of some sort was given to the candidate who is experienced in automation testing and passing that they can be called for a 2nd interview but also it gives you a feel as to where they are at?

Our first interview is mostly a personal interview, to see if he/she fits into the organization. 2nd interview is a technical interview. We’ll challenge him/her into the beyond!
It does not matter if he cannot answer all the questions, it matters if he’s willing to learn and being able to admit that he doesn’t know everything and should look it up. (we do consultancy, for context)

Ask them to draw his favorite or previous test automation architecture landscape and see what he includes. Crappy image below but some pointers to think about. If he only draws his test frameworks and nothing about logging, reporting and test data, …

[zed0011 (Mo Z)]

Really appreciate everyone’s responses thanks a lot.
We have an interview coming up next week so I spent some time with a dev to come up with something. I wouldn’t say its brilliant but its better than what we have, which is nothing!
We’ve stripped down our automation tests project to the bare minimum. The plan was to give a quick overview of the app, break a test in several places and ask them to fix the tests. If that goes according to plan then add another gherkin step to the test

[conrad.braam (conrad)]

100% Agree with @kinofrost Chris knows his stuff.

1. I hate people giving me “homework” to do beforehand, and will only do homework for a job where I already like the company already. In todays world it’s a buyers market, so don’t hurt your chances. A screening test, perhaps online that takes less than an hour or two is my limit these days.
2. Definitely have a “problem solver” type test, places I have worked at have used things like “test this wristwatch”, all the way to actual product-related ones, perhaps revolving around detecting a defect class you recently had in the product. Design the “exercise” to gain insight into how the candidate views topics such as documentation and process. Throw in a question on easy “controversial” things like the testing pyramid The Software Testing Pyramid - #10 by paulmaxwellwalters if the candidate is not very forthcoming. “Miro” boards might be a good tool for technical stage interviews if you have to interview remotely.
3. Don’t let the dev manager run the entire interview process, unless the company is hiring a tester to work “in” the dev team? Your strategy of using the 1st round to check “people-fit” of a candidate and the second round to check “technical ability” is what most people use. Don’t second-guess, and always have 2 interviewers present in all stages of the process, a bit like having a safety-latch.
4. It’s good to have a batch of insider knowledge questions about the tech stack that only a practitioner will know the answer to. Choose questions based on their CV, so it does require some pre-reading of their CV. Where i work, we have a curated list of hard questions that we keep on our wiki. It’s OK if they don’t know, but if they get enough of them wrong, you can know to stop the process.

Aside: Still shocked that new hires managed to leave you so soon after joining, that may point to a onboarding problem or structure problem…but . The crowd I work for are hiring too, we also notice we get a lot of candidates who cannot do what their CV says, so it’s not a unique problem.

[ernie]

I’d take a step back and ask what the automation role is and what the expectations are for it? We’re actually making a shift at our company, that we don’t want to hire automation engineers, but we want to hire people to be change agents/quality advocates/coaches/whatever. That is, we want them to be able to write automated tests, but their technical skills are secondary to us as we’re going to be seeking folks who can improve our quality, and do what the team they’re embedded on needs. This might be doing PRs and asking good questions, helping with the requirements gathering, figuring out what integration tests look like, what should be end-to-end tested, how to get faster feedback to devs, etc.

I haven’t figured out what our interview is going to look like, but it’s likely going to be a HackerRank exercise to make sure they can at least write some code, and a lot of behavioral stuff. A question I’ve really liked in the past is asking them to explain their current role and how their team works, and then asking them what they’d change about that process.

This isn’t really if what you’re looking for is people to enumerate/automate test cases, but if that’s what you’re looking for, that might be the problem.

[mirza (Mirza Sisic)]

@ernie HackerRank makes sense, many good companies use that in their interviewing process.

[antji (Antje Rohrbeck)]

We use a kind of similar approach in agreement with all the given answers here. For the technical interview we ask for:

1. solve a simple programming task
2. explain the last automation implementation

This filters roughly 2/3 or 3/4 of all applicants out. With the remaining 1/3 we go into more depth with our specific expectations to see where are the applicants limits and if she/he can admit that she/he requires assistance. In another interview we ask for the personal aspects as well, of course.

With this we achieved a very high level of teams satisfaction with the new hire and a very low quitting rate. Therefore, I’m really not a fan of large homework assignments or intensive programming sessions.

[zed0011 (Mo Z)]

Everyone’s responses have been very helpful. So much valuable information, where have I been??

Our current situation is we’ve only written UI automated tests for some of our apps around 4 years ago using selenium c# and these tests were implemented by devs. We (qa’s) watched tutorials and with devs support learnt to write basic UI automated test using selenium c#. Like I earlier mentioned we had a couple of automation testers that came and went within a few weeks.

So why do we need automation testers?

We have a number of apps and gaps in the UI automation so they would be filling in the gaps in automating the tests for regression testing and new features.
We can also learn a lot from the automation tester. Where there is a need for manual testing they will also contribute to that as well.

What technical exercise have we come up with ?
With a couple of the UI automated tests we’ve removed a few bits i.e. made a change in regex so they need to fix that along with broken code.
They would need to add another step in a test to do an assert along with find an element.

We did have 3 interviews planned last week all of them cancelled!

[ernie]

I think you may want to narrow down what you’re looking for and figure out why folks are leaving before you focusing on finding a new hire. My gut says you’re expecting too many things and the role is not defined well enough for anyone to really succeed at it.

@zed0011

We have a number of apps and gaps in the UI automation so they would be filling in the gaps in automating the tests for regression testing and new features.

This is a lot in and of itself, and it’s not clear if the new hire would be doing new tests, or if much of their role is the old tests.

@zed0011

We can also learn a lot from the automation tester. Where there is a need for manual testing they will also contribute to that as well.

Absolutely yes, but if you’re expecting them to mentor the team or help individuals level up, realize that that could easily be a full-time job depending on the team size.

@zed0011

With a couple of the UI automated tests we’ve removed a few bits i.e. made a change in regex so they need to fix that along with broken code.
They would need to add another step in a test to do an assert along with find an element.

These seem more like “gotcha” problems than actual programming problems. I’d be worried about people failing because they missed a nuance in the question/problem as phrased, and not something I’d want to use a strict pass/fail. If you do it as a live exercise and can watch how they approach the challenge, what questions they ask, etc, then maybe, but I wouldn’t use it as a take-home.

@zed0011

We did have 3 interviews planned last week all of them cancelled!

We’re in a similar boat here. I think we’ve got 3 or 4 current test/quality related openings, but with everyone going remote, it’s getting harder and harder to find candidates.

[karishma30 (Karishma Kothavale)]

Interview for Manual tester can be conducted in 3 rounds. One round can be, we ask candidates basic testing questions to understand their basic knowledge and communication skills. The second round can be Test Round where we can ask candidates to write test cases and find defects in any test website (which company can develop it). The third round can be asking candidates a few scenario-based questions to understand their thinking capacity for testing any application.