Writing a technical interview question isn’t a subject that most people study in school. It can be a bit overwhelming, especially if you haven’t had the opportunity to do interview training or shadow your peers as they conduct interviews. Maybe you’ve just been hired yourself and find yourself already conducting interviews, trying not to let the candidate know how nervous you are.
We’ve developed this checklist to answer some questions you might have about writing a technical interview question. While not intended to be a comprehensive list, it should help make sure your questions don’t suffer from some of the common problems that cause candidates and interviewers to waste time or get a poor interview signal.
✅ Does it assess the real-world skills you’re hiring for?
Take a step back from the details of your question and start by asking what skills you want to assess with your interview question. The more comprehensive and specific you can be here, the better. What might be a good interview question for, say, a junior React developer wouldn’t be relevant for a senior DevOps engineer.
Consider, also, that your question isn’t the only question that the candidate will be asked during their interview: it’s part of a series of tasks that, together, should give you multiple perspectives on the candidate’s skills. So, to evaluate what your interview question will assess for, it’s important to also know what your coworkers will be asking.
✅ Is it clear and approachable, regardless of someone’s cultural background?
Put yourself in the shoes of someone with a completely different cultural background from you—a person with whom you have very few experiences in common. Would they understand your question? It’s natural to assume that the candidate knows the rules to certain sports or games, or that they are familiar with specific places and events. But you don’t want to discover those assumptions were wrong when you’re in the middle of an interview.
It’s also best to stay away from interview questions that could introduce negative associations. For instance, medical records might be a good analogy for the technical problem you want the candidate to solve. But you might never know if a candidate’s personal situation—like having gone through a difficult medical situation recently—could come to mind and distract them from performing their best.
✅ How leak-proof is it?
Let’s face it: if you work for a company that conducts a lot of interviews, the odds are good that after you ask your question a certain number of times, it will be leaked on a platform like LeetCode. Helpfully, though, there are a few measures you can take:
- Write an open-ended, progressive question. An open-ended, progressive question with layers is harder to leak because there isn’t just one right answer. While candidates can certainly still prepare to solve a more open-ended problem, they’ll also have trouble anticipating how the conversation will go—and the many different directions and explanations that you could give in response to their approach.
- Create variations. If you develop many variations of your question that are similar but not identical in what they ask, it’s like having redundant servers: when one of them fails (gets leaked), you’ll have the others to fall back on.
- “Watermark” your question. When your question leaks, you want to be able to find out quickly and retire the question, minimizing the damage to your interview signal. But since many technical interview questions look the same, it’s a good idea to sneak in an idea that will stand out. Imagine being able to search for something like “pink toothpaste problem”—you’d be able to know if you got leaked with high confidence!
✅ Have you tested it with your teammates?
Calibration is key for interview questions. Otherwise, you could end up with a coding task that’s too hard to complete in the allotted time, or too easy, leaving you with a poor signal.
It’s good practice to test your interview questions with a group of your teammates, especially those with the type of experience that you’re hiring for. They’ll give you a sense of how difficult the question is for a qualified candidate to solve, what kinds of questions candidates will ask, and where hints might be needed. Most likely, your teammates will also point out areas where you can improve the task.
✅ Do you have a scoring rubric?
Deciding how to score your interview question can sometimes be just as challenging as writing it. To introduce objectivity and reduce bias, you can introduce test cases that should each count towards the candidate’s final grade. You may want to weight these tests differently so that more complex tests have a bigger contribution towards a candidate’s final score. You might also decide that some of the tests should be hidden, while others should be shown to the candidate as they complete the task.
A better way to write interview questions
Writing effective technical interview questions takes a lot of time and effort, but good questions are key to making sure that your interviews are predictive and fair. At CodeSignal we take writing interview questions so seriously that we have an entire team of assessment design engineers and IO psychologists dedicated to the task. We develop skills evaluation frameworks that assess for on-the-job coding skills while preventing bias and question leaks, so that your engineering team can spend less time maintaining interview questions. To learn more, request a free demo here.