When I joined Maxwell Health in 2014, I was interviewed by our entire product team: a co-founder, our 2 engineers, and our designer. Even as our team grew to ~10 engineers, we all participated in every engineer interview. This was important to us because we had one scrum team, so we’d all be working directly with a new colleague.
Eventually, our engineering team grew large enough that it became impossible to have everyone interview a candidate. It also became less important; we had multiple teams, so not everyone worked directly with everyone else. We made some minor changes, having groups of 2-3 people meet with a candidate for 30-45 minutes. They would ask whatever questions they wanted, then we had a recap meeting where each interviewer gave a thumbs up or thumbs down on a candidate and spoke briefly about their reasons. After discussing everyone’s recommendation, the group came to a consensus about whether to make an offer.
First steps toward change
Over time our interview process evolved. The candidate screening process starts with a manager phone screen, a remote pair coding exercise, then we bring the candidate into the office for the interview. We continue to do this and believe it’s a good process for both us and candidates. The onsite interview gained some light structure as well. We added themes for each group to focus on in their interview with a candidate: communication, technical ability, and process.
We got feedback from new hires that our small groups of interviewers ended up repeating the same questions. Each group had a theme to evaluate a candidate on, but the cadence and flow of conversations was quite similar between interview groups, regardless of their assigned theme. It seemed that as interviewers we innately felt that we needed to ‘get to know’ a candidate as a whole, which resulted in many of us asking the same set of questions. As we subsequently learned, this may feel like the right way to make a smart hiring decision, but it is not backed by science.
My colleague Raj and I both started to see an opportunity for improvement. We have limited time to get to know a candidate and make a decision on whether to hire them; repeating questions wastes that opportunity to get the most information possible. It also felt like people’s assessments of candidates were quite subjective. With no written feedback, we opened ourselves to being swayed by each other’s opinions in our recap meetings.
Looking to the experts
Raj and I shared notes on reading we each had been doing. The two books that provided evidence-based suggestions on how to hire better were What Works: Gender Equality by Design by Iris Bohnet and Thinking Fast and Slow by Daniel Kahneman. What does hiring ‘better’ mean? To us, it means a fair, accurate assessment of a candidate’s potential for success in our organization.
Research in these books shows that structured interviews using quantitative scorecards lead to better decision making, more successful hires, and decreased bias in hiring. In contrast, hiring based on a ‘gut’ feeling alone does not result in successful hires. In its ideal form, a structured interview means that the same questions are asked to each candidate in the same order. Each interviewer fills out their scorecard once, without skipping around to different sections. The scorecards are shared with the hiring manager, who can use that quantitative score to compare candidates.
The first step in creating an interview questionnaire is to identify the characteristics we believe make someone successful on our team. Helpfully, we already had an Engineering Career Development Rubric that define our core values: Technical, Autonomy, Influence, Business, and Execution. Within each of these values we list qualities that define them. For example:
- Technical: Code & Quality, Problem Solving, Operational Excellence
- Autonomy: Self awareness, Empathy, Self Improvement
- Influence: Collaboration, Communication, Leadership
- Business: Customer Focus, Perspective, Domain Knowledge
- Execution: Time Management, Process Oriented, Scope
Raj and I assembled a list of interview questions that we believe assess a candidate on these core values. We are currently in the process of finalizing our first scorecard to use in interviews starting this month. Creating the scorecard is the first step on our path to more fair, accurate assessments of candidates.
What’s Next
There will be challenges along the way that we will need to tackle.
First, Kahneman notes that people are very resistant to adding structure and would much rather rely on instinct. It’s a hard sell to get people to believe research when their intuition tells them the opposite.
Also, we currently have no way of measuring how successful a hire is after they join our team. Comparing performance reviews to scorecard evaluations will allow us to see if the core values we use to evaluate candidates are accurate indicators of success on our team.
Lastly, we may not be interviewing multiple candidates at the same time, and we can’t keep a candidate waiting until we get another candidate to compare them to. We aren’t sure yet how to solve for this, and we will provide updates with what we learn.
Process change is hard, especially when it asks you to stop relying on your gut feelings, but change that has a huge payoff backed by unequivocal research is a clear step in the right direction.
If you are applying to work with us, we are excited to be transparent about our core values and the criteria you will be evaluated on, and we promise you an interview process that we work hard to make better all the time.