Challenge Hiring Assumptions with Data

We use experiments to model reality (sometimes to create alternative realities as in A/B experiments), to understand reality, and ultimately, to make decisions moving us ever closer to our goals. Improving iteratively, we learn not only from successful experiments but also from failed attempts. Experiments are important because they provide us with measurements. And measurements outweigh opinion.

One accurate measurement is worth more than a thousand expert opinions. 

Grace Hopper

But… can we experiment in hiring?

When it comes to experiments, A/B tests tend to be the default option, but A/B tests are not always well suited for the data we have for hiring. There are some reasons for that:

• We cannot really assure randomness or independence in the sample.
• Causal actions are not under our control.
• The opportunity cost could be very expensive.
• We have a limited number of experimental units.
• We cannot always assure that units do not interfere with one another.

There are also important ethical considerations:

Our data is linked with people who deserve our maximum respect. To ensure that applicants are treated justly, we apply the Beneficence Principle (we protect people from harm from our research) and the Justice Principle (we ensure that participants are not exploited and that when this is applicable, there is a fair distribution of risks and benefits).

So the question is, can we do research and test ideas in hiring? The answer is YES! We won’t use A/B tests but alternate methods. When you cannot run these “properly controlled experiments” (Kohavi, Tang, and Xu), you can still use methods lower in the hierarchy of evidence. Observational studies such as cohort studies or case-control studies can be a suitable approach when you have gathered detailed data for a long period.

Dare to ask!

We sometimes focus on the method and forget about what sparked the process. If you want to innovate, improve, and make the hiring process better for both your candidates and your hiring team, you need questions that challenge your assumptions. You need to dare to ask!

Revolution is asking when the answers are already given.

Miguel Delibes —The Hedge

Our aim is to make the culture of experimentation concept extensive to hiring. In our creed, we say that there is no such thing as a status quo. And this is the spirit we aim for in our statistical studies: “experimentation is a mindset, a team sport, a company culture” that should soak every corner of the company if we want to innovate and improve.

Our goal was to foster people’s curiosity, to have people rethink about what is obvious, and distill ideas out of the beaten track that could improve our processes and outcomes. This is the fuel of our research!

Questions that have sparked detailed data research

The results came fast and we have been able to launch meaningful investigations in an environment where we thought experiments might be challenging to do. Fancy some examples?

What would you ask?

• We can use data to understand relationships between elements in our funnel to make the hiring process lighter and more straightforward for our candidates and ourselves. For instance, we investigated code test scorecards and their correlations with getting an offer after following funnel stages, including the trial. As a result of several of these studies we were able to (1) remove some scorecard items that weren’t effective or were redundant/overlapping, making the process more agile; (2) move scorecards to a different place in the funnel to make the process more fluid; and (3) modify priorities integrating the new learnings, to make the process more accurate.
• We studied correlations of our process and stage length with the candidate’s success with no conclusive result. Important fact: we consider this statistical study to be really insightful too ==> we learn from not-so-unsuccessful research.
• We experimented with removing some stages in our hiring process with diverse results, some of them inspiring.
• We aggregated stats and looked at independence between variables (With χ² tests) to answer to what extent reapplicants (people applying two+ times for the same position) are successful candidates. We got to understand some particularities of reapplicants vs first-time applications and we discovered that under some circumstances, reapplicants have been up to two times more likely to be hired than regular applicants. Having learned this, we took action to tap into this fact where it happens.
• We have launched several analyses comparing two different funnel paths’ behaviors. We’ve taken a look at the time-to-hire for successful candidates, the passthrough rate for each stage, and the time in stage. This has originated some data-backed decisions and funnel changes down the line that made our process more lightweight, faster, and more authentic.
• Thanks to asking the right questions, our studies have given us a better understanding of referral dynamics including scenarios where our referrals happen more or less often and when they are more likely to end up in a new hire. This study has originated actionable recommendations that we are implementing!

Excited about helping people try out their ideas? Why don’t you add your magic to the mix?

Yes! We’re looking forward to hearing from you! We’re looking for:

• Senior Data Engineer → [Apply]
• Senior Data Scientist (Tumblr) → [Apply]
• Senior Data Scientist – Growth Marketing, WooCommerce → [Apply]
• Sales and Marketing Analytics Engineer, Jetpack → [Apply]