How to hire smarter than the market: a toy model

Let's consider a toy model where you're hiring for two things and that those are equally valuable. It's not very important what those are, so let's just call them “thing A” and “thing B” for now. For one set of abilities, the scatter plot looks like this:

The assumption here is that A and B are drawn from a 2D-Gaussian with a mild positive correlation. I crammed a whole lot of stuff into this plot: the scatter plot shows the distribution of A vs B, and the two histograms (on top and on the right) shows the distribution over A and over B.

We're going to hire some people, so we look at a bunch of resumes and try to decide who's going to make it to the next stage. The best candidates are the ones that are great at both A and B, and we'll obviously bring them in. But some candidates are going to be good at A but not B, or vice versa. So you might choose to evaluate candidates on some combination of the two. For instance, bring in people for which A+B>kA+B>k where kk is some constant:

We can already see something interesting here, which is that the candidates we bring in exhibit a negative correlation between thing A and B, despite those being independent. This is something that's called Berkson's paradox.

I've written about a particular example of this previously: Google found that experience with programming competitions was negatively correlated with actual work success. This would happen if we relabel the x-axis above to say “general interview feedback” and the y-axis as “programming competition success”. The problem isn't that programming competition success is somehow bad: it might have a strongly positive correlation with future work performance. The problem is that Google probably overweighted programming competition success in their hiring process versus other things that would more accurately predict future work performance. This caused an “artificial” negative correlation between those two qualities among the group that were hired.

An interesting paper claims a negative correlation between sales performance and management performance for sales people promoted into managers. The conclusion is that “firms prioritize current job performance in promotion decisions at the expense of other observable characteristics that better predict managerial performance”. While this paper isn't about hiring, it's the exact same theory here: the x-axis would be something like “expected future management ability” and the y-axis “sales performance”.

This problem of overweighting is a consistent theme throughout this post and we'll get back to it!

Because you're hiring in a market with many other players, the really good candidates may simply have so many options that they are going to go to whatever company they want to and make a zillion dollars. Let's say A and B are equally valued by you, as well as the market. We end up we something like this:

Look at the green segment here: now there's an even stronger negative correlation between A and B.

This isn't only for recruiting, and I think this negative correlation is intuitively more clear in the context of buying a home. You might value an extra bathroom as worth +$50,000, and an extra bedroom as +$100,000, but so does the market. As a result, given your budget, you'll see a negative correlation between having an extra bathroom and having an extra bedroom, because the market prices you out of having both.

But this also presents an opportunity. The tradeoffs forces you to focus on the things that you value more than the market. Maybe you don't think a 4th floor walk-up is more than a -$10,000 penalty for you, but the market values it as -$20,000: then in fact you should hone in and target exactly those apartments. As we will see with recruiting, the trick to figure out your own preference versus the market. Let's dig into a few cases.

A common human bias is to interpret confidence as a sign of competence, and as a first order approximation, let's say the market prices both equally. But let's assume that we decide we're not going to succumb to the same bias as the market as a whole. We conclude that the market overvalues confidence, and that we're going to do everything we can to eliminate this bias from our interviewing process. While competence isn't easy to observe, let's say we can get very close by having a carefully thought out interview process. The resulting chart is something like this:

Our cutoff is going to be a vertical line, since we only care about competence, not confidence. The “market cutoff” is going to be the diagonal line. The people we end up considering will be the green triangle.

Here's the weird thing though: our group of people that we consider will:

The same phenomenon arises in more complex situation. Let's say everything else equals, it is better to hire someone from a fancy school, but that the market overvalues it. In contrast, let's say we value general competence slightly more than the market. The market cutoff will be a 45 degree line, but our cutoff will be a line with a different angle. We end up with something like this:

There's a similar conclusion here: we actually see that the candidates we're interested in went to a less than average fancy school.

More generally, the conclusion when you're hiring in a competitive market is that even if you think some quality is desirable, if you think the market overvalues that quality, you should look at the other side of the spectrum. This goes back to my example about buying a home.

The model so far is easy to understand and helps us explore a few tradeoffs when hiring but I think it falls short in a few areas. Let's create a slightly more complex model that is a bit less intuitive but I think slightly more realistic. This section is a bit more math, so feel free to skip if it's not your thing.

What we really want to optimize for is our estimated value divided by the market's value since the market determines the salary, roughly. Assumptions:

Throughout the next few plots, let's set the “market vector” for all of them to (αm,βm)=(1,1)(αm,βm)=(1,1) i.e. the market values the two things equally. We then vary how much we care about those things.

First of all, let's say we value A and B equally, but we care about them a bit less than the market. We bucket the candidates into three different buckets based on the value z=vc/(vm+k)z=vc/(vm+k). If we plot this, we see the same negative correlation (the blue line):

If we decide we're going to outbid the market we can set (αc,βc)=(2,2)(αc,βc)=(2,2) on the other hand:

The more interesting things happen if we decide we don't care about quantity B, but the market still does. This would correspond to the competence-confidence case where the x-axis is competence and the y-axis is confidence. The market values both, but we are smarter and we only value the former. We set (αc,βc)=(1,0)(αc,βc)=(1,0):

Similarly to what we saw earlier, we see that we if we go after the “best” people (in terms of the quantity z=vc/(vm+k)z=vc/(vm+k) then we actually end up hiring the people that have less-than-average confidence. This is because the market systematically underprices those.

In the other case when we think thing A and B are both positive, but we think the market overvalues thing B, we can set (αc,βc)=(1,0.5)(αc,βc)=(1,0.5). This would correspond to the example where A is the fanciness of the school they went to.

The outcome is somewhat similar. Even though we prefer candidates from fancy schools, we still end of being better off hiring people from “average” schools. Again, this is because the market systematically undervalues those. The fact that we consider B a “good” quantity is less relevant than the fact that we consider it less valuable than the market does.

A different case would be a company that only pays attention to some superficial measurement at the cost of things that matter more. For instance, let's say the x-axis is “task-relevant experience” and the y-axis is “fanciness of their degree”. This situation isn't so contrived: I've talked to recruiters at bigcorp enterprises, where there's a strong mandate to only hire Ph.D.‘s but where (my guess) the interview process is pretty noisy. We set (αc,βc)=(0,1)(αc,βc)=(0,1):

The interesting observation is the histogram at the top: the blue distribution of hired candidates end up having a much lower task-relevant experience than the ones that were rejected. Not a hiring strategy I would endorse. There is however one odd twist to this. Let's say we keep ignoring feature A (task-relevant experience), but we throw all our money bidding out basically anyone else based on B (fanciness of their degree):

Because there's some small correlation between A and B, we actually do end up at this point getting higher-than-average skills in terms of A. The cost is that we're going to have to out-spend everyone else. A far more cost-conscious way would be instead to pay a medium premium for A and a small for B, rather than a huge for B but none for A.

If you ask the average recruiter how they find people, it's usually some type of Boolean search on LinkedIn, and when you ask them how they grade resumes, it's typically some combination of having CS degrees from fancy schools, having the exact experience with the tech stack you have (down to frameworks), etc. God forbid if someone has a gap on their resume, or if they need visa sponsorship.

What my model implies, is that there's an “arbitrage opportunity” here. In fact, it's a bit of a silver lining to the fact that the market is biased. Are companies systematically putting a premium on something? Then bet against them! Go after the underdogs. If every hiring manager acts in their own rational self-interest (which unfortunately, they don't) then over time these biases will vanish and the market will converge towards efficiency.

You might have your own personal preferences here, and I'm not going to judge you, but here's a few thoughts on things that may be undervalued by the market:

I want to be extra clear about what my conclusion here is. I'm not saying you should think of it as a bad thing that someone is coming from a fancy school. Everything else equals, it's typically a good thing! What I'm saying is that if you're hiring, then you will be more successful going after candidates that the market undervalues. And this doesn't just apply to measurable things (what school they went to), but also things that people subconsciously value (eg. confidence of a candidate). On the other hand, overvaluing things (that are less predictive of future work performance) can lead you to hire worse candidates.

In all these cases, it turns out your preference versus the market's preference matters more than your preference in itself.

All of this stuff may or may not sound obvious to you!

Tagged with: hiring, business, management, statistics, math