Joel Spolsky has a classic blog post on "Finding Great Developers" where he popularized the meme that great developers are impossible to find, a corollary of which is that if you can find someone, they're not great. Joel writes,

The great software developers, indeed, the best people in every field, are quite simply never on the market.

The average great software developer will apply for, total, maybe, four jobs in their entire career.

...

If you're lucky, if you're really lucky, they show up on the open job market once, when, say, their spouse decides to accept a medical internship in Anchorage and they actually send their resume out to what they think are the few places they'd like to work at in Anchorage.

But for the most part, great developers (and this is almost a tautology) are, uh, great, (ok, it is a tautology), and, usually, prospective employers recognize their greatness quickly, which means, basically, they get to work wherever they want, so they honestly don't send out a lot of resumes or apply for a lot of jobs.

Does this sound like the kind of person you want to hire? It should.The corollary of that rule--the rule that the great people are never on the market--is that the bad people--the seriously unqualified--are on the market quite a lot. They get fired all the time, because they can't do their job. Their companies fail--sometimes because any company that would hire them would probably also hire a lot of unqualified programmers, so it all adds up to failure--but sometimes because they actually are so unqualified that they ruined the company. Yep, it happens.

These morbidly unqualified people rarely get jobs, thankfully, but they do keep applying, and when they apply, they go to Monster.com and check off 300 or 1000 jobs at once trying to win the lottery.

Astute readers, I expect, will point out that I'm leaving out the largest group yet, the solid, competent people. They're on the market more than the great people, but less than the incompetent, and all in all they will show up in small numbers in your 1000 resume pile, but for the most part, almost every hiring manager in Palo Alto right now with 1000 resumes on their desk has the same exact set of 970 resumes from the same minority of 970 incompetent people that are applying for every job in Palo Alto, and probably will be for life, and only 30 resumes even worth considering, of which maybe, rarely, one is a great programmer. OK, maybe not even one.

Joel's claim is basically that "great" developers won't have that many jobs compared to "bad" developers because companies will try to keep "great" developers. Joel also posits that companies can recognize prospective "great" developers easily. But these two statements are hard to reconcile. If it's so easy to identify prospective "great" developers, why not try to recruit them? You could just as easily make the case that "great" developers are overrepresented in the market because they have better opportunities and it's the "bad" developers who will cling to their jobs. This kind of adverse selection is common in companies that are declining; I saw that in my intern cohort at IBM¹, among other places.

Should "good" developers be overrepresented in the market or underrepresented? If we listen to the anecdotal griping about hiring, we might ask if the market for developers is a market for lemons. This idea goes back to Akerlof's Nobel prize winning 1970 paper, "The Market for 'Lemons': Quality Uncertainty and the Market Mechanism". Akerlof takes used car sales as an example, splitting the market into good used cars and bad used cars (bad cars are called "lemons"). If there's no way to distinguish between good cars and lemons, good cars and lemons will sell for the same price. Since buyers can't distinguish between good cars and bad cars, the price they're willing to pay is based on the quality of the average in the market. Since owners know if their car is a lemon or not, owners of non-lemons won't sell because the average price is driven down by the existence of lemons. This results in a feedback loop which causes lemons to be the only thing available.

This model is certainly different from Joel's model. Joel's model assumes that "great" developers are sticky -- that they stay at each job for a long time. This comes from two assumptions; first, that it's easy for prospective employers to identify who's "great", and second, that once someone is identified as "great", their current employer will do anything to keep them (as in the market for lemons). But the first assumption alone is enough to prevent the developer job market from being a market for lemons. If you can tell that a potential employee is great, you can simply go and offer them twice as much as they're currently making (something that I've seen actually happen). You need an information asymmetry to create a market for lemons, and Joel posits that there's no information asymmetry.

If we put aside Joel's argument and look at the job market, there's incomplete information, but both current and prospective employers have incomplete information, and whose information is better varies widely. It's actually quite common for prospective employers to have better information than current employers!

Just for example, there's someone I've worked with, let's call him Bob, who's saved two different projects by doing the grunt work necessary to keep the project from totally imploding. The projects were both declared successes, promotions went out, they did a big PR blitz which involves seeding articles in all the usual suspects; Wired, Fortune, and so on and so forth. That's worked out great for the people who are good at taking credit for things, but it hasn't worked out so well for Bob. In fact, someone else I've worked with recently mentioned to me that management keeps asking him why Bob takes so long to do simple tasks. The answer is that Bob's busy making sure the services he works on don't have global outages when they launch, but that's not the kind of thing you get credit for in Bob's org. The result of that is that Bob has a network who knows that he's great, which makes it easy for him to get a job anywhere else at market rate. But his management chain has no idea, and based on what I've seen of offers today, they're paying him about half what he could make elsewhere. There's no shortage of cases where information transfer inside a company is so poor that external management has a better view of someone's productivity than internal management. I have one particular example in mind, but if I just think of the Bob archetype, off the top of my head, I know of four people who are currently in similar situations. It helps that I currently work at a company that's notorious for being dysfunctional in this exact way, but this happens everywhere. When I worked at a small company, we regularly hired great engineers from big companies that were too clueless to know what kind of talent they had.

Another problem with the idea that "great" developers are sticky is that this assumes that companies are capable of creating groups that developers want to work for on demand. This is usually not the case. Just for example, I once joined a team where the TL was pretty strongly against using version control or having tests. As a result of those (and other) practices, it took five devs one year to produce 10k lines of kinda-sorta working code for a straightforward problem. Additionally, it was a pressure cooker where people were expected to put in 80+ hour weeks, where the PM would shame people into putting in longer hours. Within a year, three of the seven people who were on the team when I joined had left; two of them went to different companies. The company didn't want to lose those two people, but it wasn't capable of creating an environment that would keep them.

Around when I joined that team, a friend of mine joined a really great team. They do work that materially impacts the world, they have room for freedom and creativity, a large component of their jobs involves learning new and interesting things, and so on and so forth. Whenever I heard about someone who was looking for work, I'd forward them that team. That team is now full for the foreseeable future because everyone whose network included that team forwarded people into that team. But if you look at the team that lost three out of seven people in a year, that team is hiring. A lot. The result of this dynamic is that, as a dev, if you join a random team, you're overwhelmingly likely to join a team that has a lot of churn. Additionally, if you know of a good team, it's likely to be full.

Joel's model implicitly assumes that, proportionally, there are many more dysfunctional developers than dysfunctional work environments.

At the last conference I attended, I asked most people I met two questions:

1. Do you know of any companies that aren't highly dysfunctional?
2. Do you know of any particular teams that are great and are hiring?

Not one single person told me that their company meets the criteria in (1). A few people suggested that, maybe, Dropbox is ok, or that, maybe, Jane Street is ok, but the answers were of the form "I know a few people there and I haven't heard any terrible horror stories yet, plus I sometimes hear good stories", not "that company is great and you should definitely work there". Most people said that they didn't know of any companies that weren't a total mess.

A few people had suggestions for (2), but the most common answer was something like "LOL no, if I knew that I'd go work there". The second most common answer was of the form "I know some people on the Google Brain team and it sounds great". There are a few teams that are well known for being great places to work, but the fact that they're so few and far between that it's basically impossible to get a job on one of those teams. A few people knew of actual teams that they'd strongly recommend who were hiring, but that was rare. Much rarer than finding a developer who I'd want to work with who would consider moving. If I flipped the question around and asked if they knew of any good developers who were looking for work, the answer was usually "yes"².

Another problem with the idea that "great" developers are impossible to find because they join companies and then stick is that developers (and companies) aren't immutable. Because I've been lucky enough to work in environments that allow people to really flourish, I've seen a lot of people go from unremarkable to amazing. Because most companies invest pretty much nothing in helping people, you can do really well here without investing much effort.

On the flip side, I've seen entire teams of devs go on the market because their environment changed. Just for example, I used to know a lot of people who worked at company X under Marc Yun. It was the kind of place that has low attrition because people really enjoy working there. And then Marc left. Over the next two years, literally everyone I knew who worked there left. This one change both created a lemon in the searching-for-a-team job market and put a bunch of good developers on the market. This kind of thing happens all the time, even more now than in the past because of today's acquisition-heavy environment.

Is developer hiring a market for lemons? Well, it depends on what you mean by that. Both developers and hiring managers have incomplete information. It's not obvious if having a market for lemons in one direction makes the other direction better or worse. The fact that joining a new team is uncertain makes developers less likely to leave existing teams, which makes it harder to hire developers. But the fact that developers often join teams which they dislike makes it easier to hire developers. What's the net effect of that? I have no idea.

From where I'm standing, it seems really hard to find a good manager/team, and I don't know of any replicable strategy for doing so; I have a lot of sympathy for people who can't find a good fit because I get how hard that is. But I have seen replicable strategies for hiring, so I don't have nearly as much sympathy for hiring managers who complain that hiring "great" developers is impossible.

When a hiring manager complains about hiring, in every single case I've seen so far, the hiring manager has one of the following problems:

1. They pay too little. The last time I went looking for work, I found a 6x difference in compensation between companies who might hire me in the same geographic region. Basically all of the companies thought that they were competitive, even when they were at the bottom end of the range. I don't know what it is, but companies always seem to think that they pay well, even when they're not even close to being in the right range. Almost everyone I talk to tells me that they pay as much as any reasonable company. Sure, there are some companies out there that pay a bit more, but they're overpaying! You can actually see this if you read Joel's writing -- back when he wrote the post I'm quoting above, he talked about how well Fog Creek paid. A couple years later, he complained that Google was overpaying for college kids with no experience, and more recently he's pretty much said that you don't want to work at companies that pay well.

2. They pass on good or even "great" developers³. Earlier, I claimed that I knew lots of good developers who are looking for work. You might ask, if there are so many good developers looking for work, why's it so hard to find them? Joel claims that out of a 1000 resumes, maybe 30 people will be "solid" and 970 will be "incompetent". It seems to me it's more like 400 will be solid and 20 will be really good. It's just that almost everyone uses the same filters, so everyone ends up fighting over the 30 people who they think are solid. When people do randomized trials on what actually causes resumes to get filtered out, it often turns out that traits that are tangentially related or unrelated to job performance make huge differences. For example, in this study of law firm recruiting, the authors found that a combination of being male and having "high-class" signifiers on the resume (sailing, polo, and classical music instead of track and field, pick-up soccer, and country music) with no other changes caused a 4x increase in interview invites.

   The first company I worked at, Centaur, had an onsite interview process that was less stringent than the phone screen at places like Google and Facebook. If you listen to people like Joel, you'd think that Centaur was full of bozos, but after over a decade in industry (including time at Google), Centaur had the best mean and median level of developer productivity of any place I've worked.

   Matasano famously solved their hiring problem by using a different set of filters and getting a different set of people. Despite the resounding success of their strategy, pretty much everyone insists on sticking with the standard strategy of picking people with brand name pedigrees and running basically the same interview process as everyone else, bidding up the price of folks who are trendy and ignoring everyone else.

   If I look at developers I know who are in high-demand today, a large fraction of them went through a multi-year period where they were underemployed and practically begging for interesting work. These people are very easy to hire if you can find them.

3. They're trying to hire for some combination of rare skills. Right now, if you're trying to hire for someone with experience in deep learning and, well, anything else, you're going to have a bad time.

4. They're much more dysfunctional than they realize. I know one hiring manager who complains about how hard it is to hire. What he doesn't realize is that literally everyone on his team is bitterly unhappy and a significant fraction of his team gives anti-referrals to friends and tells them to stay away.

   That's an extreme case, but it's quite common to see a VP or founder baffled by why hiring is so hard when employees consider the place to be mediocre or even bad.

Of these problems, (1), low pay, is both the most common and the simplest to fix.

In the past few years, Oracle and Alibaba have spun up new cloud computing groups in Seattle. This is a relatively competitive area, and both companies have reputations that work against them when hiring⁴. If you believe the complaints about how hard it is to hire, you wouldn't think one company, let alone two, could spin up entire cloud teams in Seattle. Both companies solved the problem by paying substantially more than their competitors were offering for people with similar experience. Alibaba became known for such generous offers that when I was negotiating my offer from Microsoft, MS told me that they'd match an offer from any company except Alibaba. I believe Oracle and Alibaba have hired hundreds of engineers over the past few years.

Most companies don't need to hire anywhere near a hundreds of people; they can pay competitively without hiring so many developers that the entire market moves upwards, but they still refuse to do so, while complaining about how hard it is to hire.

(2), filtering out good potential employees, seems like the modern version of "no one ever got fired for hiring IBM". If you hire someone with a trendy background who's good at traditional coding interviews and they don't work out, who could blame you? And no one's going to notice all the people you missed out on. Like (1), this is something that almost everyone thinks they do well and they'll say things like "we'd have to lower our bar to hire more people, and no one wants that". But I've never worked at a place that doesn't filter out a lot of people who end up doing great work elsewhere. I've tried to get underrated programmers⁵ hired at places I've worked, and I've literally never succeeded in getting one hired. Once, someone I failed to get hired managed to get a job at Google after something like four years being underemployed (and is a star there). That guy then got me hired at Google. Not hiring that guy didn't only cost them my brilliant friend, it eventually cost them me!

BTW, this illustrates a problem with Joel's idea that "great" devs never apply for jobs. There's often a long time period where a "great" dev has an extremely hard time getting hired, even through their network who knows that they're great, because they don't look like what people think "great" developers look like. Additionally, Google, which has heavily studied which hiring channels give good results, has found that referrals and internal recommendations don't actually generate much signal. While people will refer "great" devs, they'll also refer terrible ones. The referral bonus scheme that most companies set up skews incentives in a way that makes referrals worse than you might expect. Because of this and other problems, many companies don't weight referrals particularly heavily, and "great" developers still go through the normal hiring process, just like everyone else.

(3), needing a weird combination of skills, can be solved by hiring people with half or a third of the expertise you need and training people. People don't seem to need much convincing on this one, and I see this happen all the time.

(4), dysfunction seems hard to fix. If I knew how to do that, I'd be manager.

As a dev, it seems to me that teams I know of that are actually good environments that pay well have no problems hiring, and that teams that have trouble hiring can pretty easily solve that problem. But I'm biased. I'm not a hiring manager. There's probably some hiring manager out there thinking: "every developer I know who complains that it's hard to find a good team has one of these four obvious problems; if only my problems were that easy to solve!"

Thanks to Leah Hanson, David Turner, Tim Abbott, Vaibhav Sagar, Victor Felder, Ezekiel Smithburg, Juliano Bortolozzo Solanho, Stephen Tu, Pierre-Yves Baccou, Jorge Montero, Alkin Kaz, Ben Kuhn, and Lindsey Kuper for comments and corrections.

If you liked this post, you'd probably enjoy this other post on the bogosity of claims that there can't possibly be discrimination in tech hiring.