The data or the hunch

By Ian Leslie

John Hammond was a boy of ten when he fell in love with the new music called jazz. Rather than heading home after school to his family’s mansion on the Upper East Side of Manhattan, he would jump on an uptown bus and deposit himself, 30 blocks away, in a different world. The world he left behind was monied, white, sedate; the one to which he travelled was poor, black and popping with energy. To John Hammond, it felt like real life. The shop-owners and doormen of Harlem got used to the sight of the skinny white kid in a blue blazer and peaked cap, riffling through records in music stores, flashing a toothy grin at every-one he encountered.

This was the early 1920s. By 1930, Harlem had usurped the South Side of Chicago as the prime destination for America’s jazz and blues musicians. At venues like the Lafayette, Big John’s Gin Mill, Minton’s and the Cotton Club, players and fans would drink, flirt, smoke and play. Hammond was still making the trip uptown, only now they let him into the clubs. In his button-down shirt and tie, he cut an incongruous figure. But he was friendly with dozens of black musicians and club-owners who knew that this lemonade-sipping young white man loved the same music they did and knew all about it.

Hammond was born to immense wealth (his mother was a Vanderbilt) but longed to fly the gilded cage. To his father’s dismay, he dropped out of Yale in 1931 to work in the fast-growing record business. To succeed, he needed to find and record new artists. So he crisscrossed New York, from Greenwich Village to Harlem, in search of undiscovered talent. “Drop into almost any nightclub…any recording date or broadcast or audition or rehearsal,” wrote a jazz critic, Otis Ferguson, “and if you stick around long enough, you are almost sure to see John Henry Hammond, Jr, in the flesh, if briefly.”

One February night in 1933, Hammond rapped on an anonymous door on 133rd St. One of his singer friends, Monette Moore, ran a new speakeasy, and he had come to see her perform. As it turned out, she couldn’t make it. Her replacement was a girl called Billie Holiday. Hammond hadn’t heard of her – which meant nobody had – but she took his breath away. Just 17, Holiday was tall, unconventionally beautiful, with an imperious bearing. Her artistry gave Hammond shivers. She sang just behind the beat, her voice wafting languidly over the accompaniment like smoke from a cigarette. She didn’t just sing the songs, she played them with her voice. “I was overwhelmed,” Hammond said.

Billie Holiday became the first big find of John Hammond’s career, which is recounted in Dunstan Prial’s biography “The Producer”. Hammond’s protégés included many influential jazz artists: the “king of swing” Benny Goodman, the swing pianist Teddy Wilson, the vibes player and bandleader Lionel Hampton, the guitar wizard Charlie Christian. Hammond had exceptional antennae for talent. One night in 1936, restless after watching Goodman perform in Chicago, he went to his car, switched on the radio and spun through the airwaves. Through the crackle at the end of the dial, he picked up the faint sound of a swing band with a driving rhythm section. It was a live transmission from the Reno club in Kansas City, and the band was the Count Basie Orchestra. It wouldn’t be long before Basie, sitting at his piano at the Reno, was presented with the open hand of a conservatively dressed man with an outsized grin. “Hi. I’m John Hammond.”

At the time, black musicians were not supposed to play with white ones. Hammond thought this was crazy. He hated segregation and pushed for racially integrated bands at every opportunity. He also believed that virtually all good popular music had its roots in black culture, and thought it an outrage that, as jazz became popular across America, its origins were being obscured from view. So he decided to educate white people. In 1938 he organised a concert at Carnegie Hall called “From Spirituals to Swing”, tracing the lineage of popular music from African drumming to slave chants, Southern blues, gospel and jazz. It featured Basie, Sister Rosetta Tharpe, Big Joe Turner and many others. It was a sell-out.

the moment he saw Holiday, John Hammond knew she was going to be a star. He just had a feeling about this girl. A hunch

Nobody had told Hammond to go and see Billie Holiday that night in Harlem. She had no fan base, no manager pressing her claims. Nobody would record her. But the moment he saw Holiday, John Hammond knew she was going to be a star. He just had a feeling about this girl. A hunch.

The gift for talent-spotting is mysterious, highly prized and celebrated. We love to hear stories about the baseball coach who can spot the raw ability of an erratic young pitcher, the boss who sees potential in the guy in the post room, the director who picks a soloist out of the chorus line. Talent shows are a staple of the TV schedules. We like to believe that certain people – sometimes ourselves – can just sense when a person has something special. But there is another method of spotting talent which doesn’t rely on hunches. In place of intuition, it offers data and analysis. Rather than relying on the gut, it invites us to use our heads. It tends not to make for such romantic stories, but it is effective – which is why, despite our affection, the hunch is everywhere in retreat.

Strike one against the hunch was the publication of “Moneyball” by Michael Lewis (2003), which has attained the status of a management manual for many in sport and beyond. Lewis reported on a cash-strapped major-league baseball team, the Oakland A’s, who enjoyed unlikely success against bigger and better-funded competitors. Their secret sauce was data. Their general manager, Billy Beane, had realised that when it came to evaluating players, the gut instincts of experienced baseball scouts were unreliable, and he employed statisticians to identify talent overlooked by the big clubs.

The analysts were students of the game but outsiders, less susceptible to the illusory certainties of baseball lore. The scouts might say a young player had “a great body” for the game, or rate a pitcher just by watching him throw. Beane’s stat-heads would find that a player deemed too fat was actually an above-average catcher, or that a player with unorthodox movement was effective. The analysts also noticed that scouts were unduly swayed by whatever they had seen in the last game, and relied on received notions like “clutch ability” that proved, on closer examination, to be meaningless.

The scouts were being hoodwinked by their own brains. The year before “Moneyball”, the psychologist Daniel Kahneman had won a Nobel prize for his research into the rickety apparatus of human cognition, and his work influenced Paul DePodesta, a Harvard statistician who was Billy Beane’s main analyst at the A’s. At first, baseball coaches scorned the idea that nerds with computers could teach them anything, or that there could be any substitute for “good eyes”. But the success of the A’s proved that nerds can see things jocks can’t. “The idea that I trust my eyes more than the stats, I don’t buy that,” said Beane, “because I’ve seen magicians pull rabbits out of hats and I just know that the rabbit’s not in there.”

In football (the non-American version), expert hunches are similarly flawed, and top clubs now employ analysts to offset them. Size doesn’t matter as much as they thought: analysts found that clubs recruit a lot of big players, but use smaller ones more. Some players with a languid style, labelled slow or lazy by coaches, turn out to be covering a lot of ground. And studies suggest that even experienced coaches recall only 60% of the critical events in a match they have just watched – while believing that they noticed everything.

These days, when a football club is interested in a player, it considers the average distance he runs in a game, the number of passes and tackles or blocks he makes, his shots on goal, the ratio of goals to shots, and many other details nobody thought to measure a generation ago. Sport is far from the only industry in which talent-spotting is becoming a matter of measurement. Prithwijit Mukerji, a postgraduate at the University of Westminster in London, recently published a paper on the way the music industry is being transformed by “the Moneyball approach”. By harvesting data from Facebook and Twitter and music services like Spotify and Shazam, executives can track what we are listening to in far more detail than ever before, and use it as a guide to what we will listen to next.

Historically, the music industry has run on hunches. John Hammond is the archetype of what later became known as A&R (artists and repertoire) – the business’s talent scouts. Being an A&R man in the industry’s heyday was a dream job: you were paid to go to gigs and hang out with musicians eager to win your approval. If you spotted more than one or two successes, you were said to have good ears, and handed a fat salary. But the ears of A&R now compete with algorithms.

Next Big Sound in New York is one of a growing number of firms that sell data-based analyses to record companies. According to Forbes magazine, it has found that musicians who gain 20,000 to 50,000 Facebook fans in a month are four times more likely to reach a million fans. It claims to be able to predict album sales within 20% accuracy for 85% of artists. That may not sound so impressive, but then it depends on what you’re comparing it with: the track record of A&Rs was not pretty. In his novel “Kill Your Friends”, John Niven, a former A&R man himself, paints a vivid picture of the job, in the words of his narrator, Steven Stelfox: “Now, I don’t have a perfect track record. No one does. But I’m pretty fucking good. On average I only get it wrong maybe eight or nine times out of ten. That is to say, if you played me ten pieces of unsigned music I might instantly dismiss three or four acts that might go on to enjoy enormous success…We, my label, have spent millions, literally millions, signing and developing music that, as it turned out, no sane person wanted to hear.”

The old music industry turned many young acts into big stars. But it placed many, many more wagers on acts that didn’t sell enough records to pay back; William Goldman’s axiom, “Nobody knows anything”, applies to music as much as the movies. In the social-media era, big bets on untested talent are rarer. This is partly because there’s less money to spray around. But also because the record companies are using data to lower the risk.

The judges tended to agree on who was effective and ineffective, but, 60% of the time, they were wrong. They would have been better off flipping a coin

This is the day of the analyst. In education, academics are working their way towards a reliable method of evaluating teachers, by running data on test scores of pupils, controlled for factors such as prior achievement and raw ability. The methodology is imperfect, but research suggests that it’s not as bad as just watching someone teach. A 2011 study led by Michael Strong at the University of California identified a group of teachers who had raised student achievement and a group who had not. They showed videos of the teachers’ lessons to observers and asked them to guess which were in which group. The judges tended to agree on who was effective and ineffective, but, 60% of the time, they were wrong. They would have been better off flipping a coin. This applies even to experts: the Gates Foundation funded a vast study of lesson observations, and found that the judgments of trained inspectors were highly inconsistent.

The last stronghold of the hunch is the interview. Most employers and some universities use interviews when deciding whom to hire or admit. In a conventional, unstructured interview, the candidate spends half an hour or so in a conversation directed at the whim of the interviewer. If you’re the one deciding, this is a reassuring practice: you feel as if you get a richer impression of the person than from the bare facts on their résumé, and that this enables you to make a better decision. The first theory may be true; the second is not.

Decades of scientific evidence suggest that the interview is close to useless as a tool for predicting how someone will do a job. Study after study has found that organisations make better decisions when they go by objective data, like the candidate’s qualifications, track record and performance in tests. “The assumption is, ‘if I meet them, I’ll know’,” says Jason Dana, of Yale School of Management, one of many scholars who have looked into the interview’s effectiveness. “People are wildly over-confident in their ability to do this, from a short meeting.” When employers adopt a holistic approach, combining the data with hunches formed in interviews, they make worse decisions than they do going on facts alone.

The interview isn’t just unreliable, it is unjust, because it offers a back door for prejudice. In 2009, sociologists at Harvard sent volunteers to apply for actual jobs, armed with identical résumés and similar interview training. They found that white applicants with a criminal record were offered jobs at the same rate as black applicants with no criminal record. Researchers at Bowling Green State University in Ohio followed the progress of applicants to postgraduate schools, and found that when face-to-face interviews were conducted, thinner candidates were systematically favoured over fatter ones. At Rice University in Houston, researchers found that candidates who had a facial blemish, like a scar on the cheek, were rated more negatively than those who didn’t.

In a new book, “Pedigree”, the sociologist Lauren Rivera documents the myriad ways in which the members of American elites favour each other. She examined the hiring processes of leading law firms, banks and management consultants, and found them subtly biased towards applicants from wealthy backgrounds, usually in ways of which the employers weren’t aware. At the heart of her book is a fine-grained study of the interview and the emphasis it puts on “chemistry”. Rivera found that interviewers were much more likely to recommend candidates with whom they struck up a rapport, which in practice meant those from a similar social background – someone they could talk to about skiing or deep-sea diving. When interviewers made this kind of personal connection, they were less likely to question the candidate’s résumé, and strong candidates from less privileged backgrounds were overlooked. A lawyer at an elite firm told Rivera that an interview was like a date. “You just kind of know when there’s a match.”

Claudio Fernández-Aráoz is an Argentine headhunter who has interviewed 20,000 executives in over 40 countries. Even he doesn’t trust his own intuition, unsupported by analysis. “The uneducated hunch is very dangerous,” he tells me. Company boardrooms are filled disproportionately with people who are male, tall, handsome, white and speak in a deep voice. Fernández-Aráoz argues that this is because our intuitions about talent are influenced by criteria that made sense 60,000 years ago, when the vacancy you were looking to fill was a mating partner or someone who could kill wildebeest, and you needed to surround yourself with people like you to survive. The more a job requires brainpower of some kind, and the more diverse our society, the more redundant such ancient heuristics become.

The interview does nobody any favours. Employers miss out on talent, self-starters miss out on jobs and nerves can distort the signal. Employers are beginning to address the issue: Google, which has around 50,000 employees, combines data on a candidate’s career with highly structured interviews, using consistent protocols, in an effort to address unconscious bias. The mystery is why it has taken so long. Social scientists treat the enduring place of the unstructured interview in modern society with the baffled wonder of an explorer observing a bizarre native ritual. They don’t even bother studying its efficacy anymore. They’re more interested in its persistence, and what that says about us.

Why, despite the evidence, do we cling to the hunch?

YouTube is awash with child prodigies, playing the violin or singing “Somewhere Over the Rainbow”. Impressive as they are, the performances tend to be inferior to those of accomplished adults – but they are much more likely to attract our attention. If you wanted to find the greatest reading of Bach’s Invention in C Major, you might choose the late Glenn Gould over Alexander Chen, aged four. Yet, when it comes to views, Gould (341,000) cannot compete with Chen (1.3m).

The reasons for this include the sheer novelty of watching a child performing complex music, and the inexorable draw of that cosmological force, cuteness. George Newman, a psychologist at Yale School of Management, is interested in how, when people consume a clip of a prodigy, they’re also consuming an idea of the future. As we watch a little girl singing an aria from “The Marriage of Figaro”, we’re enjoying not just the performance but a vision of that girl, now a woman, on stage at La Scala, bouquets at her feet. It gives us a thrill we don’t get from an adult’s performance, however virtuosic.

To test this hypothesis, Newman and his collaborator, T. Andrew Poehlman, presented 111 adults with two paintings, one of which, based on a prior test, was typically judged superior to the other. The participants were all told that the artist behind the superior painting was 42. But they were given varying information about the inferior one: some were told the artist was 37, others that it was created by a seven-year-old, and a third group were told nothing at all. People told that the painting was by a child were significantly more likely to rate it highly, and to want to see an exhibition of the artist’s works.

In a second experiment, participants were asked to evaluate a book of poetry. Some were told the author was nine, others that he was 39. They were also given information about its date, either that it had been published 20 years earlier, or that it had just come out. When people thought the book was new, they were much more likely to be interested in buying it if they thought the author was nine (as you’d expect, given the previous experiment). But the effect was greatly reduced when people thought the book was 20 years old. It is the ability to project imagined achievements into the future that influences evaluation. We all want to be ahead of the game.

The prodigies on YouTube rarely go on to become world-beaters, just as people who shine in interviews often perform poorly on the job. The truth is, predicting individual performance in any field with consistency is almost impossible, even for experts: according to research, experience on the job doesn’t improve the predictions made by clinicians, marketers, admissions committees or judges. An organisational psychologist, Scott Highhouse, estimates that only about 30% of the variance in executive success can be predicted, with the rest down to chance. He calls this the “validity ceiling”.

We desperately want to believe our hunches can tell us what will happen in a year or five years’ time

The low level of the validity ceiling makes sense when you think about the web of interacting forces – individual ability, organisational culture, social and economic change, pure luck – involved in any success or failure. Weather forecasters using vast databases can say with confidence if it’s going to rain only a few days in advance. Predicting the outcome of human endeavour is even more complex – imagine if clouds had feelings – yet we desperately want to believe our hunches can tell us what will happen in a year or five years’ time.

Both the analytical and the intuitive approaches to talent-spotting have a low validity ceiling. The difference is that only one of them accepts that the ceiling is there. The analytical way accepts that the future is highly uncertain. The hunch doesn’t accept this for a moment. It knows. Human beings crave the dopamine rush of certainty – what the neurologist Robert Burton calls “the feeling of knowing”. But there’s little correlation between how strongly we feel someone is going to be a success, and the likelihood we’re right. So we keep banging our heads against the ceiling.

Is it time to give up on the hunch altogether, and base every recruitment decision on cold, hard numbers?

We couldn’t if we tried. The hunch will always find its way back in, even if it’s just deciding which metrics to use. Statisticians agree that an analyst should begin with “priors”: assumptions, based on experience, that define the parameters of enquiry. The problem with hunches is not that they are based on preconceived ideas, but that those ideas are little more than bundles of secondhand prejudices, insufficiently reflected on or tested. The hunch can be useful as long as it is educated.

The best hunches are products of original minds. At 22, when he came across Billie Holiday, John Hammond was already drawing on a decade’s immersion in jazz, but it wasn’t just how much he knew, but how he thought, that gave him an edge. Other would-be impresarios were driven by profit as much as pleasure; they had to worry about what people would buy, and so felt safer signing artists who sounded like what had gone before. Hammond, thanks to the luxury of his wealth, never fretted about sales – indeed, he had an aristocratic disdain for those who did. He was only interested in what blew him away. Recalling the first night he saw Basie’s band, he said, “Of course, they completely destroyed me.”

Hunches make people over-confident. But if the hunch runs in the opposite direction to everyone else’s certainties, then over-confidence can be a strength.

Hammond was a contrarian. Katharine Graham, the owner of the Washington Post in its Watergate heyday, said of her old friend, “John always had to be in opposition to everything. I think he wanted to go in the opposite direction of his family…He was overtly against everything establishment…He liked to shock.” His innate perversity drove him away from the paths followed by his peers. He looked for talent in places nobody else would go, took up causes others considered lost.

It’s not as if data solve the over-confidence problem, anyway. Indeed, they can exacerbate it, especially when people get hooked on the wrong measures. That great Manchester United manager Alex Ferguson once sold his best defender, Jaap Stam, partly because the statistics showed Stam making fewer tackles than before. At the time, tackles per match were thought to be the killer metric for defenders. After leaving the club, Stam enjoyed several more years of success in Italy, and Ferguson later admitted he had made an error. Further analysis revealed that Stam was making fewer tackles because he was getting better: he was making interceptions before the need to tackle even arose.

The hunch and the data can educate one another. We need human judgment to correct for cardinal bias – the tendency to place more importance on what can be quantified than what cannot. In baseball, pitching and batting are easier to measure than defence (fielding). As a result, the Oakland A’s manned their outfield with slow and cumbersome players; in his book “The Signal and the Noise” (2013), Nate Silver reports that the A’s gave up eight to ten wins a year because of their poor defence. Billy Beane told Silver that, over the years, he actually increased the scouting budget.

Used intelligently, data can give us an objective view of someone’s past performance and ability. But in our volatile world, the past is an increasingly erratic guide to the future. Claudio Fernández-Aráoz believes that firms frequently get hiring decisions wrong by focusing too narrowly on a candidate’s track record. Instead, he says, they should ask how good she will be at adapting to change. Answering that question requires an understanding of who she is, rather than simply what she does. Daniel Lloyd-Jones, vice-president of A&R (it still exists) at Sony/ATV in London, says that when he signs artists it is not just because of the music they make. “I ask, will they evolve and grow? How will they respond to pressure? Do they have the mental muscle to go from a small club to a big arena?”

The flipside of this is that the talent-spotter must understand how the context for talent is changing. Data-based analysis is good at identifying people who are flourishing in the game as it has been played in the recent past, but the most brilliant individuals have one foot in the future. John Hammond could see talent before it was considered talent. He had a knack for finding individuals who would, in the words of the jazz writer John McDonough, “make us reimagine the most fundamental nature of what talent is”.

In football, the introduction of data-based analysis has been uneven and contentious. Tor-Kristian Karlsen, a former sporting director of AS Monaco, likens the current state of the game to post-Reformation Europe; for Catholic and Protestant, read jock and nerd. There is good reason for wariness. Baseball, like cricket, golf and tennis, is a series of discrete events, most of which involve one or two individuals: pitch, hit, putt, run. Football is far more complex, fluid and unpredictable. Penalties and other set-pieces aside, every minute of play is unique, no two situations the same. That makes the job of quantifying talent much harder.

The most vital skill of an elite footballer, Karlsen tells me, is not physical but mental: the ability to “convert ideas quickly on the pitch”. A great player can read the velocities and vectors of those around him and somehow place the ball, or himself, into a future that he sees a split second before every-one else. When Barcelona’s Andrés Iniesta lacerates an opposing team with a pass that threads its way through four defenders to the feet of Lionel Messi, he is not just kicking a ball, but committing an act of improvisatory creativity. What’s the metric for that?

Jazz is the product of the need to improvise, a mode of existence that African-American musicians, on their way from the cotton fields of Louisiana to the speakeasies of Harlem, had got down to a fine art. Modern societies and, increasingly, modern workplaces, are more like jazz than classical music, football rather than baseball. There are rules, but the action is mercurial, fast and unpredictable. Data-based analysis can reduce the risk of mistakes and help to eliminate bias, by making us better at seeing the world as it is. But there remains a place for those who can see in a flash what it might become.

John Hammond understood jazz through society and vice versa, and he knew that the future of neither was written down (jazz, said the critic Whitney Balliet, is “the sound of surprise”). At the time that he came across Billie Holiday, a vocalist who did not also play an instrument or front a band was not even considered a jazz singer, but Hammond sensed that the world was ready for one. Later, Holiday became one of the first singers to perform regularly in mixed-race jazz clubs, and her popularity cut across the rigid ethnic lines of the day. In Monette Moore’s bar that night, Hammond saw the future of jazz and the future of America at the same time.

After the glorious early peaks of Hammond’s career came the valley. He saw active duty in the second world war, and, on his return to New York, spent a few years casting around for a purpose. He and his first wife divorced. His finger no longer on the jazz pulse, Hammond devoted more time to recording classical music. What he had achieved as a young man was enough to book his place in the history of 20th-century music. But, after a quiet second act, Hammond was to enjoy a spectacular third.

In 1959 he was asked to rejoin the label he had worked for in his 20s, Columbia, by a friend from that time, Goddard Lieberson, who had become president of the company. Paid a nominal salary, Hammond, now in his 50s, was given free rein to find musicians he loved. In the early 1960s, Columbia, though still profitable, was in a precarious position. Young people were listening to new artists that Columbia’s A&R men didn’t really have a handle on. A gale of change was in the air. Hammond, as ever, followed the music.

In 1960 a musician sent him a demo tape with several tracks on it, the last of which featured a female singer singing to a piano accompaniment. Hammond passed on everything but the girl, tracked Aretha Franklin down in Detroit and produced her first album. In 1967 he signed an oddball Canadian poet called Leonard Cohen after hearing him play a few songs in his room at the Chelsea Hotel. In 1972 he signed a young singer from New Jersey called Bruce Springsteen.

In 1961, Hammond had been listening to the new folk singers in Greenwich Village and persuaded Lieberson to take on a singer-songwriter called Carolyn Hester. At a rehearsal for Hester’s first album, he found himself captivated by the skinny young man she had brought along to help out. Later, Hammond described the scene: “I saw this kid in a peaked hat playing not terribly good harmonica but I was taken with him.” He asked him, “Can you sing? Do you write? Why don’t you come up to the studio?”

Hammond became convinced that the kid would be the biggest star he had signed. None of his colleagues agreed. After the young man’s first album sold poorly, other executives at Columbia began to whisper about “Hammond’s folly”. But Hammond urged Lieberson to be patient. He had felt something that day at the rehearsal, and he trusted what he felt.

“I was sitting there thinking, ‘What a wonderful character, playing guitar and blowing mouth harp, he’s gotta be an original.’ It was just one of those flashes.”

The kid in the peaked hat was Bob Dylan.

Illustrations Peter Horvath