We know about size bias in drafting, but do we understand age bias?

Brady Tkachuk, pictured above, is widely ranked in the top five of the coming 2018 NHL Entry Draft. He is also the oldest of the top players, having been born on September 16, which as we all should know, is one day before Auston Matthews’ birthday, and one day after the cut off date for the NHL draft.

If Tkachuk had arrived a day earlier, he would have been the youngest player in last year’s draft. And most of the more savvy draft-watchers think we have a pretty good handle on what that means. We assume that we should mentally (or mathematically) shave a little off his stats and drop him down the rankings a little. You have likely commonly seen people refer to 1999 birthdays and 2000 birthdays (or whatever year is applicable) and we all nod knowingly and say, yeah, maybe he’s not that good, he’s older.

I did this very thing when looking at Andreas Johnsson’s history to discuss his coming contract amount. Johnsson, who was born in November, would have been much older than most of the other players in his draft year, and I wondered if that might have led him to be drafted later than it now seems he should have been.

It turns out, he may have been drafted later because of all the time he spent younger than everyone else. And Brady Tkachuk and Auston Matthews may also have been underrated at the draft (although not in terms of draft position in Matthews’ case, obviously).

Can understanding age bias help us identify players who are ranked too low and might be a draft-day steal?

Malcolm Gladwell and Canadian Junior Hockey

Many years ago there was research done on the birth date distribution in Canadian Junior Hockey (CHL), and this has been repeated many times. It shows that there is a consistent over-representation of players born in the first quarter of the year in the CHL.

There’s a study from 1988 that showed the phenomenon.

It was found that players possessing a relative age advantage (born in the months of January–June) were more likely to participate in minor hockey and more likely to play for top teams than players born in July–December and thereby disadvantaged by their relative age.

In 2008, Malcolm Gladwell made this idea more famous by looking at it in relation to the NHL in his book Outliers.

It’s a beautiful example of a self-fulfilling prophecy. In Canada, the eligibility cutoff for age-class hockey programs is Jan. 1. Canada also takes hockey really seriously, so coaches start streaming the best hockey players into elite programs, where they practice more and play more games and get better coaching, as early as 8 or 9. But who tends to be the “best” player at age 8 or 8[sic]? The oldest, of course -- the kids born nearest the cut-off date, who can be as much as almost a year older than kids born at the other end of the cut-off date. When you are 8 years old, 10 or 11 extra months of maturity means a lot.

He found that this pattern persists into the NHL. The explanation for what is called a Relative Age Effects (RAE), that Gladwell gives above is considered a “rational” one. That is, there are very good reasons for the players in junior hockey to be sorted by relative ages, with advantages going to the older players.

This isn’t news. Anyone who went to grade school can remember children too small or immature to really fit in with their classmates, or maybe the opposite: The boy who shaved first or the girl who was so tall she won all the running races because they were born on New Year’s Day.

This phenomenon persists beyond junior hockey and into the draft. And understanding the history of who was selected when, and how they performed can help us understand that the common understanding that a player like Tkachuk must be overrated is exactly backwards.

Selection Bias at the Draft

Now before we begin, a note on applying effects seen in large populations to individual players: It’s a dangerous thing to do. In my examples, both Tkachuk and Johnsson grew up in households with hockey-playing fathers who had enjoyed high levels of success. Johnsson’s father player over 800 pro games for the club that Johnsson trained in. It’s entirely possible that any issues caused by his age were outweighed by his status or all that extra coaching.

I am largely drawing on the analysis and conclusion of one study that was conducted on all Canadian-born skaters drafted between 1980 and 2007.  The study was looking for evidence that there is selection bias at work in the NHL draft. That is, that players with later birth dates were drafted lower than they should be and players born early in the year were drafted higher. They also wanted to see if this bias was irrational.

The study grouped players by quarter of the year they were born in and ignored players returning to the draft a second time.  Remember, as you read along, that all players born in the fourth quarter of a year (October to December) are the absolutely oldest players in their draft cohort, and they’ve flipped from years spent as the youngest players on their teams. The absolutely youngest players in the draft are those born in the third quarter (June to September).

First the study authors confirmed that players with  first-quarter birth dates are consistently drafted at a higher rate and fourth quarter at a lower rate.

This is an indication that the RAEs are persistent well past the point where the physical and maturity differences are dramatic because of a few months difference in age. One point made in the study is that draft rankings are based on years of observation and opportunities that are affected by the rational reasons to select the older players and give them the best chances on the best teams, but no one has ever studied if that CHL imbalance is actually beneficial to the teams’ success.

The study used a measure of future productivity of a player by NHL games played and points. And they expressed productivity as a percentage of total achieved by all players. The comparison for all the players in the study, sorted by which quarter of the year they were born in, looks like this:

So players like Matthews and Tkachuk who are in quarter three, or Johnsson in quarter four, outperform their share of the spaces in the NHL — in the case of the fourth quarter players, by a very large margin.  And for every January baby ripping up the NHL, there’s a lot more who aren’t.

So one of the issues this raises is that if draft rankings are rationally assigned, they might have more younger players higher in the draft because, to take Matthews as an example, he’s spent his life the smallest, youngest guy on his team, so he must be even better than we think.  The study accounted for that, and do go and read the details on the regression techniques they used, but they found that in fact, while younger players are drafted a bit earlier than older players, they aren’t drafted as early as they “should” be. The selection bias is still suppressing their placement.

They also looked at whether this bias persisted in the top 30 players taken each year, with the idea in mind that those players were more carefully scouted and more clearly understood.  The same situation persisted.

They then looked at who reaches various career benchmarks and found that fourth-quarter birth dates outperform all others:

Escalation

Escalation is the term used in studies of the NBA for the phenomenon of teams giving more playing opportunities to players selected earlier in the draft. This study found evidence for this in the NHL as well. Draft position was shown to have a predictive ability for future games played.

Put that together with the evidence that the draft order is “wrong” by over-emphasizing early birth dates at the expense of later birth dates, and you have NHL teams wasting resources and time on the wrong players, and making the wrong decision on ambiguous players. Johnsson or Leivo? The seventh rounder or the third?

Conclusion

The study concluded that they had proven the presence of selection bias — that is, that players whose birth dates are later in the year are more likely to be drafted lower than their innate talent and skill indicates:

Thus, if NHL teams simply drafted based on (unbiased) perceptions of talent, there would be no relationship between relative age and productivity. We showed, however, that there is a strong relationship, even when draft slot is controlled.

They also considered potential rational reasons for the bias:

Perhaps the most plausible idea is that relatively older individuals are rationally preferred, despite their worse long-term productivity, because they are more likely to make an early contribution to a team. In particular, a relatively older player would be absolutely older than a relatively younger one, meaning that they would be further along in their development.

And if you’ve been remembering that issue with the draft eligibility date, you will immediately see why that’s wrong. If that was the case, then the fourth quarter birthday boys would be the over-selected group since they are absolutely older than everyone else. And yet, that is not the case, and persists in being not the case over time. Knowing this phenomenon exists hasn’t stopped it happening.

Paradoxically, then, relatively older individuals (e.g., born in first quarter) are absolutely younger than those in their draft-eligible cohort who were born in the fourth quarter. Furthermore, our results show that relatively older individuals are not only less likely to reach long-term career benchmarks (i.e., 400 career games); they are also less likely to even play a single game (Fig. 3; Table 1). Thus, it is reasonable to conclude that the selection bias we have shown is not in the drafting teams’ best interest (i.e., it is irrational).

They consider if this is all a result of persistent RAEs. That is, that differences in opportunity experiences when younger determine draft order when older, but actual ability is what determines outcomes, so that explains why the younger players are out-performing their draft position.

This argument has some validity, yet it overlooks that being drafted ‘later than one should,’ or not being drafted at all, can be costly. Our analysis of escalation effects shows that, once in the league, players who are drafted earlier receive greater playing opportunities than is warranted by their productivity. These opportunities should allow them to more easily establish their reputations, and these opportunities can translate directly into compensation because recently drafted players can earn most of their income through playing-dependent performance bonuses

The most plausible cause of this bias in drafting remains the persistent effect of evaluations made on players when they are much younger than 18, and also the opportunities to play on elite teams and other ways of getting draft buzz. How often does the Canadian team at the WJC have a draft-eligible player on it? It might be a reputation effect at play.

They also consider this intriguing idea:

… selection bias in NHL drafting is based on some sort of ‘underdog’ effect whereby relatively younger individuals, because they have faced greater social challenges [9], [14] (see also [34]), compensate by developing greater adaptability to different roles or better work habits; these traits then lead to greater long-term achievement.

That sounds like a profile of Auston Matthews. But they quickly point out there is little direct evidence for this explanation, and any advantage you get from it is offset by the disadvantages of relative age you’ve been experiencing.

This is much like saying a second rounder has to work harder to get noticed, so they show up at the Combine as the players in the best shape, something I’ve seen anecdotally.  That work ethic is all well and good, but they still aren’t as good as a better player who skips leg day more often.

My Theory

I have no evidence for this, but I think some of the large discrepancy in performance vs draft position for players with fourth-quarter birth dates (and a few of the third-quarter ones as well) is due to manual over-correction by scouts for the very fact that they are the oldest players on draft day.

Those age differences are the easiest to see: The year is different. And when we try to mentally adjust something, we usually get it really wrong. Much like the way we tend to overestimate the effect on players’ results of quality of competition or zone starts, NHL scouts could be overcompensating for the older players’ age in their draft lists.

It’s also possible that statistical models used by scouts downgrade the results for these players when they should be doing the opposite.

That doesn’t explain the overall bias, however, and the study leaves open the idea that this is a broadly applicable phenomenon, and the NHL draft might actually do a better job of accounting for it than some other situations that judge adolescents on their potential. The research on choices made by teachers should make you consider how your children’s futures are steered based on their birth dates.

At the end of the day, the why is a problem for the scouts to solve, what we should remember is that the later in a year the player was born, the more likely they will be drafted later than they should be.

The 2018 Draft Class

Here are this year’s draft prospects, sorted by birth date. This is NHL SC’s top 31 skaters from both Europe and North America as listed on Elite Prospects. And if you’d like to not have to mentally measure them by quarters, look at this version with its helpful shading.

Interesting, isn’t it, all those first quarter draft prospects (40% of the total). But don’t forget it’s the 1999 birth year prospects who are most likely to be underrated even though they are older.

By the way, Ryan Merkely isn’t on this list because CS has him 45th on their final list. His birth date is August 14, which would make him the third-youngest player on the list. I wonder how often he’s been told he’s immature in one way or another as one of the  younger players on his team? Compound that with the fact that he is one of the youngest players in the draft, and you might have the player who will be considered a steal in a few years because he’s being ranked too low now.

2018 NHL Draft Prospects