Draft valuation is one of the oldest forms of hockey analysis. It's been done in every sport, and there is a fairly famous story about a guy who was hired by an NFL team based on his draft value analysis, and when he gave the team his advice on how to achieve a better team with a clear understanding of the value of the various picks, they fired him. The owner of the team did not want information that conflicted with "what he just liked to think" was true.
Whenever discussion of draft picks and their value comes up, it pays to remember that sports is rife with the gambler's mentality, and so draft picks get imagined as lottery tickets, and all the fantasies a gambler just likes to think are true get pasted right onto the mythos of the draft.
Sensible GMs in the NHL understand draft pick value for trading, and you'll rarely see outlandish pick trades these days. Eric Tulsky of the Hurricanes has personally done draft pick analysis. Lou Lamoriello, the former math teacher, makes reasonable pick trades – even if the point of them seems mysterious. He's still Lou. Sure there's a lot of fans who can't stop rolling every player taken in the first round into a single valuation, but they're just looking for reasons to be mad about something. They don't want facts.
If you do want facts, the pick valuation can be unsatisfying because it answers a question most fans don't really have. That is: how do picks vary in value relative to each other. Value here is only in the sense of the probability of that pick getting your team a player who meets an arbitrary criteria. The arbitrariness is necessary, but it gives you an oversimplified answer to the question fans actually have which is: Is my team getting a good player, one who matters? Or really: How should I feel about this?
To further reduce the real-world applications of pick valuation for anything but pick trades, the values are measured with all players drafted – not always way back to the first ever draft in 1963, but often deep into the past. And that means goalies are rolled up into the much bigger pool of skaters. Goalies aren't like skaters nor is the draft record of goalies like that of skaters.
What I want to know is this: How likely is a drafted goalie to be a truly valuable player in the future? Does it matter how high they are drafted? Which implies that it's possible to know when a goalie is 16 or 17 that they are more likely to be an impact player.
This is less a statistical question and more the feelings question I set out to measure this. And that required some choices and some definitions.
What is a goalie?
I'm not being glib here, I seriously am asking if a guy who stood in a net in 1975 has any relationship to a goalie of today. My answer is no he does not. So which drafted goalies should I look at – just the absolute bona fide butterfly goalies or how about the goalies from the years before when the changes in technique were gradual. What about just goalies who have played in the era of modern goalie skate technology and the current design of posts. You can end up chopping the pool of available goalies a little too fine here.
Since 1963, there have been 1266 goalies drafted and a few of them are duplicates since they were drafted twice like Frederick Andersen. That's not a very big number to start with and yet it's a bit too big for me to want to hand filter it. I need an arbitrary line to say this is where the modern world begins. And the obvious place to find it is with Patrick Roy. Roy was drafted in 1984, ninth overall, so there's one guy that was obviously readable as good while a teenager. My arbitrary line became 1980 to catch the few goalies contemporary with Roy who were also modernizing the game. That leaves me with 1070 individuals. Enough to avoid too much noise. Barely.
What is value?
Walling off goalies from antiquity was the easy part. The hard part is deciding what method of valuing to use on the drafted goalie's ultimate career. There is a very good reason why pick valuations are usually done on number of games played, and that's because that data is right there easy to access. Let's have a look at all 1070 goalies (right up to this summer) and where their NHL games played puts them.
The number of goalies ever drafted very late in the draft is so small, the error bar gets very big on the correlation, but consider it flat for most of its run. The number of goalies with 500 or more NHL games played is only 52. So if value means long career, there are very few and you basically have little hope of picking one on purpose even if they come up in the top 10 of the draft. You can see how rare that is just by looking at the dots.
Games played as a value frustrates me, and when I first started thinking about this, I was thinking skaters, and the usual cut off of 100 games played which then calls Frederick Gauthier or Ben Smith a successful draft pick. But the same holds true for goalies. Ilya Samsonov has 171 games played, and he's not a guy I'd put on my list of successes at the draft.
I wanted something more robust. Short of cutting the pool of data down to anyone playing in the expected goals era, I pretty much had to use save %. So here's the criteria I set: The goalie has to have played more than 1/3 of the games available in a given season (seasons change length a lot over this period) and he has to at least be over that year's league average save % to be said to have a good season.
How many good seasons is enough?
And the last and toughest question was how to cut off this data at the recent end. The image above doesn't do that, so every goalie taken last summer is clogging up the zeros, but this is feelings I'm measuring not absolute probability, so I decided to just show them all at first.
This is the same idea with number of seasons that met the value criteria instead of games played:
Goalies taken with early picks get played a lot whether they're good or not. Which is one of the other problems with draft pick valuation based on games played and is likely why there's so many one good season guys early on.
I considered some complicated processes to chop off the more recent end to avoid skewing things with draftees with no chance to have even played two seasons far less good ones, but all we want to know is how big the pool is we're fishing in. So out of the 1070 draftees, most of whom never play in the NHL, this is the little collection of goalies since 1980 who have more than two good seasons.
That's 104 people with a better record than Ilya Samsonov. And once you aren't drafting in the top half of the first round, draft pick number becomes irrelevant. If you use a pick in the 15 to 40 range on a goalie, I'd question your actions pretty hard. And in general this sure looks like that outside the extremely obvious and extraordinarily rare star goalies who get picked early, no one has a clue who is going to be useful when goalies are teenagers.
Because I used the arbitrary cut off of "greater than League Average Save %" I chopped out the big bulk of goalies who are really close to that level every year. On purpose. If you want one of those guys, you can get them, and they aren't any more valuable than Freddy Gauthier, but they sure are a lot less likely to be consistent. I advise buying in bulk.
So what's the single number chance? I cut the main group down to every goalie born before 2000 who was drafted after spot 15 and I get 905 goalies. Of those, the number with over two value seasons is 92. So the simplified number is 10%. Use a draft pick on a goalie every year for 10 years running and you might get one who can give you more than 2 seasons to make you at all happy you spent those picks. Emphasis on the might.
What about the top 15, though? Twelve goalies out of 22 drafted. So that means the opportunity to try for a hit on the better odds came along once every other year, give or take. This data is from over 47 years of drafts to produce the 22 drafted goalies, of which 12 were good.
I'm wondering what the hit rate is on just trading your fourth round pick every summer for the best goalie you can find. Or just signing a UFA, or even two.
But that's how you should feel about that goalie who got drafted. I'm not sure this even counts as whelmed.
