James Reimer, Jonathan Bernier, and the difficulty of evaluating goalies

There's been a lot of discussion about the goaltending situation for the Toronto Maple Leafs ever since Dave Nonis trade for Jonathan Bernier in June 2013. Some people, like me, thought that Reimer was good enough to be the Leafs' starting goalie, and that the trade was therefore unnecessary. Others lacked faith in Reimer and saw Bernier as a potential all-star starter. Some fans felt their time was better spent arguing about unrelated topics, like whether they'd rather fight one horse-sized duck or 100 duck-sized horses.

We're now in the midst of the 3rd season of the James Reimer vs Jonathan Bernier battle for the net, and it's not clear if we're any closer to figuring out which of them is better. It seems like whoever is on a hot streak at any given moment is the one fans believe is better. In 2014, that was Bernier. Right now it's Reimer. But surely there must be some better way to figure out how talented a goalie is than whether he's on a hot streak at the time you answer the question, right?

I prefer to try to answer questions about player evaluation by using statistics. That doesn't mean that I only use statistics, but they usually give us a pretty good picture, and they help to guard against biases that can interfere with our judgement. So it would seem like I should be able to answer the question about which goalie is better statistically.

Thoughts About Charts

The problem is, I can't. While save percentage is an imperfect statistic, various forms of SV% (overall, 5v5, adjusted, etc.) are still the best tools we have for evaluating goalies. But SV% can be just as subject to the vacillations of hot streaks as the eye test can. Here, for example, is the overall career SV% for each of our two goalies, updated after every 50 games played (both are currently just under 200 career GP):

Reimer looked considerably better early on. Bernier took a reasonably-sized lead for much of their careers, but now Reimer has pulled slightly ahead again. If you were using full career SV% to decide which goalie to put more faith in, you'd have chosen Reimer at some points, and Bernier at others. Maybe that would make sense if we were talking about a long span of time, but Reimer hit the 50 GP mark just four years ago, while for Bernier it was even more recent (about 3.5 years ago). Surely our evaluation of a goalie shouldn't be changing on a nearly annual basis!

To highlight just how thin the margins are here: If you take out Reimer's last game (against the Panthers on Tuesday), his career SV% jumps up to .916, giving him a bigger-looking lead over Bernier.

Maybe you think one way to cut down on this effect is to use more recent data. Maybe you think more recent data is more reliable. Well here's what it looks like if we use 100 game samples, ignoring everything else:

Once again, we're jumping all over the place. Bernier opened up a huge lead in the middle 100 games. If we were able to speak to goalie talent with much confidence, and if the most recent 100 games were more telling, at the 150 game mark, we should have been able to say that Bernier was the much better goalie.

Yet, when we use the most recent 100 games, Reimer's opened up a pretty big lead. And it's worth keeping in mind, the most recent 100 games overlaps with games 51-150 for about half the sample. Even shifting part of the sample is leading us to wildly different conclusions.

"But!" I hear you shouting across the aether, "5v5 SV% is better than overall SV%, especially as the sample gets larger!"

Well, firstly, stop shouting, it's rude and I was getting there.

Secondly, here's the same sets of data I posted above, but this time including only 5v5 shots. First, cumulative career 5v5 SV%:

And here's the various 100 game slices:

The numbers here tilt slightly more in Reimer's favour, especially the cumulative total. And yet we still see that you could change your analysis on a roughly yearly basis if you used these numbers to make your evaluation.

Thoughts About Cut-Offs

I've used games played as my cut-off in all of these charts because it means we're using a very similar sample for both goalies. This works well in the case of these two goalies because they're about the same age and have been in the NHL for about the same length of time. It means we're comparing like to like.

But in most discussions about goalies, the cut-off we would use wouldn't be a number of games or shots; the cut-off we'd use is "today". That's can make the problems I've already discussed even worse, especially when talking about two goalies fighting for one net, because one goalie may be getting a chance to improve his totals while the other sits around. This is especially problematic for evaluating a goalie who is performing under his true talent level for a while because he won't get as many starts, so he won't have the opportunity to bring his numbers back up.

Thoughts About Concluding

One conclusion you might draw from all of these numbers is that the reason it's hard to figure out which of these two goalies is better is that they're so close in talent. And I do think that's true; James Reimer and Jonathan Bernier are probably very close in terms of what level of goaltending we can expect from them.

You might say, "OK, but we would expect to have trouble differentiating two similar goalies. But if you throw a goalie who is clearly much better into the mix, it'll be much easier to tell who has more talent."

To which I would firstly respond, "Thanks for not shouting your interjection this time." And secondly, let's take a look.

The consensus best goalie in the world right now is probably Carey Price. I pulled some numbers for him to compare to the two Leafs. To make sure we're comparing similar samples as much as possible, I grabbed Price's first 200 games played starting at the age James Reimer and Jonathan Bernier became NHLers, 22 (Bernier played 7 games before that, but close enough).

Here's how Price's overall SV% compared to Bernier in that span:

Once you get over 150 games played, Price takes a clear lead, but up to that point they were nearly indistinguishable.

What if we compare Price's 5v5 SV% to Reimer's?

Price does eke out a lead after about 100 games, though it all but vanishes another 100 games later.

It does turn out to be true that Carey Price looks consistently better than Reimer and Bernier through a similar number of games, but even then the gaps are not that big. And most goalies are not Carey Price, but we've still got to figure out how good they are.

So we arrive back at my initial question, which is whether we can use statistics to clearly distinguish between James Reimer and Jonathan Bernier, and the answer is that we can't (or at least, I can't). And this is a problem for goalie evaluation. We have to accept a pretty high degree of uncertainty, because the answer to "How good is this goalie?" can easily swing depending on what day you ask the question.

Right now, James Reimer is king of the hill. But if we ask again six months from now, maybe it'll be Bernier again. So in the end, we probably need to accept that it's close, and beyond that we just don't know.