Thoughts (entirely not mine) on the Crapshootout

The comments section has been alive with debate on shoot-out stats. I thought I would write a post about it. But since the posts have already been written, I will cut/paste. Follow the links though; the articles are great; the blogs should already be in your RSS feed.

Crapshoot - Behind the Net
[...T]here is not enough evidence to suggest that individual skill levels make a difference in player shootout performance. [...] From 2005-6 through the 2009-10 regular seasons there have been 5711 shootout attempts. There have been 1878 goals which is a 32.88% success rate.

[...] There were 571 shooters and 112 goalies involved in this analysis. The players that are do well and those that do poorly are often those that we would not associate with being the highest skilled. [...]

[...W]e looked at year to year correlation among players with more than 10 shots per year and none of those correlation were larger than 0.1. Thus, player performance does not seem to be related (except about the mean) from year to year.

[...] the variability in goalie save percentages is roughly what we would expect for a binomial process. Again this does not suggest that there is not skill differences between players (in this case goalies) but it does mean that the randomness (or noise) of the shootout is completely dominating the skill of the players.

[...[ the effect of each player is the difference from the mean effect [...]. Among these intervals we found one goalie whose credible interval showed evidence of performing better than the league average. That goalie was [...] Marc Denis. We found that no shooter outperformed the league average but several underperformed. Among those with more than 10 shots, the shooters that underperformed were Michael Frolik 1/11 (9.1%), Marian Gaborik 2/18 (11.1%), Martin Havlat 3/18 (16.7%), DanyHeatley 4/25 (16.0%), Tomas Plekanec 2/16 (12.5%), Alexei Ponikarovsky 1/12 (8.3%), Taylor Pyatt 1/13 (7.7%), Bobby Ryan 1/11 (9.1%), Michael Ryder 4/22 (18.2%), Stephen Weiss 4/24 (16.7%). All of this evidence supports that notion that we can treat the NHL shootout as a coin flip. Now this does not mean that a coach should send out any random player. What it means is that for the shooters that are selected --- the best on each team --- and the goalies in the NHL there is not enough evidence after five years of performance data that among these players any one is better than another at the shootout. This is not to say that there is not skill involved in the shootout. Rather the skill is vastly overwhelmed by the noise and, so, the shootout is a crapshoot.

KuklasKorner : The Puck Stops Here : Top 20 Scorers By Shootout Goals Versus Threshold
[...] It is alarming that 10 staged breakaways by Sidney Crosby should have so much value. Sidney Crosby played 1778 minutes of hockey in the 2009/10 regular season. He is credited with 29.4 goals versus threshold on the entire season. In over 29 hours of regular season ice time, almost 13% of Crosby’s value comes from ten staged breakaways. That is far too much emphasis given to the shootout. This is a fundamental criticism of shootouts in general. [...]

One question that needs to be addressed is the repeatability. Crosby went 8/10 in 2009/10, but went 3/10 in 2008/09 in the shootout. There are not enough shootouts to establish reliable statistics. The player with the best numbers in any given season is lucky. His few chances that year are above his more established success rate that years of shootout data would suggest. Basically, the idea is that in ten flips of a fair coin, it is not that unlikely that you will see 8 heads of 8 tails a reasonable amount of time, even though 5/10 is the most probable number. The player who has luck on his side will put up the best number in any given season, when his longterm average level is well below it. In short, shootout success can be measured, but it has limited predictive value.

KuklasKorner : The Puck Stops Here : Top 20 Goalies By Shootout Goals Versus Threshold
The shootout can be extremely important in the valuation of [goalies]. Antti Niemi was worth 11 goals versus threshold last year in the regular season and over 30% of them came on the shootout. This shows that the shootout is far too important in the NHL results. It is a skills competition that is often essentially a coin toss that largely serves to keep the standings random and help ensure a false parity in the NHL. A couple goalies have enough shots faced to show that they are strong in the shootout. Evgeni Nabokov and Pekka Rinne are the two goalies who have the best case to show they are strong shootout goalies. It turns out that wasn’t enough keep Nabokov in the NHL.

The League of Extraordinary Statisticians: The Inaugural - Behind The Net
Greg Ballentine, The Puck Stops Here: Shootouts might be spectacle, but they don't do a good job of determining which team wins a game any more than a hardest shot competition would. [...]Teams have not been able to sustain shootout success from one year to the next. Few players show significant success in shootouts for a sustained period of years. It is not clear that it is much of a consistent skill and it is overvalued. A player who scores three goals on three shootout attempts is worth almost two wins to his team that season and that is far too significant a value placed upon essentially a fluke.

The dangers of Overtime Losses - Behind The Net
While the shootout and the overtime loss have probably made the NHL more exciting for fans, the way they're presented in the standings has made it very difficult to determine how good a team is. Right now, game outcomes are grouped into Wins, Losses and Overtime Losses so that we can see how many points a team has, but nobody tracks whether a team won its game in regulation or in overtime or the shootout. Some teams [...] can get incredibly lucky in the extra frames, leading to a record that's way out of line with their actual performance. This is a very key piece of information - there's no correlation between OT/SO performance and overall performance, a fact that's acknowledged by playing overtime indefinitely in the playoffs to decide a winner.

Shootouts: Goaltender True Talent - Behind The Net
It doesn't matter how we slice the data, we get roughly the same result. At the absolute most, a top ten goaltender is five points better in the shootout than an average goalie (28% vs 33%). Over an average season - 10 shootouts or 33 shots - the 28% goalie makes a .500 team into a .570 team. That is worth 0.7 points in the standings, or 0.35 wins. That's perhaps 10% of a top goaltender's total value above average, so it's not insignificant, but given that the best performer so far is Johan Hedberg, who's well below-average in the thousands of minutes he plays during regulation play, shootout ability is unlikely to drive a goaltending decision.

Shootouts: Does past performance mean anything? - Behind The Net
The problem with assuming past shootout performance is predictive of future performance is that the sample size is too small. When coaches keep calling a certain player in the shootout it's presumably because he's shown breakaway abilities over the course of far more practice and game repetitions than the few shootouts we get to see. That's why the "below-average" groups both get to keep shooting even though they've done poorly in the past and also ultimately end up with a league-average shooting percentage. Their coaches are good enough to assess who has league-average talent. The [...] absolute best few shootout takers in the league appear to be slightly above average. [...] A handful of very good shooters have 45% talent; the next ten have 37% talent; and the vast majority of the rest of the players in the league who get to continue in the shootout are right around the 33% league-wide average. [...] With an average of 10 shootouts per team over the least four seasons, that's a 0.4 point boost, or 0.2 wins. When you get that for free, it's nothing to sneeze at - it's double what I estimate the value of a good fighter is - but it's still on the order of what you get from having the equipment guys make sure nobody's playing with an illegal stick.