ESPN Total QB Rating- Week 1 (and the end?)

[gchwood]

So with much fanfare, ESPN released its new rating system for QBs earlier this summer. With most of the first week in the books, they have released the current TQBR rankings and it seems a bit shady. Cam Newton barely out classes Rex Grossman, which isn't that much of a shocker, but the real surprise and the crushing blow (to me) of the fledgling rating system is that Donovan McNabb it his abysmal play yesterday was not the worst rated QB or even next to last. According to ESPN, McNabb's 47% completion 39 yd 1 TD 1 INT performance was better than that of Colt McCoy, Mark Sanchez (who won), Matt Cassel, Ben Roethlisberger, and Kerry Collins.

McNabb has a TQBR of 20.7 yet Kerry Collins (who threw for 197 yds with 52% and 1 TD 0 INTs) had a 2.3!!!

TQBR Link

So what do you all think of this rating system? Is it fair, or is it as flawed as I am seeing it?

[killerbee99]

It's stupid and it will die soon..........

[Sellersfan]

How does Romo sits to pee, who put up yardage last night, but single handily cost his team the win with a fumble and a pick end up at #11?

I thought this new rating system of theirs was going to fix this.

We need ASF's opinion on this rating system

[justice98]

They really want us to care about this.

[Atlanta Skins Fan]

We need ASF's opinion on this rating system

It's complete horse****.

The first problem is that it's a black box. They won't document the formula.

Second problem, is that they won't document the variables sufficiently, so the black box can't be independently verified to work. (They won't post a form on a Web site, allowing the formula to be tested by others. This would have been a method of proving a proprietary algorithm.)

Third, the variables apparently are based on expert analysis of video for each play, which means that the QBR can't be back tested adequately.

Fourth, they haven't demonstrated quantitatively if and how the QBR has bettter value than the NFL passer rating. They appeal via argument, such as "the NFL passer rating does not incorporate sacks and running plays," therefore it's inadequate. But, this doesn't demonstrate inferiority of the NFL passer rating with regard to effectiveness of QBs (winning games). It's reasonable to think that QBs who get sacked a lot will lose more, and QBs who run for a lot of yards will win more. But, that doesn't mean it's true. For example, it might be that avoiding turnovers is more important than running or avoiding sacks. Where is the quantitative proof of the relative importance of any variable?

Fifth, as a related point, they are looking at too many variables. I noticed in testing QB formulas that many variables that "seemed" important to me intuitively did not prove to be important on testing. For example, we might think that a come-from-behind TD pass is worth more than other TD passes. However, testing may show that QBs who don't have to come from behind very often are sometimes better QBs. They put their teams ahead early in the game with multiple TD passes, But, weighting this factor is also a mistake, because it diminishes those clutch performances.

It's very humbling to test the actual relevance of various factors independently. I ended up concluding that the available stats from a box score were perfectly sufficient in constructing a better QB rating, but how they are used (the formula) matters greatly. When I tried to use more complicated, contextual stats (from splits), these tended to break down in back testing. Splits can tell you something about a particular QB, but there are few general insights from splits that hold up over time in a QB formula.

[laxpck]

Here is my rating

This BS = 2nd to good old QB Rating.

All they got to do is change QB rating to a 100 pt scale and it would be perfect

[GSF]

What a joke. ESPN outdid themselves with this one.

[SkinsCrushCowboys]

They are trying and failing to make this math problem relevant.....

[Sellersfan]

If Collins finished last yesterday they clearly can not seperate the QB from the overall team play.

Colts had breakdowns in every area of their team, and Collins is the goat for it all.

[rebornempowered]

Any rating system that wouldn't have McThrowsattheground second from the bottom just above Big Ben has to be a turd. They need to go back to the drawing board.

[FanboyOf91]

McNabb has the highest Clutch Weight Avg?

[MattFancy]

I haven't taken this new rating seriously from day 1. This gives me more reason to keep feeling that way.

[Veretax]

To paraphrase ASF's argument here. It's the context silly. WIthout context, they could be pulling them out of a hat and we wouldn't even know it.

[Oldfan]

It's complete horse****.

The first problem is that it's a black box. They won't document the formula.

The logical basis makes much more sense than what you have offered for your black box college QB projection system.

Here;s the logic: If it were possible to have the team’s offensive coordinator grade the quarterback’s performance as they do after every game and publish their results, and if all OCs did it with the same method, we would have a more accurate QB rating method. Lacking that, the next best method is to gather stats that are screened play by play in the same way by someone with football knowledge.

For example, we know that all completions. incompletions, and interceptions aren’t equal, so we should lack confidence in any formula that treats them as such, and have more confidence in a formula that makes intelligent distinctions between them.

Bottom line: It’s impossible to say how accurate this new approach is, but logically it makes much more sense than the NFL’s formula.

[Atlanta Skins Fan]

It’s impossible to say how accurate this new approach is, but logically it makes much more sense than the NFL’s formula.

And what I'm telling you is that many "logical" inferences about QB play prove not to be true when tested as an independent variable over a good sample size. Even more important, the weighting and balancing of variables is critical.

Also, your analogy that the ESPN formula is like an OC grading a QB is also horse****. The ESPN formula actually gave McNabb the highest "clutch" rating for all QBs in 2011. When that happens, you fire the statisticians and apologize to readers.

The NFL passer rating is far from perfect, but it's many times more useful than this crap.

[Brett81]

Fixed it for ya guys...

We need ASF's opinion on this rating system

It's complete horse****.

End Thread.

[greenspandan]

If Collins finished last yesterday they clearly can not seperate the QB from the overall team play.

Colts had breakdowns in every area of their team, and Collins is the goat for it all.

totally agree. i watched that game, and kerry collins was kerry collins. the entire rest of the team was completely horrible, though. did peyton manning also run the defense and special teams? because they just looked lost.

---------- Post added September-12th-2011 at 03:32 PM ----------

i admit i've never paid attention to this before.

but this is hilarious: the thing is apparently a numerical rating, based on some black-box formula, based somehow on experts watching film (and presumably making some kind of subjective analysis)? what a joke! don't pretend it's some mathematical Nate Silver Moneyball type statistical anaylsis when you're really just asking experts what they think!

[Oldfan]

And what I'm telling you is that many "logical" inferences about QB play prove not to be true when tested as an independent variable over a good sample size. Even more important, the weighting and balancing of variables is critical.

Let's use one example of a logical inference: a Hail Mary interception should not count as an INT in the formula. Does your argument that "many logical inferences about QB play prove not to be true" contradict ALL logical inferences? Of course not.

Also, your analogy that the ESPN formula is like an OC grading a QB is also horse****. The ESPN formula actually gave McNabb the highest "clutch" rating for all QBs in 2011. When that happens, you fire the statisticians and apologize to readers.

Please explain how the clutch rating is computed and how you know McNabb's shouldn't have been the highest.

[greenspandan]

For example, we know that all completions. incompletions, and interceptions aren’t equal, so we should lack confidence in any formula that treats them as such, and have more confidence in a formula that makes intelligent distinctions between them.

that's only true if there is a good reason to think QBs don't all experience similar numbers INTs that weren't really their fault, etc, and that the difference is pronounced enough to be relevant. i'm not convinced it is.

[Oldfan]

that's only true if there is a good reason to think QBs don't all experience similar numbers INTs that weren't really their fault, etc, and that the difference is pronounced enough to be relevant. i'm not convinced it is.

There is ample evidence to think that all QBs don't experience similar numbers because we know that they don't all have equal supporting schemes and supporting cast. Isolating the QB from his support is, in fact, the core problem in grading them.

[Destino]

Let's use one example of a logical inference: a Hail Mary interception should not count as an INT in the formula. Does your argument that "many logical inferences about QB play prove not to be true" contradict ALL logical inferences? Of course not.

Please explain how the clutch rating is computed and how you know McNabb's shouldn't have been the highest.

Q. Explain the Clutch Index.

A. The Clutch Index or Clutch Weight or, for those familiar with baseball analytics, the Leverage Index is a measure of how important any play is towards changing the winning percentage of the game. This Clutch Index is calculated pre-play and does not depend on the outcome of the play. We think of it as generally reflecting how much pressure a player may feel on the play. For example, imagine a team down 4 with 3 seconds to go in the game. In one case, it is 3rd and goal from the 3 yard line. In the second case, it is 3rd and 10 from midfield. In the first case, it is a high pressure situation. The team can win with the right play call, the right block, or the right pass, but they can also lose the game with the wrong call, block, or pass. Whether they win or lose on that last play, people will be talking about the last play after the game. In the second case, it is unlikely a Hail Mary works, so if they don't win on that last play, people will be talking about the rest of the game, not the failure on the last play. Only if they win will people be talking about that last play; there was less pressure then. The Clutch Index reflects this by assigning a much higher value in the first case than in the second.

One of the big things a Clutch Index was set up to do was to minimize the value of plays made in games already decided. When a quarterback is piling up yards in the 4th quarter down several scores, they are doing it against a defense that is not working as hard as they would normally. The defense may be in prevent mode, they may take out their better players, but they just aren't competing the same as you would see in a tight game. Our advisors, from Ron Jaworski to Trent Dilfer, felt strongly that this should be accounted for.

To develop the Clutch Index, we looked at the game time and the game closeness (defined to be how close the win probability was to 50%) and how plays affected the win probability on average. The Clutch Index then became a calculated function just of time in the game and how close it was. Late close games were more clutch than early close games. But early close games were more clutch than a lot of other situations. Blowouts at any time of the game received fairly low clutch indices. This generally did what our advisors asked for, also rewarding performance in tight games.

For those aware of AdvancedNFLStats, that site has WPA, which stands for win probability added. WPA looks at the actual change in winning percentage with every play. It is an intuitively nice concept, but it has strange consequences. Single plays can completely dominate the rest of the game. A game decided on the last play gives almost all weight to that one play, even though the other plays building up to it were important in putting the team in a position to win on that last play. WPA also has the flaw of weighting every win the same amount. A 45-3 win is not viewed any differently than a 24-20 win even though there is a big difference in those games in that one represented domination and the other could be luck. A quarterback in that first game probably played great, whereas in that second game, the QB may have played great, but threw an incompletion on 3rd and goal from the 3 with 3 seconds left to "lose" the game. WPA for that second QB would probably reflect too much of that last play and not enough of the whole game.

Finally, a question we got from Aaron Schatz at Football Outsiders was this, "If you have two QBs with the exact same performance, but one has a bad defense and the other has a good defense, will the QB with the bad defense get a better rating?" The question is motivated by the idea that a good quarterback with a bad defense will be facing more close games than a good quarterback with a good defense. As a result, their clutch opportunities will be higher. The answer to the question is actually not clear. Because QBR normalizes by how many clutch opportunities quarterbacks get, there is no straightforward answer to the question. If they are good quarterbacks and both do exactly the same in clutch situations vs non-clutch situations, then they will both have the same value for QBR. That is as straightforward an answer as we can give.

http://espn.go.com/nfl/story/_/id/6909058/nfl-total-qbr-faq#faq5

When it takes you that long to explain how you arrive at a number... and you still failed entirely to explain how you arrive at a number. It's bull****.

[Oldfan]

Thanks, Destino. That quote explains what the Clutch index is (which is pretty much what I expected) but it doesn't answer the questions I asked ASF.

[Atlanta Skins Fan]

Please explain how the clutch rating is computed and how you know McNabb's shouldn't have been the highest.

Formula: 39 yards, 1 INT, 46.7% = clutch QB.

[gchwood]

I think that their concept of clutch, refers to the closeness of the game, thus each play would be more clutch than that of say Kerry Collins who was getting blown out. Thus why Romo sits to pee had a high clutch rating. That being said, the clutch rating still does not explain why McNabb would be anything but the worst QB yesterday, even if you factor in his 32 rushing yards, you only bring him to 71 yards of Total offense.

[Oldfan]

Formula: 39 yards, 1 INT, 46.7% = clutch QB.

You ducked both questions.