NFL QB Rankings & Ratings (ASF Formula), 2008 Week 1

[Atlanta Skins Fan]

Using only stats available on NFL.com, I've come up with an NFL QB Rating formula that is superior to the official NFL formula. I tested this successfully on 2007 stats, and now I've run it on 2008 Week 1 stats, shown below.

Week 1 ratings based on one game of stats will be distorted for any system. However, I think it is clear that my formula is much more accurate than the official NFL system.

I won't share the formula itself, but I will share the results of the formula from time to time. If anyone doubts that this is an actual formula, I can show it to a trusted mod for verification.

The general concept is to capture key stats not captured by the official NFL formula. There are a small number of source variables (raw stats), and the formula converts these stats into a single rating. Unlike the NFL version, this rating is not capped at the high end, which is why some players have unusually high ratings.

My motivation for this was my irritation with how the official NFL system distorts reality with artificially high QB ratings in some cases. For example, Jason Campbell got a rating of 81.2 for his performance last week. That's obviously false from a comprehensive view of performance, and the fault is the NFL formula.

This is a fixable problem. My formula is one solution.

NFL QB Ratings (ASF Formula), 2008 Week 1

Rank Player Rating

1 Donovan McNabb 236

2 Tony Romo sits to pee 207

3 Jay Cutler 193

4 Drew Brees 150

5 Jake Delhomme 123

6 Philip Rivers 113

7 Peyton Manning 106

8 Trent Edwards 105

9 Ben Roethlisberger 103

10 Kurt Warner 103

11 Jon Kitna 93

12 Matt Cassel 93

13 Aaron Rodgers 92

14 Chad Pennington 91

15 Jeff Garcia 86

16 Joe Flacco 76

17 Eli Manning 73

18 Kyle Orton 58

19 Brett Favre 50

20 Matt Schaub 45

21 Tavaris Jackson 35

22 JaMarcus Russell 33

23 Jason Campbell 32

24 Matt Hasselbeck 32

25 Brodie Croyle 28

26 Vince Young 23

27 Derek Anderson 15

28 David Garrard 14

29 Carson Palmer 10

30 J.T. O'Sullivan 8

31 Marc Bulger 0

Official NFL QB Ratings, 2008 Week 1

Rank Player Rating

1 Ben Roethlisberger 147.0

2 Jay Cutler 137.5

3 Donovan McNabb 131.0

4 Brett Favre 125.9

5 Philip Rivers 125.1

6 Drew Brees 124.9

7 Matt Cassel 116.0

8 Aaron Rodgers 115.5

9 JaMarcus Russell 111.1

10 Tony Romo sits to pee 103.6

11 Jon Kitna 103.3

12 Trent Edwards 95.8

13 Kurt Warner 93.3

14 Kyle Orton 83.4

15 Chad Pennington 82.6

16 Jake Delhomme 82.1

17 Peyton Manning 81.8

18 Jason Campbell 81.2

19 J.T. O'Sullivan 80.2

20 Matt Schaub 75.6

21 Derek Anderson 74.0

22 Marc Bulger 72.3

23 Jeff Garcia 71.3

24 Brodie Croyle 69.6

25 David Garrard 68.2

26 Joe Flacco 63.7

27 Eli Manning 61.1

28 Tavaris Jackson 59.0

29 Matt Hasselbeck 53.9

30 Vince Young 45.6

31 Carson Palmer 32.3

[ttr77]

I appreciate what you're trying to do, but how can anyone possibly discuss a system about which no one knows the details?

[brandymac27]

"The general concept is to capture key stats not captured by the official NFL formula. There are a small number of source variables (raw stats), and the formula converts these stats into a single rating. Unlike the NFL version, this rating is not capped at the high end, which is why some players have unusually high ratings."

Can you share some of these variables with us, or is this giving away too much info?

[brandymac27]

I appreciate what you're trying to do, but how can anyone possibly discuss a system about which no one knows the details?

Yeah, I feel the same way. I don't know enough to really comment on this.

[Atlanta Skins Fan]

I appreciate what you're trying to do, but how can anyone possibly discuss a system about which no one knows the details?

OK, fair enough. The formula takes the NFL QB rating and factors four additional variables:

• First downs from QB passing
  
• QB Sacks
  
• QB Fumbles
  
• Win / Loss for starter, or substitute with 18+ attempts
  

These factors boost the value of total production (first downs) over simple efficiency, which can be deceiving. However, conservative game management is also rewarded, through credit for the win. High production along with a win is rewarded with extra weight. Important negative factors are also considered -- sacks and fumbles.

The formula nails some QBs with deceiving NFL QB ratings. For example, Jason Campbell threw efficiently, but with little production, in a loss. Quarterbacks who pad their QB rating by taking sacks are nailed. Quarterbacks who fumble are penalized heavily, much as they are in the NFL formula for throwing interceptions.

More positively, the formula also rewards QBs with high production while avoiding sacks and fumbles, to a greater degree than the NFL formula. This reflects QB dominance as opposed to efficiency.

[ttr77]

The formula nails some QBs with deceiving NFL QB ratings. For example, Jason Campbell threw efficiently, but with little production, in a loss. Quarterbacks who pad their QB rating by taking sacks are nailed. Quarterbacks who fumble are penalized heavily, much as they are in the NFL formula for throwing interceptions.

More positively, the formula also rewards QBs with high production while avoiding sacks and fumbles, to a greater degree than the NFL formula. This reflects QB dominance as opposed to efficiency.

Seems like a good system. Now if we could only find a way to incorporate the 'garbage-time yards' factor. This is still my pet-peeve. JC had a good percentage of his yards and completions in the last few minutes of the game when the Giants were more than happy to allow for 8-10 yard passes. It happens in almost every game in which there is more than a 1 score lead.

[Atlanta Skins Fan]

Seems like a good system. Now if we could only find a way to incorporate the 'garbage-time yards' factor. This is still my pet-peeve. JC had a good percentage of his yards and completions in the last few minutes of the game when the Giants were more than happy to allow for 8-10 yard passes. It happens in almost every game in which there is more than a 1 score lead.

The Win/Loss factor solves that problem. It's a huge factor in the rating. Garbage time production in a loss is absolutely not rewarded. See Campbell's rating for demonstration.

[ttr77]

The Win/Loss factor solves that problem. It's a huge factor in the rating. Garbage time production in a loss is absolutely not rewarded. See Campbell's rating for demonstration.

Well then, I wholeheartedly endorse your system.

[Oldfan]

The NFL's QB rating should not be taken seriously because we can't measure something unless it can be isolated. We can't take a measurement of factors A - J and claim that it measures only A.

A. The quality of the QB's play

B. The quality of the passing scheme

C. The quality of the play calling

D. The quality of the receivers

E. The quality of his pass protection

F. The quality of the running game to keep the defense honest

G. The quality of the defenses the QB faced

H. The average starting position given the offense

I. The frequency of short-field advantages

J. The frequency of come-from-behind situations faced by the QB

[Atlanta Skins Fan]

The NFL's QB rating should not be taken seriously because we can't measure something unless it can be isolated. We can't take a measurement of factors A - J and claim that it measures only A.

I somewhat agree, with three important caveats:

• Don't let the perfect stand in the way of the good. One should not let the impossibility of perfection prevent the use of good and useful assessments based on comprehensive statistics.
  
• Since there exists an NFL QB rating that is taken as bible, anything that makes this rating more accurate is worth the effort, as a corrective.
  
• The worst rating system of all is individual bias, which everyone has, including NFL coaches. Since you can't argue bias, it is useful for everyone to have some metrics in common, as a reality check. Otherwise stupid decisions (such as sticking by bad QBs) persist longer than they should, due to the forces of ego and power.
  

[jrockster21]

The Win/Loss factor solves that problem. It's a huge factor in the rating. Garbage time production in a loss is absolutely not rewarded. See Campbell's rating for demonstration.

You call what Campbell did in the second half garbage time stats?? We were down 9, with a legitimate shot at coming back; FAR from what should be discounted. Garbage time stats are when your team is down 3 or 4 scores with half a period left - i.e. no possible way to come back. With slightly better clock management, the Skins would have had that game in their sights.

[jrockster21]

The Win/Loss factor solves that problem. It's a huge factor in the rating. Garbage time production in a loss is absolutely not rewarded. See Campbell's rating for demonstration.

We can't know that unless you share the formula. What is the big deal? Are you hoping to sell this formula and make money off of it? OLS

[Oldfan]

I somewhat agree, with three important caveats:

• Don't let the perfect stand in the way of the good. One should not let the impossibility of perfection prevent the use of good and useful assessments based on comprehensive statistics.
  
• Since there exists an NFL QB rating that is taken as bible, anything that makes this rating more accurate is worth the effort, as a corrective.
  
• The worst rating system of all is individual bias, which everyone has, including NFL coaches. Since you can't argue bias, it is useful for everyone to have some metrics in common, as a reality check. Otherwise stupid decisions (such as sticking by bad QBs) persist longer than they should, due to the forces of ego and power.
  

In a recent article on NFL.com, they did a survey of coaches and GMs in an attempt to discover which statistics they found useful. None used the QB rating. The only value I see for it is in debate in forums like this where no one knows what they're talking about anyway.

Only Jim Zorn is capable of grading Jason Campbell, and only after reviewing game film. The plays are his design, so only he can tell who to blame when they break down.

[Xameil]

You call what Campbell did in the second half garbage time stats?? We were down 9, with a legitimate shot at coming back; FAR from what should be discounted. Garbage time stats are when your team is down 3 or 4 scores with half a period left - i.e. no possible way to come back. With slightly better clock management, the Skins would have had that game in their sights.

completely agree... who decides what is garbage time? definitely not garbage time second half of the skins game. Gints went to prevent yeah, but we still had a shot. 9 points down 6 mins left...hell we had a shot 9 points down 2 mins left....

monday night mircale anyone?

[Atlanta Skins Fan]

completely agree... who decides what is garbage time? definitely not garbage time second half of the skins game. Gints went to prevent yeah, but we still had a shot. 9 points down 6 mins left...hell we had a shot 9 points down 2 mins left....

If you are suggesting that I am "deciding" garbage time, I am not. The formula rewards QB production (first downs) and wins, and punishes negative plays (sacks and fumbles). These are not factors in the NFL QB rating.

Campbell already got credit for passing efficiency in the second half. That is still reflected in the rating. His failure to throw for first downs and his failure to get a win did mark down the rating. But, it could have been worse. Since he took only one sack and did not fumble, his rating could have been worse. I think the rating is pretty accurate, all things considered.

[MUSICCITY]

Troy Aikman uses a system that includes YPA, because the west coast offense skew the numbers. In the west coast the passing game is an extension of the running game. It appears neither system takes this into account.

[theTruthTeller]

Or you could go to footballoutsiders.com which uses brain-numbing analysis of each individual play to come up with ratings.

[Oldfan]

Troy Aikman uses a system that includes YPA, because the west coast offense skew the numbers. In the west coast the passing game is an extension of the running game. It appears neither system takes this into account.

Yards per attempt (YPA) and yards per carry (YPC) correlate well with winning. However, they should be thought of as team stats, not individual stats for the same reason I mentioned earlier: they don't isolate individual performances.

[Atlanta Skins Fan]

Troy Aikman uses a system that includes YPA, because the west coast offense skew the numbers. In the west coast the passing game is an extension of the running game. It appears neither system takes this into account.

YPA is reflected in the NFL QB rating formula (which factors both yards and attempts). That QB formula is the base number for my own formula, which adjusts the NFL factor for additional variables.

Among my added factors, total production by first downs per game played will also reward QBs throwing for longer YPA. However, my factor rewards sustained drives more than long bombs. A QB who throws four 15-yard completions in a drive will do better in my formula than a QB who throws one 60-yard completion. This rewards game control and lessens the distortions that occur when long gains occur due to YAC instead of sustained QB dominance during a drive.

[Oldfan]

Or you could go to footballoutsiders.com which uses brain-numbing analysis of each individual play to come up with ratings.

Their work is about the best available. It's a work-in-progress, though. They don't have the answers yet, but at least they understand the problems.

[Atlanta Skins Fan]

Originally Posted by theTruthTeller

Or you could go to footballoutsiders.com which uses brain-numbing analysis of each individual play to come up with ratings.

Their work is about the best available. It's a work-in-progress, though. They don't have the answers yet, but at least they understand the problems.

Yes, they do excellent work. In fact I intend to improve my own formula with the additional factor of DVOA, taken from FootballOutsiders.

However, my goals are different. I am trying to create a QB formula that uses the same scale as the NFL system, but with superior accuracy.

If my final formula is as strong as I expect it will be, I will then test how reliable it is on short samples. For example, is it possible to assess young QB performance accurately after only a few games?

In general I am interested in the simplest possible formulas that are accurate and also predictive of future performance.

[Thinking Skins]

I somewhat agree, with three important caveats:

• Don't let the perfect stand in the way of the good. One should not let the impossibility of perfection prevent the use of good and useful assessments based on comprehensive statistics.
  
• Since there exists an NFL QB rating that is taken as bible, anything that makes this rating more accurate is worth the effort, as a corrective.
  
• The worst rating system of all is individual bias, which everyone has, including NFL coaches. Since you can't argue bias, it is useful for everyone to have some metrics in common, as a reality check. Otherwise stupid decisions (such as sticking by bad QBs) persist longer than they should, due to the forces of ego and power.
  

So what's the method you're using to judge that your formula is 'better'? Better is a subjective word. So while you may find QBs who are better at getting First downs or continuing drives, it doesn't really tell who the better QB is. And who's to say that you've given the right weight to each of your factors? One thing about the QB rating system is that its a nice formula that, while not perfect, gives a nice 'idea' how how a QB performed - particularly to fans.

Really all the QB rating system is is a tool for fans to judge QBs and a method for the media to rank QBs. Coaches spend much more time focusing on game film and actual scenario representation to judge QBs.

So while I appreciate any way to judge QBs, when we start to argue which formula is a better one to judge QBs, we sound more like math geeks who are trying to use football to prove that we're good at math.....and as much as I love the whole "Revenge of the Nerds" theme here at Extremeskins (especially over the "Die Redskins With a Vengance"), I'm not for getting into an argument of why one formula is better than another - particularly when the formula you're trying to knock down is so widely accepted.

[Oldfan]

Yes, they do excellent work. In fact I intend to improve my own formula with the additional factor of DVOA, taken from FootballOutsiders.

However, my goals are different. I am trying to create a QB formula that uses the same scale as the NFL system, but with superior accuracy.

If my final formula is as strong as I expect it will be, I will then test how reliable it is on short samples. For example, is it possible to assess young QB performance accurately after only a few games?

In general I am interested in the simplest possible formulas that are accurate and also predictive of future performance.

Tough task.

No one has as yet come up with a team rating that does well predicting future performance. If they could, they'd make a fortune.

It seems to me that predicting individual performance is about 20 times as difficult since football is truly a team sport.

Good luck with it, though.

[Atlanta Skins Fan]

So what's the method you're using to judge that your formula is 'better'? Better is a subjective word. So while you may find QBs who are better at getting First downs or continuing drives, it doesn't really tell who the better QB is.

My method is iterative. First I look at QB ratings and notice a divergence from my own view of QB performance, using very broad observations that most people will share. For example, Campbell got an 81.2 NFL QB rating for game 1. And yet, the universal opinion (including his coach's opinion) was that his performance was not good.

What was not good about his performance? Well, he did not throw for first downs (only five) and obviously the team did not win the game, despite the score being close enough for a comeback all second half. So, winning the game is an important goal, and throwing for first downs is a fairly reliable measure of production, as opposed to accuracy / efficiency.

I looked at a variety of other QBs and tried to determine what was missing from the QB rating when it was "off." For example, Jon Kitna took 51 sacks in 2007, but had the same 80.9 QB rating that he would have had if he took no sacks at all. He also fumbled 17 times, while Carson Palmer fumbled only 5 times while throwing for more attempts.

To give another example, Eli Manning had a mediocre 73.9 QB rating, but led his team to the playoffs, won four games in the playoffs, and almost-single-handedly got the Super Bowl win on the final drive. It seems that if one wants to develop a formula to rate QBs for their aptitude in winning, actually *winning games* should be a factor. Otherwise Eli Manning looks just like Jason Campbell, when we know that in 2007, Campbell on multiple occasions would *lose* games on the final drive with unforced interceptions.

The formula is the result of many iterative adjustments resulting from these additional factors and varying their weight. After tweaking the formula for 2007, I ran it today on 2008 Game 1, with no alteration in the formula. The test for 2008 is to see if the formula holds up without further tweaks. I expect it will do better than the official NFL formula, but it still lacks defense-related adjustments, which is why I want to add DVOA.

[Atlanta Skins Fan]

I'll save the recap of 2007 for another day, but there are some shockers in the final rankings. For example, Sage Rosenfels blew away Matt Schaub. Uh-oh.

At the very top, Tom Brady had an extraordinary rating (212), vastly exceeding the #2 rated player, Peyton Manning (137).

My top 5 from 2007:

1. Brady (212)
   
2. Manning (137)
   
3. Romo sits to pee (136)
   
4. Favre (135)
   
5. Brees (126)
   

Actual NFL rankings:

1. Brady (117.2)
   
2. Roethlisberger (104.1)
   
3. Garrard (102.2)
   
4. Manning (98.0)
   
5. Romo sits to pee (97.4)