Welcome to the 15th annual NFL Prediction Tracker Awards. The purpose of these awards is
to help raise the publicity for those systems that have been superior in various qualities of
interest. The awards will be based entirely on the numbers I've monitored with my prediction
tracker web pages. In 2014 I tracked 73 rating/pediction systems that had data in he NFL for
at least half of the season. This was up 2 from the prior season.  This year I am adding plots
that tracked the performance of each system as the season progressed.  The winner of the
category will be in a bold black line.


BEST STRAIGHT UP WINNERS (Entire Season)

Winner: Dokter Entropy

straight up plot

 Jon Dokter's Entropy ratings at timetravelsports.com had the best record straight up
for the entire season, 190-76, 71.429%.  Dokter had a comfortable four game lead over
the nearest competitor, StatFox.  The 71% this season is a very good mark, it does not
reach the record of 72% set in 2005 but it still looks to be perhaps in the top 2 or
3 seasons alltime. The winner is tyically somewhere in the mid to upper 60s.  The updated
line finished in 63rd place with only 63.9% of the favorites winning. That has to be
one of the worst ever years for the line.  Like last year the opening line did better
than the midweek line which did better than the updated line.  Did that poor performance
predicting winners allow for good ATS numbers?
 From the above plot, Dokter was clearly among the best or the best for the entire season.
Another thing I have noticed from adding these plots for both college and pro is that
they all seem to level off around five weeks into the season.  For college I had assumed
that might have been the switch from nonconference to conference play.  But it may just be
how long it take systems to get dialed in to what is taking place in the current season.


Winner 2014: Dokter Entropy
Winner 2013: Ed Bemiss
Winner 2012: Sagarin Pure Points (predictive)
Winner 2011: Nationalsportsranking.com
Winner 2010: LVSC opening line
Winner 2009: JFM Power Ratings, Sportrends
Winner 2008: Ken Ashby
Winner 2007: Sonny Moore
Winner 2006: Least Absolute Value Regression
Winner 2005: Vegas Line (record 72.28%)
Winner 2004: Least Absolute Value Regression
Winner 2003: Beck Elo
Winner 2002: Herman Matthews
Winner 2001: JFM Power Ratings, ARGH Power Ratings
Winner 2000: CPA Rankings
Winner 1999: Kenneth Massey Ratings




  BEST AGAINST THE SPREAD (Entire Season)

Winner:  Computer Adjusted Line
Winner: Turnover adjusted least squares regresssion

ATS plot

 Some consider the true measure of a ranking system to be its record against the Vegas line.
Although that record for every game is probably not the best measurement but that is what I have.
This award is a little complicated this years and I am nameing co-champions. The two highest
systems can be tossed out because they do not have enough games to qualify.  The third highest
record was the updated line.  This is a little unusual, how can the updated line be the best
against the line?  This is because I now use the midweek line, usually taken Thursday morning,
to be the line of record.  The problem with giving the award to the line is that the updated
line and midweek line were equal about half the time.  But when the line moved off the updated
line it was correct direction almost 59% of the time.  In fourth place, and a co-winner this
year is the computer adjusted line.  This has some of the same problem as the updated line but
did have a prediction in two thirds of the games.  So the first co-winner is the computer adjusted
line with a record of 106-75, 58.564%.  In 5th place and our second co-winner is turnover
adjusted least squares regression, 147-113, 56.538%.  This was the best system that had predictions
available for every game.  Pythagorean ratings was also very close behind.  These numbers are down
a little bit from last seaon but then last season was the all time record.
Under the previous award I mentioned the line had a poor season straight up.  A possibly result
of that is that a total of 55 sytems did better than 50% ATS.  And as we will see later, that
number was even larger for the second half numbers.   The worst record for full season data was
only 47.6%.


Winner 2014: Computer adjusted line, Turnover adjusted least squares regresssion
Winner 2013: Lou St. John  (record 59.449%)
Winner 2012: Least squares regression
Winner 2011: Pi-Rate Ratings
Winner 2010: Sagarin Predictor
Winner 2009: CPA Rankings.
Winner 2008: Nutshell Retro, Entropy System
Winner 2007: Sagarin Predictive
Winner 2006: Brian Gabrielle (record against updated line 59.0%)
Winner 2005: Least squares with team HFA
Winner 2004: Least Absolute Value Regression
Winner 2003: CPA Retro Rankings
Winner 2002: FreeSportsPlay.com
Winner 2001: CPA Rankings
Winner 2000: Yourlinx
Winner 1999: PerformanZ Ratings





  SMALLEST MEAN ABSOLUTE ERROR (Entire Season) 

Winner: Massey Ratings

ATS plot


  Deviation from the game score is the absoulte difference between the game prediction
and the actual result.  A value of zero would mean the score difference is predicted
exactly.  One property of a good system would be to minimize the system's average
game deviation.
  Last year saw a new record and broke below 10.  This year Massey Ratings is the lowest
but over a point higher at 11.2030.  Several systems had better numbers but were all
second half systems.  The computer adjusted line and system median were the only other
systems to do better than the updated line.
  Like many of the plots, the numbers appear to level off starting around week 5, with
only slow improvment from then on out. Although throw out good week 1 and bad week 4 and
they numbers would be fairly flat all season.



Winner 2014: Massey Ratings
Winner 2013: Ed Bemiss (record 9.7236)
Winner 2012: Vegas Line (updated)
Winner 2011: Dokter's Entropy System
Winner 2010: LVSC Opening line
Winner 2009: Vegas Line (updated)
Winner 2008: Ken Ashby
Winner 2007: Computer Adjusted Line
Winner 2006: Brian Gabrielle
Winner 2005: Vegas Line (updated)
Winner 2004: Vegas Line (updated)
Winner 2003: Vegas Line (updated)
Winner 2002: FreeSportsPlay.com
Winner 2001: Vegas Line (updated)
Winner 2000: Vegas Line (updated)
Winner 1999: Vegas Line (updated)





 SMALLEST BIAS (Entire Season)  

Winner: Sagarin Points Elo

ATS plot

  Bias is a little different from deviation.  Deviation measures the distance
between a prediction and the actual result.  Bias combines distance and location
of the prediction.   Bias measures whether the predictions are too high or too low.
So if a system has an average bias of +0.25 that means that on average the system
gave 0.25 points too many to the home team.
  Sagarin Points Elo had the smallest bias this season at 0.01440 points per game.
Sagarin's Elo system also won with a record low 0.001 in 2005.  But one thing you will
see with a lot of these categories, is other than the score accuracy neing related to the
line based systems, there are not a lot of multi-time winners.
 The biases were relatively low this year, with the majority being within plus or minus
one point of the actual result.  We typically see the majority fall on one side, with the
last several years seeing negative biases.


Winner 2014: Sagarin Points Elo
Winner 2013: Game Time Decision
Winner 2012: PerformanZ Ratings
Winner 2011; Lee Burdorf
Winner 2010: Stephen Kerns
Winner 2009; System Median
Winner 2008: ARGH Power Ratings
Winner 2007: CPA Rankings
Winner 2006: Tom Benson
Winner 2005: Sagarin Elo (record 0.001)
Winner 2004: System Average
Winner 2003: CPA Rankings
Winner 2002: Sagarin Pure Points
Winner 2001: ARGH Power Ratings
Winner 2000: Pigskin Index
Winner 1999: Flyman Ratings




 MOST ACCURATE SYSTEM - SMALLEST MEAN SQUARE ERROR (Entire Season)

Winner: Computer Adjusted Line

 This award is based on mean square error. Mean square error takes
into account both deviation and bias and is a commonly used measure
of evaluating estimators.
 The computer adjusted line had the lowest mean square error for the 2014
season.  The value was very high, 208.69100.  Compare that to last year
when we saw a record 157.  The updated and opening line were seond and
third.  Pigskin index was the system not tied to the line.

Winner 2014: Computer Adjusted Line
Winner 2013: Lou St. John (record 157.3140)
Winner 2012: Vegas Line (updated)
Winner 2011: Dokter Entropy System
Winner 2010: Computer Adjusted Line
Winner 2009: Vegas Line (updated)
Winner 2008: Computer Adjusted Line
Winner 2007: Computer Adjusted Line
Winner 2006: Computer Adjusted Line
Winner 2005: Vegas Line (updated)
Winner 2004: Vegas Line (updated)
Winner 2003: Vegas Line (opening)
Winner 2002: FreeSportsPlay.com
Winner 2001: Vegas Line
Winner 2000: Pigskin Index




            SECOND HALF AWARDS 


Some systems donít begin publishing rankings until every team has played a couple of games, while
the season-long systems often rely on preset starting values in the early part of the season. For
those reasons, I like to look at the results over the second half of the season when these systems
have had time to become 'burned in' to  the season's data. Second half data consists of all games
from week 10 through the Super Bowl.



  BEST STRAIGHT UP WINNERS (Second Half)

Winner: Logistic Regression

straight up plot

  Logistic regression had the most wins in the second half, 96-37, 72.18%.  This was two games better
than Pythagorean ratings and Dokter Entropy.  From the graph about it appears that Logistic regression
cruised to an easy victory, it was in first place every week.  What is notable about logistic regression
is that the only inputs that it uses is the schedule and who won and lost.  So it uses the very minimal
data, much like the old BCS systems. So it is not a system you would expect to be within two games of
a record in predicting winners.
The updated line finished way back in 65th place, almost 10 percentage points worse that logistic
regression. So the line was consistantly poor all season.


Winner 2014: Logistic Regression
Winner 2013: Ed Bemiss
Winner 2012: Sonny Moore, Kasulis Enhanced Spread
Winner 2011: Turnover adjusted least squares
Winner 2010: LVSC opening line
Winner 2009: The Vegas line, Computer adjusted line, Ashby AccuRatings, StatFox, Upset Report, CPA Rankings
Winner 2008: Nutshell Girl
Winner 2007: Computer Adjusted Line, Vegas Line (updated)
Winner 2006: Least Absolute Value Regression
Winner 2005: Vegas Line (updated), Nutshell Sports (record 100-37, 72.99%)
Winner 2004: Pigskin Index
Winner 2003: JFM Power Ratings, Kasulis Enhanced Spread, Beck Elo
Winner 2002: Least Absolute Value Regression, JFM Power Ratings, CPA Rankings
Winner 2001: Pythagorean Ratings, Elo Ratings, Ken Massey Ratings
Winner 2000: PerformanZ Ratings, Scoring Effeciency
Winner 1999: Ken Massey, Ed Kambor, Pythagorean Ratings




  BEST AGAINST THE SPREAD (Second Half)

Winner:  Computer Adjusted line
Winner:  Winner: Least Squares regression

straight up plot


 I'll split this between Least Squares regression, 81-50, 61.832% and the computer adjusted line,
55-29, 65.476%.  The reason for the split is that the percentage for the computer adjusted line
is an all time record.  But perhaps it deserves an asterix since it gave a no play for 37% of
the games.  But it had a remarkable second half.  It was a very good season for ATS numbers.
In the second half only 7 systems failed to do bettert han 50% and the median results was over 53%.
The plot looks a like a big tangle of lines, which you might expect if the picking ATS is
a random event.


Winner 2014: Least Squares regression, computer adjusted line (record  65.476% in 63% of games)
Winner 2013: Sonshine forecast (record 68-39, 63.551%)
Winner 2012: Sagarin Predictor, Sagarin Overall
Winner 2011: Kambour Ratings
Winner 2010: Sagarin Predictor, Edward Kambour
Winner 2009: Grid Iron Gold
Winner 2008: Ken Ashby
Winner 2007: Least Squares with Team Specific HFA
Winner 2006: CPA Rankings
Winner 2005: Tom Benson
Winner 2004: ARGH Power Ratings
Winner 2003: Logistic Regression, Viacheslav Ugolnikov
Winner 2002: Least Absolute Value Regression (record against updated line 78-52, 60.0%)
Winner 2001: Stat Fox
Winner 2000: Pythagorean Ratings
Winner 1999: Least squares using team specific home field advantages





  SMALLEST MEAN ABSOLUTE ERROR (Second Half)

Winner: Least Squares regression

straight up plot

 Least Squares Regression had the lowest mean error in the seond half, 10.8700.
This is over a point lower than last year's record setting number.  Eleven systems
did better than the line.


Winner 2014: Least Squares regression
Winner 2013: Ironrank.com (record  9.7386)
Winner 2012: Lou St. John
Winner 2011: Turnover adjusted least squares
Winner 2010: Massey Ratings
Winner 2009: Vegas Line (updated)
Winner 2008: The Entropy System
Winner 2007: Vegas Line (updated)
Winner 2006: Jeff Self
Winner 2005: Vegas Line (updated) (record 9.9489)
Winner 2004: Least Squares Regression
Winner 2003: Vegas Line (updated)
Winner 2002: FreeSportsPlays.com
Winner 2001: Vegas Line (updated)
Winner 2000: Pythagorean Ratings



  SMALLEST BIAS (Second Half)    


Winner: Locksmith picks

straight up plot

 Locksmith picks had the lowest average bias over the second half with a value of -0.14390.
That doesn't seem too close to zero but it was one of only six systems with a value less than .5.
The numbers were overwhelmingly on the positive side with the median being around +1 point.

Winner 2014: Locksmith picks
Winner 2013: Covers.com
Winner 2012: Nutshell Sports
Winner 2011: Lee Burdorf
Winner 2010: CPA Retro Rankings
Winner 2009; Game Time Decision
Winner 2008: CPA Rankings
Winner 2007: Pigskin Index (record 0.0077)
Winner 2006: CPA Retro Rankings
Winner 2005: Kasulis Enhanced Spread
Winner 2004: Tom Benson Gridmarks
Winner 2003: CPA Rankings
Winner 2002: StatFox.com
Winner 2001: Monte Carlo Markov Chain
Winner 2000: Flyman Performance Ratings
Winner 1999: Flyman Performance Ratings




  MOST ACCURATE SYSTEM - SMALLEST MEAN SQUARE ERROR (Second Half)


Winner: Iron Rank

straight up plot

  Ironrank.com wins the second half with a mean square error of 207.425.  Ironrank was
the only system to do better than the line.


Winner 2014: Ironrank.com
Winner 2013: Steven Jens (record  158.0850)
Winner 2012: Vegas Line (updated)
Winner 2011: Turnover adjusted least squares
Winner 2010: Jeff Sagarin
Winner 2009: Vegas Line (updated)
Winner 2008: Hank Trexler
Winner 2007: Vegas Line (updated)
Winner 2006: CPA Rankings
Winner 2005: Vegas Line (updated)
Winner 2004: Computer Adjusted Line
Winner 2003: Viacheslav Ugolnikov
Winner 2002: FreeSportsPlay.com
Winner 2001: Hank's Power Ratings
Winner 2000: Pythagorean Ratings






   BEST PREDICTIVE SYSTEM (Entire Season)



Winner: Pythagorean Ratings

To come up with the best overall predictive system I give each system
points for how well they do in all of the above categories.  I then sum up
the points and the system with the highest total is dubbed predictive system
of the year.  So this award goes to the system that is the most well rounded.
One flaw can totally take a system out of the running.
  For those that had pedictions for the full season, or at least most of the
season, the best overall predictive system was Pythagorean Ratings.  I don't
remember noticing this during the season by this system was good in everything.
It was top 10 in every category except absolute error where it was 14th. The closest
competitors were the mean/median or line based, computer adusted and
updated line.  Massey and Dokter were the next best independent ratings but they
were a good 40% behind a system that uses nothing but a teams point for/against totals.


Winner 2014: Pythagorean Ratings
Winner 2013: Ed Bemiss
Winner 2012: Sagarin Pure Points
Winner 2011: nationalsportsrankings.com
Winner 2010: LVSC opening line
Winner 2009: Vegas Line
Winner 2008: Ken Ashby
Winner 2007: Jeff Sagarin
Winner 2006: Brian Gabrielle
Winner 2005: Vegas Line
Winner 2004: Least Absolute Value Regression
Winner 2003: Beck Elo
Winner 2002: Matthews Grid
Winner 2001: CPA Rankings
Winner 2000: Ed Kambour




  BEST PREDICTIVE SYSTEM (Second Half)


Winner:  GameTimeDecision

  The top point getter in the second half was actually the computer adjusted line but I prefer
to give these things to something other than the line unless there is not a decent choice.  This
year GameTimeDecision was a fairly close second.  They were almost very good.  Top 6 in everything
except for bias.  Pytagorean Ratings weren't too far behind.

Winner 2014: GameTimeDecison
Winner 2013: IronRank.com
Winner 2012: Sagarin Pure Points
Winner 2011: Edward Kambour Ratings
Winner 2010: LVSC opening line
Winner 2009: Game Time Decision
Winner 2008: Dokter Entropy System
Winner 2007: Cover 81
Winner 2006: CPA Retro
Winner 2005: Hank Trexler
Winner 2004: ARGH Power Ratings
Winner 2003: CPA Rankings
Winner 2002: Kenneth Massey
Winner 2001: Monte Carlo Markov Chain
Winner 2000: Pythagorean Ratings