Welcome to the 14th annual NCAA PT Awards.
The purpose of these awards is to help raise the publicity for those systems that have been
superior in various qualities of interest.  Most awards will be based entirely on the numbers
I've monitored with my prediction tracker web pages. For the 2014  season I followed the
weekly performance of 72 computer rating  systems. This is up 2 from last year.
2014 was an interesting year in that it ended with a four team playoff that crown Ohio State as
the national champion.  This year I am going to experiment with adding plots of the week to
week performance in each of the categories.

  MOST STRAIGHT UP WINNERS (Entire Season)

Winner: Ted Thompson CAL

straight up plot

  This year I collected the predictions of 760 games involving two FBS teams.
This year Ted Thompson's Computer Ajusted Line had the best straight up record, 562-198, 73.947%.
Thompson also had a system that won last year with a record 78%.  So it was a down year. In terms
of 'games back', this year would have been 24.5 games behind last years record.  We had seen new
records for two consevutive years until now. However, in historical terms, this was a fairly
average season.  It should be noted that Thompson CAL is very highly correlated with the line, and
it is the updated line that finished in second place two games behind. Thompson's ATS system was also
two games back.  This season the updated line out performed the opening line by 13 games.  So the line
definitely moved towards the winner this season.
 In the plots the category winner will be denoted with the dark black line. You can enlarge the plots
by right clicking and selecting view image. In the above plot you can see that week one was a fairly
rough week, the the numbers stabilize a little higher over the non-conference schedule before dropping
and staying relatively flat through the rest of the season.  We also had a very poor bowl season.

2014 Winner: Ted Thompson CAL
2013 Winner: Ted Thompson SPRS  (record 78.095%)
2012 Winner: Hank Trexler
2011 Winner: Covers.com, Tempo Free Gridiron
2010 Winner: Dokter Entropy
2009 Winner: Pigskin Index
2008 Winner: System Average, System Median
2007 Winner: StatFox
2006 Winner: Thompson SPRS, system average, Sagarin predictive, & Sagarin overall
2005 Winner: Pigskin Index
2004 Winner: Jeff Sagarin's Overall Rating
2003 Winner: Edward Kambour & Jeff Sagarin's Overall Rating
2002 Winner: ARGH Power Ratings
2001 Winner: Massey Ratings
2000 Winner: ARGH Power Ratings
1999 Winner: Vegas line



  SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)

Winner:  Updated Line

absolute deviation

  Deviation from the game score is the difference between the game prediction
and the actual result.  A value of zero would mean the score difference is predicted
exactly.  One property of a good system would be to minimize the system's average
game deviation.
  The updated line had the smallest average absolute error this year, 12.477 points.
Think about that for a minute, the very best out of 72 systems is off by an average
of twelve and half points per game.  So predictions for NCAA football games have a
lot of room for improvement. A lot of room.  The numbers have worsened for three years
in a row now.  The computer adjusted line had won this category for the last five years,
that system was third this season.  In all the top 7 systems are all based off the line.
You have to look to 8th place Dokter Entropy before you find a system not heavily tied
to the line.
  From the plot you can see that the line pretty much was the leader for the entire season.
Again you can see the evidence that week one was a difficult week. The numbers then continued
to improve each week through the non-conference season, at which point the systems leveled
off and started to slowly converge with each other.

2014 Winner: Updated Vegas Line
2013 Winner: Computer Adjusted Line
2012 Winner: Computer Adjusted Line
2011 Winner: Computer Adjusted Line
2010 Winner: Computer Adjusted Line
2009 Winner: Computer Adjusted Line (record 11.26)
2008 Winner: Vegas Line
2007 Winner: Vegas Line
2006 Winner: Computer Adjusted Line
2005 Winner: Vegas Line
2004 Winner: Vegas Line
2003 Winner: Vegas Line
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
1999 Winner: Vegas Line




  BEST AGAINST THE SPREAD (Entire Season)


Winner:  Laffaye RWP

ats

  Beating the spread is not the number one goal for most computer
rating systems.  But it is something that the average person likes
to look at and probably the most popular subject that I get emails
about.  I know that looking at all the games is probably not the
most meaningful of measurements but it is what I have been reporting.
The line of record is taken midweek.  Usually Wednesday afternoon or
Thursday morning.
  This year the system with the best record against the spread across
all of their games was Laffaye RWP, 400-340, 54.054%.  Two systems were
higher overall but they only had data over the second half of the season.
From the plot above you can see that RWP was very consistent from week
six on.  The numbers as a whole look like they were getting better as
the season went on but for the most part cluster around 50%.
Nobody did really bad, the worst full season performance was only 47.7%


2014 Winner: Laffaye RWP
2013 Winner: CPA Rankings (record 56.906%)
2012 Winner: Laz Index, Computer Adjusted Line
2011 Winner: Dunkel Index
2010 Winner: Catherwood Ratings
2009 Winner: Covers.com
2008 Winner: Stephen Kerns
2007 Winner: Nutshell Sports
2006 Winner: Harmon Forecast  (record 56.4% against updated line)
2005 Winner: Keeper's College Football Ratings
2004 Winner: Edward Kambour
2003 Winner: Edward Kambour
2002 Winner: Dunkel Index
2001 Winner: BMC Picks
2000 Winner: Dunkel Index
1999 Winner: Average across all systems



  SMALLEST AVERAGE GAME BIAS (Entire Season)

Winner:  Billingsley Report

Bias



  Bias is a little different from deviation.  Deviation measures the
distance between a prediction and the actual result.  Bias combines
distance and location of the prediction.   Bias measures whether the
predictions are too high or too low.  So if a system has an
average bias of +0.25 that means that on average the system
gives 0.25 points too many to the home team.  But I want to add that
some people are making this out to be nothing more than a measure of
how good a systems measures home field advantage.  But that is not the
whole story because the accuracy of the ratings of the two teams are just
as important.
  Richard Billingsley's system had the lowest average bias at 0.0035.
This is not a new record but it is on the closer to zero numbers that
we have ever had.  As you can see int he plot the number finished mostly
positive, or giving to many points to the home team.  But you can see the
season started off with a heavy road bias in week one.  Can you say powder
puff schedules.  By the time conference schedules started the numbers were
positive and slowly moved towards zero from there.



2014 Winner: Billingsley Report
2013 Winner: Ted Thompson Average
2012 Winner: College Football By the Numbers
2011 Winner: The Pi-Rate Ratings
2010 Winner: Sagarin Predictor (record 0.0010)
2009 Winner: CPA Retro Rankings
2008 Winner; Opening Line
2007 Winner: Thompson SPRS
2006 Winner: Congrove Computer Rankings
2005 Winner: Computer Adjusted Line
2004 Winner: CPA Rankings
2003 Winner: Edward Kambour
2002 Winner: Gupta Power Ratings
2001 Winner: Flyman
2000 Winner: ARGH Power Ratings
1999 Winner: Jeff Sagarin




  MOST ACCURATE PREDICTOR (Entire Season)

Winner:  Updated Vegas Line.

  This award is based on mean square error. Mean square error takes
into account both deviation and bias and is perhaps the most commonly
used measure of evaluating estimators.
  I am leaving off the plot here because it looks very similar to the
plot for absolute deviation.  For the second half I will use the plot
for MSE.  The updated Vegas line was the most accurate predictor of the
final score for the season.  The computer adjusted line, which is just
an adjustment to the updated line had won the last 4 years.  These are the
only two systems to ever lead in this category. The lowest number this
season was 249.765. This is up a litte bit from the last two seasons.
As with absolute deviation you have to go all the way down to dokter at
8th place to find a system that is not tied to the line.


2014 Winner: Updated vegas line
2013 Winner: Computer Adjusted Line
2012 Winner: Computer Adjusted Line
2011 Winner: Computer Adjusted Line
2010 Winner: Computer Adjusted Line
2009 Winner: The Vegas Line  (record 206.152)
2008 Winner: The Vegas Line
2007 Winner: The Vegas Line
2006 Winner: The Vegas Line
2005 Winner: The Vegas Line
2004 Winner: The Vegas Line
2003 Winner: The Vegas Line
2002 Winner: The Vegas Line
2001 Winner: The Vegas Line
2000 Winner: The Vegas Line
1999 Winner: The Vegas Line



    ------------------------


  It is debatable how meaningful rating systems are in the early
parts of the season.  How can you rate the teams in the first
week of the season when no teams have played a game yet?  The playoff
comitte waits until mid October before releasing it's first ranking
of teams.  Some systems choose to wait until around this time
before being made public. Looking at only the second half of the
season also gives an estimate of how well a system does based
only(or mostly) on data from the current year.


   MOST STRAIGHT UP WINNERS (Second Half of Season)

Winner:  Dave Congrove

Bias


  The second half data started with week 8 and consisted of 421 games
between two division 1A teams.  Dave Congrove of CollegeFootballPoll.com
led the second have in straight up wins with a record of 315-106, 74.822%.
The margin was a wide six game lead over Payne and StatFox.  I would catagorize
the results as a whole as poor for the second half.  In large part this was
due to the very bad performance in the bowls.  The average performance for the
bowls was only a little better than 50% correct.


2014 Winner:  Dave Congrove
2013 Winner:  CPA Rankings
2012 Winner:  PerformanZ Ratings
2011 Winner:  Tempo Free Gridiron.
2010 Winner:  Brent Craig
2009 Winner:  Massey Ratings  (record 77.688%)
2008 Winner:  Nutshell Sports
2007 Winner:  Edward Kambour
2006 Winner:  Ed Bemiss
2005 Winner:  Dunkel Index
2004 Winner:  Jeff Sagain, Wesley Colley
2003 Winner:  Born Power Index & Edward Kambour
2002 Winner:  Born Power Index
2001 Winner:  Chris Montgomery
2000 Winner:  Geoff Freeze


  SMALLEST DEVIATION FROM ACTUAL GAME SCORES  (Second Half of Season)

Winner:  Ashby Accuratings

Bias

  Ashby Accuratings managed to beat out all the line based systems, moving ahead only in
the final week when the lines bombed in the bowls.  Accuratings had a mean error of 12.8931
from week 8 through the bowls.  This is up over half a point from the winner last year and
up a point and a half worse than the all the record.  The second through fourth place finishers
were in order, midweek line, opening line, updated line.


2014 Winner:  Ashby Accuratings
2013 Winner:  CPA Rankings
2012 Winner:  Computer Adjusted Line
2011 Winner:  Vegas Line
2010 Winner:  Computer Adjusted Line
2009 Winner:  Computer Adjusted Line (record 11.2917)
2008 Winner:  StatFox
2007 Winner:  Edward Kambour
2006 Winner:  Computer Adjusted Line
2005 Winner:  Vegas Line
2004 Winner:  Ken Ashby AccuRatings
2003 Winner:  Vegas Line (opening)
2002 Winner:  Vegas Line
2001 Winner:  Vegas Line
2000 Winner:  Vegas Line




  SMALLEST AVERAGE GAME BIAS  (Second Half of Season)

Winner:  Laz Index

Bias

  Laz Index had the lowest bias in the second half, -0.0067.  The majority of
the systems were on the positive side, the median being about +0.5 points.
The results were fairly flat across the last nine weeks.


2014 Winner:  Laz Index
2013 Winner:  Colley matrix
2012 Winner:  College Football by the Numbers
2011 Winner:  Stat Fox
2010 Winner:  Sonny Moore
2009 Winner:  Nutshell Girl
2008 Winner:  Dunkel Index
2007 Winner:  CPA Rankings
2006 Winner:  Vegas Line
2005 Winner:  Brent Moore
2004 Winner:  Least Squares Regression
2003 Winner:  Darrly Marsee
2002 Winner:  Massey BCS
2001 Winner:  PerformanZ Ratings
2000 Winner:  Darryl Marsee's Rankings



  BEST AGAINST THE SPREAD  (Second Half of Season)

Winner:  Massey Consensus Rank

Bias

  I am giving this category to the Massey Consensus Rank, with a record of 224-184, 54.902%.
Two systems, Loudsound.org, and logistic regression, actually had better percentages, but missed
at least one week.  What I did was throw out the week and recompute, and in both cases the
Consensus rank came out on top.  So that is the choice here.  Note that Massey's consensus
rank doesn't give predictions directly.  Because the page is so well known and contains so many
systems I believe it is worth including.  I take the means ranks for the teams and tranform them
to a different scale which allows for predicting the point spreads.
  The ATS numbers during the second half of 2014 were good.  Over 80% finshed with a better then 50%
record.  From the dark line on the plot you can see the consensus rank looks to have improved almost
every week through the season, slowly climbing up to the top.  Loudsound and logistic regression were
actually at the top most of the time, the problem is they just happened to skip down weeks.


2014 Winner:  Massey Consensus Rank
2013 Winner:  CPA Rankings (Record 59.322%)
2012 Winner:  Least Squares with team HFA
2011 Winner:  Dunkel Index
2010 Winner:  Least squares regression
2009 Winner:  Sonny Moore
2008 Winner:  StatFox
2007 Winner:  Nutshell Sports
2006 Winner:  Ed Bemiss
2005 Winner:  The Sports Report
2004 Winner:  StatFox          (record 58.94% for updated line)
2003 Winner:  Edward Kambour
2002 Winner:  The Sports Report
2001 Winner:  BMC Picks
2000 Winner:  The Buck System




  MOST ACCURATE PREDICTOR (Second Half of Season)


Winner: Thompon computer adjusted line

Bias

  The Thompson computer adjusted line had the smallest average mean square
error over the second half, 266.308.  This is 32 points lower than the leader
last season.  Ashby Accuratings is the top non line based system and was only
about 1 point behind. With the excpetion of starting off high in week 8 the
numbers were quite stable for the rest of the season.


2014 Winner:  Thompson CAL
2013 Winner:  CPA Rankings
2012 Winner:  Computer Adjusted Line
2011 Winner:  Computer Adjusted Line
2010 Winner:  Computer Adjusted Line
2009 Winner:  Computer Adjusted Line    (record 208.618)
2008 Winner:  Born Power Index
2007 Winner:  Atomic Football
2006 Winner:  Vegas Line
2005 Winner:  Vegas Line
2004 Winner:  Ashby Accuratings
2003 Winner:  Vegas Line
2002 Winner:  Vegas Line
2001 Winner:  Vegas Line
2000 Winner:  Vegas Line



  BEST PREDICTIVE SYSTEM  

Winner:  Billingsley +MOV

  To come up with the best overall predictive system I give each system
points for how well they do in all of the above categories.  I then sum up
the points and the system with the highest total is named predictive system
of the year.  So this award goes to the system that is the most well rounded
One flaw can be enough to pull a system out of the running.
  This award is often generates a surprise winner in seasons like this past year
where no one system dominates across the board.  This year that system is
Billingsley +MOV.  Other systems did good in one category but all it takes to
be a well rounded system is to be somewhere around the top quarter in every
category.  Two other solid systems this year were Pigskin Index, and Ashby
Accuratings.


2014 Winner:  Billingsley+
2013 Winner:  CPA Rankings
2012 Winner:  System Average
2011 Winner:  Dunkel Index
2010 Winner:  Catherwood Ratings
2009 Winner:  Ed Kambour
2008 Winenr:  StatFox
2007 Winner:  Harmon Forecast
2006 Winner:  Computer Adjusted Line
2005 Winner:  Dunkel Index
2004 Winner:  Ken Ashby AccuRatings
2003 Winner:  Ed Kambour
2002 Winner:  Ed Kambour
2001 Winner:  Kenneth Massey (non-BCS)
2000 Winner:  ARGH Power Ratings