Welcome to the 17th annual NCAA PT Awards. (How many more is an open question at this point)
The purpose of these awards is to help raise the publicity for those systems that have been
superior in various qualities of interest.  Awards are be based entirely on the numbers
I've monitored with my prediction tracker web pages. For the 2017 season I followed the
weekly performance of 61 computer rating  systems. This is down 3 from last year, and 11 over
the last three seasons.  So we are definitely shrinking.  This was the fourth year of the four
team playoff system and the #1 team still has not won.


  MOST STRAIGHT UP WINNERS (Entire Season)

Winner:  Ashby Accuratings

straight up plot

  This year I collected the predictions of 761 games involving two FBS teams.  The system that picked
the most correct winners was Ken Ashby's Accuratings, 579-201, 74.231%.  That winning winning
percentage is up a little over the 73% seen last season. There are a few questions at the top. First
off with the winner, Ashby was one of only one or two ratings that always posted predictions after the
early week games had already taken place. So that win maybe deserves an asterisk.  And Ashburn's
Atomic Football finished a half game behind but had one missing game.  Looking at this game it was one
of the rescheduled games that almost everyone got wrong. So I don't feel it safe to he would have
been correct on that one, otherwise I would have named them co-winners.
 This season the Vegas line was very good, finishing in 5th place. Only Ashby, Ashburn, Keeper and
Sports Cruncher managed to do better.    The updated line did three games better than the midweek line.
and 16 games better than the opening line.
 In the plots the category winner will be denoted with the dark black line. You can enlarge the plots
by right clicking and selecting view image.  Here you can seen the mismatched scheduling early in the
season and Ashby never really took the lead until the very end.

2017 Winner: Ashby Accuratings
2016 Winner: Dokter Entropy, Catherwood Rating System
2015 Winner: Ted Thompson Average record(78.361%), Pi-Rate Mean
2014 Winner: Ted Thompson CAL
2013 Winner: Ted Thompson SPRS
2012 Winner: Hank Trexler
2011 Winner: Covers.com, Tempo Free Gridiron
2010 Winner: Dokter Entropy
2009 Winner: Pigskin Index
2008 Winner: System Average, System Median
2007 Winner: StatFox
2006 Winner: Thompson SPRS, system average, Sagarin predictive, & Sagarin overall
2005 Winner: Pigskin Index
2004 Winner: Jeff Sagarin's Overall Rating
2003 Winner: Edward Kambour & Jeff Sagarin's Overall Rating
2002 Winner: ARGH Power Ratings
2001 Winner: Massey Ratings
2000 Winner: ARGH Power Ratings
1999 Winner: Vegas line




  SMALLEST DEVIATION FROM ACTUAL GAME SCORES (Entire Season)

Winner:  Updated Line

absolute deviation

  Deviation from the game score is the difference between the game prediction
and the actual result.  A value of zero would mean the score difference is predicted
exactly.  One property of a good system would be to minimize the system's average
game deviation.
  The updated line had the smallest average absolute error this year, 12.2750 points.
This is about half a point better than last year when we set a record for being the worst
best score.
 The Computer Adjusted line was in second only 0.0128 points behind.   As is typical in this
category all of the best systems are systems that are either the line itself or a
derivative of the line. ESPN's FPI is the top non line in 5th place.
  From the plot you can see that the line was pretty much the leader for the entire season
and that the numbers were fairly stable across the entire season.


2017 Winner: Updated Vegas Line
2016 Winner: Updated Vegas Line
2015 Winner: Updated Vegas Line
2014 Winner: Updated Vegas Line
2013 Winner: Computer Adjusted Line
2012 Winner: Computer Adjusted Line
2011 Winner: Computer Adjusted Line
2010 Winner: Computer Adjusted Line
2009 Winner: Computer Adjusted Line (record 11.26)
2008 Winner: Vegas Line
2007 Winner: Vegas Line
2006 Winner: Computer Adjusted Line
2005 Winner: Vegas Line
2004 Winner: Vegas Line
2003 Winner: Vegas Line
2002 Winner: Vegas Line
2001 Winner: Vegas Line
2000 Winner: Vegas Line
1999 Winner: Vegas Line



  BEST AGAINST THE SPREAD (Entire Season)

Winner:  Jeff Sagarin

ats

  Beating the spread is not the number one goal for most computer
rating systems.  But it is something that the average person likes
to look at and probably the most popular subject that I get emails
about.  I know that looking at all the games is probably not the
most meaningful of measurements but it is what I have been reporting.
The line of record is taken midweek.  Usually Thursday mornings.
  The Sagarin Ratings had the best ATS record this year with a record of
407-354, 53.482%  In general this is not a very good winning mark.  It was
the lowest winner since 2010.  Sportscruncher, the computer adjusted line
and updated line were the only other systems better than 53%.  Again we
see an interesting phenomenon with the updated vegas line being very predictive
against the spread.  Meaning line movements are an important thing to watch.
Only two systems did worse then 47%, so basically everyone finished in the
47-53% range.  In otherwords the distribution of all systems is just random??


2017 Winner: Sagarin Ratings
2016 Winner: Updated Line, Dave Congrove(collegefootballpoll.com)
2015 Winner: Updated Line, Computer Adjusted Line, The Sports Cruncher
2014 Winner: Laffaye RWP
2013 Winner: CPA Rankings (record 56.906%)
2012 Winner: Laz Index, Computer Adjusted Line
2011 Winner: Dunkel Index
2010 Winner: Catherwood Ratings
2009 Winner: Covers.com
2008 Winner: Stephen Kerns
2007 Winner: Nutshell Sports
2006 Winner: Harmon Forecast
2005 Winner: Keeper's College Football Ratings
2004 Winner: Edward Kambour
2003 Winner: Edward Kambour
2002 Winner: Dunkel Index
2001 Winner: BMC Picks
2000 Winner: Dunkel Index
1999 Winner: Average across all systems



  SMALLEST AVERAGE GAME BIAS (Entire Season)

Winner:  laz Index

Bias

  Bias is a little different from deviation.  Deviation measures the
distance between a prediction and the actual result.  Bias combines
distance and location of the prediction.   Bias measures whether the
predictions are too high or too low.  It could be specified in different
terms but my numbers are relative to the home team.
  Laz Index had the lowest average bias at -0.0183.  As you can see in the
plot the numbers finished positive, slightly favoring the home teams espcially
for the first half of the season.


2017 Winner: Laz Index
2016 Winner: Pi-Rate Mean
2015 Winner: SportsTrends
2014 Winner: Billingsley Report
2013 Winner: Ted Thompson Average
2012 Winner: College Football By the Numbers
2011 Winner: The Pi-Rate Ratings
2010 Winner: Sagarin Predictor (record 0.0010)
2009 Winner: CPA Retro Rankings
2008 Winner; Opening Line
2007 Winner: Thompson SPRS
2006 Winner: Congrove Computer Rankings
2005 Winner: Computer Adjusted Line
2004 Winner: CPA Rankings
2003 Winner: Edward Kambour
2002 Winner: Gupta Power Ratings
2001 Winner: Flyman
2000 Winner: ARGH Power Ratings
1999 Winner: Jeff Sagarin




  MOST ACCURATE PREDICTOR (Entire Season)

Winner:  Updated Vegas Line.

Bias

  This award is based on mean square error. Mean square error takes
into account both deviation and bias and is perhaps the most commonly
used measure of evaluating estimators. A key feature is a larger penalty
for being farther from the actual result.
  The updated line had the lowest mean square error, 246.617. The three
of the next four were all line based. The only non line based entry in
the top five was ESPN's FPI at number 4.


2017 Winner: Updated Vegas Line
2016 Winner: Updated vegas line
2015 Winner: Updated vegas line
2014 Winner: Updated vegas line
2013 Winner: Computer Adjusted Line
2012 Winner: Computer Adjusted Line
2011 Winner: Computer Adjusted Line
2010 Winner: Computer Adjusted Line
2009 Winner: The Vegas Line  (record 206.152)
2008 Winner: The Vegas Line
2007 Winner: The Vegas Line
2006 Winner: The Vegas Line
2005 Winner: The Vegas Line
2004 Winner: The Vegas Line
2003 Winner: The Vegas Line
2002 Winner: The Vegas Line
2001 Winner: The Vegas Line
2000 Winner: The Vegas Line
1999 Winner: The Vegas Line



    ------------------------



  It is debatable how meaningful rating systems are in the early
parts of the season.  How can you rate the teams in the first
week of the season when no teams have played a game yet?  The playoff
committee waits until mid October before releasing its first ranking
of teams.  Some systems choose to wait until around this time
before being made public. Looking at only the second half of the
season also gives an estimate of how well a system does based
only(or mostly) on data from the current year.




   MOST STRAIGHT UP WINNERS (Second Half of Season)



Winner:  Ashby Accuratings


  The second half data started with week 8 and consisted of 410 games
between two FBS teams. Ashby Accuratings had the best record 306-104, 74.634%.
The updated line was 11th place.  Interestingly, the opening line was 56th place, one of the worst.


2017 Winner:  Ashby Accuratings
2016 Winner:  Dokter Entropy
2015 Winner:  Thompson Average   (record 78.23%)
2014 Winner:  Dave Congrove
2013 Winner:  CPA Rankings
2012 Winner:  PerformanZ Ratings
2011 Winner:  Tempo Free Gridiron.
2010 Winner:  Brent Craig
2009 Winner:  Massey Ratings
2008 Winner:  Nutshell Sports
2007 Winner:  Edward Kambour
2006 Winner:  Ed Bemiss
2005 Winner:  Dunkel Index
2004 Winner:  Jeff Sagain, Wesley Colley
2003 Winner:  Born Power Index & Edward Kambour
2002 Winner:  Born Power Index
2001 Winner:  Chris Montgomery
2000 Winner:  Geoff Freeze




  SMALLEST DEVIATION FROM ACTUAL GAME SCORES  (Second Half of Season)

Winner:  Updated Line

The best for the second half was the updated line with an average error of 12.3256.
This is a point and a half better than last season.  ESPN FPI and SportsCruncher were
als accurate in the second half.


2017 Winner:  Updated Line
2016 Winner:  Updated Line
2015 Winner:  Updated Line
2014 Winner:  Ashby Accuratings
2013 Winner:  CPA Rankings
2012 Winner:  Computer Adjusted Line
2011 Winner:  Vegas Line
2010 Winner:  Computer Adjusted Line
2009 Winner:  Computer Adjusted Line (record 11.2917)
2008 Winner:  StatFox
2007 Winner:  Edward Kambour
2006 Winner:  Computer Adjusted Line
2005 Winner:  Vegas Line
2004 Winner:  Ken Ashby AccuRatings
2003 Winner:  Vegas Line (opening)
2002 Winner:  Vegas Line
2001 Winner:  Vegas Line
2000 Winner:  Vegas Line



  BEST AGAINST THE SPREAD  (Second Half of Season)


Winner:  ESPN FPI

  ESPN's FPI had a strong second half which includes best ATS record, 217-181, 54.523%
This season the ATS performance was rather random, meaning the average ended up being right
around 50%. And majority is just a big clump from 48 to 52%.


2017 Winner:  ESPN FPI
2016 Winner:  Lee Burdof
2015 Winner:  Updated Line, Computer Adjusted Line, SportsTrends.
2014 Winner:  Massey Consensus Rank
2013 Winner:  CPA Rankings (Record 59.322%)
2012 Winner:  Least Squares with team HFA
2011 Winner:  Dunkel Index
2010 Winner:  Least squares regression
2009 Winner:  Sonny Moore
2008 Winner:  StatFox
2007 Winner:  Nutshell Sports
2006 Winner:  Ed Bemiss
2005 Winner:  The Sports Report
2004 Winner:  StatFox          (record 58.94% against updated line)
2003 Winner:  Edward Kambour
2002 Winner:  The Sports Report
2001 Winner:  BMC Picks
2000 Winner:  The Buck System




  SMALLEST AVERAGE GAME BIAS  (Second Half of Season)

Winner:  Catherwood Ratings

  Catherwood rating system had the lowest bias in the second half, -0.1561.
For the most part the numbers here are rather large, a little less than -1,
and overwhelmingly negative.


2017 Winner:  Catherwood Ratings
2016 Winner:  PerformanZ Ratings
2015 Winner:  CPA Retro
2014 Winner:  Laz Index
2013 Winner:  Colley matrix
2012 Winner:  College Football by the Numbers
2011 Winner:  Stat Fox
2010 Winner:  Sonny Moore
2009 Winner:  Nutshell Girl
2008 Winner:  Dunkel Index
2007 Winner:  CPA Rankings
2006 Winner:  Vegas Line
2005 Winner:  Brent Moore
2004 Winner:  Least Squares Regression
2003 Winner:  Darrly Marsee
2002 Winner:  Massey BCS
2001 Winner:  PerformanZ Ratings
2000 Winner:  Darryl Marsee's Rankings



  MOST ACCURATE PREDICTOR (Second Half of Season)

Winner: Updated Line


  The updated line had the smallest average mean square error over the second
half, 254.153.   EPSN FPI, 4th,  was the highest ranking system that was not the line.


2017 Winner:  Updated Line
2016 Winner:  Updated Line
2015 Winner:  Midweek Line
2014 Winner:  Thompson CAL
2013 Winner:  CPA Rankings
2012 Winner:  Computer Adjusted Line
2011 Winner:  Computer Adjusted Line
2010 Winner:  Computer Adjusted Line
2009 Winner:  Computer Adjusted Line    (record 208.618)
2008 Winner:  Born Power Index
2007 Winner:  Atomic Football
2006 Winner:  Vegas Line
2005 Winner:  Vegas Line
2004 Winner:  Ashby Accuratings
2003 Winner:  Vegas Line
2002 Winner:  Vegas Line
2001 Winner:  Vegas Line
2000 Winner:  Vegas Line



  BEST PREDICTIVE SYSTEM  

Winner:  ESPN FPI

  To come up with the best overall predictive system I give each system
points for how well they do in all of the above categories.  I then sum up
the points and the system with the highest total is named predictive system
of the year.  So this award goes to the system that is the most well rounded
One flaw can be enough to pull a system out of the running.

  This season is kind of a tough call.  If you look at the full season data
then the updated line and computer adjusted line finish near tied for the best.
Then SportsCruncher is a bit back in third.  I prefer not to name the line the
best of the year unless there are not good alternatives.  If you look at the
second half ESPN is the winner. Followed by teamrankings.com and Ashby Accuratings.
In the end I am going with ESPN because they were more dominant in the second
half than SportsCruncher was across the full season.


2017 Winner:  ESPN FPI
2016 Winner:  Updated Line
2015 Winner:  Atomic Football
2014 Winner:  Billingsley+
2013 Winner:  CPA Rankings
2012 Winner:  System Average
2011 Winner:  Dunkel Index
2010 Winner:  Catherwood Ratings
2009 Winner:  Ed Kambour
2008 Winenr:  StatFox
2007 Winner:  Harmon Forecast
2006 Winner:  Computer Adjusted Line
2005 Winner:  Dunkel Index
2004 Winner:  Ken Ashby AccuRatings
2003 Winner:  Ed Kambour
2002 Winner:  Ed Kambour
2001 Winner:  Kenneth Massey (non-BCS)
2000 Winner:  ARGH Power Ratings