One of the outstanding questions about the results of the trial-run/simulation of the 2011 season was how well this thing performed against the spread. I was able to find a source of 2011 lines, and I imported those lines so that I could compare results.
Assuming I imported the data correctly, and that my database queries are correct, I was pleasantly surprised with the results.
Of the 680 regular season games in 2011, the model was 422-231 with 27 pushes. (I included cases where the prediction was the same as the spread as pushes.) Of the 35 bowl games, the model was 21-13 with 1 push. That means the model was a little over 60% correct against the spread.
NOTE: I do not expect the 2012 predictions to do that well until at least late October or maybe November. For the 2011 run, I was using end-of-year statistics of the teams as a basis for the predictions. So when I kick this thing up in early October, the “features” of the 2012 teams will be based on mid-season stats.
As another follow-up to the 2011 results, I wanted to take a look at the model’s accuracy as a function of the prediction value. (eg: One would think that the greater the resulting prediction value between two teams, the more likely the model is going to be correct.)
Below is the result of the analysis.
| P-Value | Count | Correct | Correct PCT |
|---|---|---|---|
| 1 | 37 | 14 | 37.84% |
| 2 | 36 | 17 | 47.22% |
| 3 | 26 | 17 | 65.38% |
| 4 | 26 | 11 | 42.31% |
| 5 | 31 | 19 | 61.29% |
| 6 | 37 | 25 | 67.57% |
| 7 | 29 | 19 | 65.52% |
| 8 | 33 | 23 | 69.70% |
| 9 | 17 | 13 | 76.47% |
| 10 | 37 | 28 | 75.68% |
| 11 | 23 | 18 | 78.26% |
| 12 | 32 | 26 | 81.25% |
| 13 | 36 | 28 | 77.78% |
| 14 | 21 | 16 | 76.19% |
| 15 | 20 | 18 | 90.00% |
| 16 | 31 | 28 | 90.32% |
| 17+ | 243 | 228 | 93.83% |
Unsurprisingly, when the model predicts a team to win by 4 or less points, it doesn’t do so well – it’s correct only about half the of the time. And when the model predicts a team to win by 17 or more points, it’s correct a whopping 94% of the time – its accuracy gradually increases the greater the prediction value.
This accuracy data is factored into my rankings formula as a way to weight the round-robin simulation prediction results used for ranking.
