Model Performance

Breaking down the season I wanted to look at how good the model was at predicting first against the actual outcome of the game and then how it did against the spread.  The model went 174-95 for 64.7% win percentage which is actually one game better than Vegas!  I further broke down the win percentage based on the predicted margin of victory for the model.  As you can see in the model below (minus the small sample size 9+) the larger the margin of victory predicted the better the model did including a 72% win rate when the model predicted a team to win by between 5 and 9 points.  This is of course not surprising, but great to see the model was performing as expected.

Now I also ran a scatter plot and plotted the trendlines for both the model predictions and outcomes and what Vegas had predicted.  For the line equation we have y = Actual margin of victory, m = slope, x = predicted margin of victory, and b = the coefficient.  Now the ideal scenario would be a slope of 1 and coefficient of 0.  For the model Y = 1.12x - 0.07 and for Vegas the trendline was Y = .97 + 0.89.  Now these numbers are really interesting because the one thing I consistently mentioned is that the model does not perform well at the extremes and that's exactly what the slope shows.  Conversely Vegas fell short at the start where they on average favored the away team more at almost a point per game.  When I broke down the win loss record predicting the home team to win vs predicting the away team Vegas had a 69% accuracy when predicting the home team to win and only a 55% accuracy when predicting the visiting team to win.  Now, my model was at ~64% for both which checks out with the coefficient right at -.07.  However, the model performed terrible when Vegas had a spread over 9 as the model almost always undershot the projection.  

The easiest level of improvement for next year is my interpreting of the model.  Since it does such a terrible job on the tails anything Vegas has a 9 or more the model should not be used to predict as the model went 10-22 on these games.  Luckily, I noticed this early on did was not recommending these games in our picks.  Now where the model did great is when the Vegas Spread was within 9 points and the model's prediction was over 2.5 points different.  In these cases, the model beat the spread by 64 ourtof a 114 times for a winning percentage of 56.1%!  Now that I have this information I'm extremely excited about next year and utilizing this model.  I will also be training another model with some additional features and I plan on tuning the hyperparameters a little more.  Than you to everyone who followed along this year!