Some of my otherwise most ardent supporters recently called out the model for its somewhat nonconventional and seemingly inconsistent attitudes towards the Dolphins of Miami. Last week, the fish traveled to Dallas as 20-something point underdogs and the model instructed us to go wide open throttle. This week, we’ve got Miami at home against LAC and the model wants us to bet AGAINST the little tadpoles. What the fuck is going on?
The most obvious explanation would be a claim that LAC is significantly stronger than Dallas. This claim has merit, at least based on those variables the model takes as inputs; LAC has the highest preseason DVOA of the three teams at 16.1%, while DAL sits at 5.1% and MIA at -30.6%. DAL has performed much better in its opening weeks, heading into its MIA game with a 50.5% total DVOA versus a significantly lower -19.5% for LAC. I put more stock in preseason DVOA right now given how early we are in the season, so I didn’t see a grave inconsistency here, but the size of the edges admittedly looks odd.
It’s easy to visualize models that only have 2 dimensions, X and y; you throw up a scatterplot, look where the points fall, and you have a good visual understanding of the model. Even with 3 dimensions, you can look at funny 3-D plots and get a feel for your data. But DomModel2019 computes across 8 dimensions, and that’s a little tougher to visualize.
In order to get a feel for how the model is handling Miami, let’s hold a few variables constant. The relationship we’re most interested in is how changes in the spread affect Miami’s cover probability. If we hold the DVOAs for MIA and LAC constant, we can fiddle with the spread and the points total for any given week to see how the model would predict MIA’s success.
Think of this as a sort of “mesh,” a non-linear object that twists itself around. This sort of non-linearity is an example of how sophisticated machine learning models are and how difficult it can be to interpret their output- and this is only across 2 of the model’s dimensions.
According to the model, no combination of spread and o/u would enable MIA to prevail a majority of the time. But more surprising is how Miami, being the underdog, actually decreased in coverage probability as their spread increased. That’s not totally impossible to reconcile- we might conclude that the larger spread is indicative of a much stronger team, making it harder for Miami to cover. But it’s the opposite of the relationship I anticipated. I would have thought that a large number of points would correlate quite strongly with coverage probability. Curious, I decided to run the same mesh for the Dallas game.
The different combination of DVOAs yield fantastically different results. Here, I get the relationship I expect- although there is a sort of “twist” where the relationship changes.
I thought about these “meshes” for a long time. The probabilities on the Dallas game seemed counterintuitively high, and the relationship for LAC this week just didn’t feel right.
I made a key error when I put the model into production. The model was tuned on 80% of my NFL data, holding 20% out completely for testing. This is of course with good intention- we need to make sure our evaluation of the model is pure, and information does not “leak” into our testing set and skew our performance via overfitting. But that 20% could also contain key information, especially so when it comes to games at the “edge” of the model, where sample sizes are smaller- games like MIA with uncommonly high spreads.
I decided to re-train the model with the same hyperparameters (thus avoiding overfitting) on the entire dataset. This changed the mesh for LAC substantially:
And for DAL mildly:
And, most importantly, provided much more realistic edges for this week- though note that lines have shifted in some games.
| team_away | team_home | cover_team | cover_spread | cover_prob | cover_edge |
|---|---|---|---|---|---|
| Los Angeles Chargers | Miami Dolphins | LAC | -17.0 | 60.25% | 15.03% |
| Dallas Cowboys | New Orleans Saints | DAL | -3.0 | 57.91% | 10.56% |
| Jacksonville Jaguars | Denver Broncos | JAX | +3.0 | 53.63% | 2.38% |
| Minnesota Vikings | Chicago Bears | MIN | +2.5 | 52.81% | 0.82% |
| Washington Redskins | New York Giants | WAS | +3.0 | 52.35% | -0.07% |
| New England Patriots | Buffalo Bills | BUF | +7.0 | 52.06% | -0.61% |
| Tennessee Titans | Atlanta Falcons | TEN | +5.0 | 51.79% | -1.13% |
| Philadelphia Eagles | Green Bay Packers | PHI | +4.5 | 51.76% | -1.18% |
| Oakland Raiders | Indianapolis Colts | OAK | +7.0 | 51.65% | -1.39% |
| Cleveland Browns | Baltimore Ravens | CLE | +5.5 | 51.49% | -1.69% |
| Carolina Panthers | Houston Texans | CAR | +4.0 | 51.34% | -2.0% |
| Tampa Bay Buccaneers | Los Angeles Rams | LAR | -10.0 | 51.07% | -2.49% |
| Cincinnati Bengals | Pittsburgh Steelers | CIN | +4.5 | 50.97% | -2.69% |
| Seattle Seahawks | Arizona Cardinals | SEA | -4.0 | 50.63% | -3.34% |
| Kansas City Chiefs | Detroit Lions | KC | -7.0 | 50.0% | -4.55% |
If you take nothing else from this post, it needs to be that the model’s outputs are incredibly complex. For the LAC game, a statement like “larger spreads are better for the dog- to a point” would hold true. For the DAL game, the exact opposite statement holds, and the difference between the two games lies in the variables we held constant- the DVOAs for the respective teams.
Leave a Reply