## How accurate would a perfect Premiership prediction model be?

Fair warning: The work below is right at the limit of my understanding of statistics – if someone spots an error or a failure in the logic I’d appreciate it if you’d let me know.

Danny Page came to my aid on twitter yesterday, and linked me to Phil Birnbaum’s terrific site. I encourage anyone interested in the statistics behind modelling North American sports to take a look. One of the things that caught my eye is that Tom Tango is pretty heavily involved in the comments section. Tom’s site is here – he writes about statistics in an elegant and engaging way that few others manage.

The comments to this post in particular caught my attention. Tom’s essentially determining how much a sports game tells you about the ability of teams in a given league. The equation works out that you know roughly as much after 12 NFL games (75% of the season) as you do after 14 NBA games (17%), 36 NHL games (44%), and 69 MLB games (43%). Thus at the end of the NBA season we’re a lot more likely to have the best teams at the top of the league and the worst at the bottom than we are over the course of an NFL season. For those interested the Premiership equivalent is 18 games, or 47% of the season.

The part that really piqued my interest though was the random variation in points that we’d expect across from ‘x’ equivalent teams over a season consisting of ‘y’ games. After digging around a little more on Phil’s site I came across these two posts (part one, and part two), which discuss the best possible prediction system you can make given that this random variation exists and I thought it’d be a great exercise to repeat for the Premiership.

Following step three of Tango’s maths from the comments section of the original post to calculate the random standard deviation of the winning percentage, then we find:

SQRT(0.5*0.5/38)*38 = 3.08

(correction – previously was missing the *38 from the equation – h/t Martin Eastwood)

To convert from wins to points then multiply by 3, so 3 x 3.08 = 9.25 *

And that’s pretty much the perfect model we can aim for, when the random deviation is accounted for **, and though it seems quite large to me it does also kind of make sense if I consider the number of teams in a given season that I wouldn’t be surprised to see in the upper mid table, nor in a relegation battle come the final standings.

So how do the models I use compare to this? Well fortunately this is something I’ve posted about recently, and the takeaway was the plot below, showing the performance of four different models over the past twelve seasons.

It’s worth mentioning there that there are numerous occasions where the models beat that expected from a perfect model. How can that be so? Well the second part of Phil’s discussion linked above (and again here) does a much more eloquent job of explaining that than I can hope to – so I encourage you to read that. Which leads to the question, how much closer are they on the whole to being ‘perfect’ when compared to the most extreme examples?

Let’s consider the example where we enter the season knowing nothing about any of the teams, and so we predict that they’ll each score 52.08 points (the average). In that case we’d be out by a standard deviation of 11.62 points, so any model we build needs to improve on that number.

The bottom row here reflects how much closer to the perfect model these teams are than the extreme one. And so it appears that a teams goal ratio in season one is a pretty damn good indicator as to how many points a team will score in year two. As mentioned in that post I’m thinking of switching to use it for my predictions next time around. Why does it look better than the ‘perfect’ value? Well I have no proof to back this up but I would proffer that it’s due to the fact that each of these models can only look at the seventeen teams that are in the Premiership in consecutive years, and it’s almost certain that the teams relegated to the Championship have had a degree of bad luck that would regress the following season. I’d suggest, because of the relative sample sizes, that goal ratio would be less likely to predict this than shots on target, which in turn would be less likely to predict this than total shots. If this holds true then the values for STR, TSR, and TSR2.4 may by lower (i.e., closer to 100%), whereas goal ratio would be higher (e.e., below 100%).

So there are two conclusions to this post – the first is not to expect any to find a holy grail of a model that’s going to predict how many points teams score season over season – random variation is going to make that impossible. Secondly what we’re working with at the minute is a pretty damn good improvement over having no knowledge at all, but we haven’t reached the pinnacle yet.

Part II of this series: The variance of points in the Premiership and La Liga
Part III of this series: After how many Premiership games does talent become more important than random variation?

*the alternative here is to multiply by the 2.74 points per game that the average Premiership game is worth but I think I can put forward a stronger argument for 3

**I also read this paper which suggests that home advantage will make this gap smaller, and the rationale seems pretty sound, so I may come back and revisit this at a later date