Homogenous vs other run modelers

In the comparison of the Homogeneous model to other run-scoring modelers out there, it actually performs pretty darn well. I want to look at a few of the most notable ones, which can beat it, and why that happens.

First of all, there's simulators. Now, simulators with the same assumptions and parameters as this model will always underperform it, but if you include extra parameters (like lineups and matchups and such), then it's not very difficult to make something which outperforms this model, given enough iterations of the simulator.

A Markov Chain Modeler - Actually, if you take the time to improve the homogeneous model by allowing advancement on outs (and include strikeouts separately, as this necessitates) and having differing values of advancement on singles and doubles depending on how many outs there are, they turn into the same thing. Now, the markov is a heckuvalot simpler to program, so that is a big plus going for it, and it also has those extra features, at least right now, which help it be more accurate as well. (Incidentally, both of them can also be improved by giving some set value for a double play if there's a guy on first with less than two outs, and then further refining this based on other factors like if there's a guy on third with nobody out and the defense is 'giving' the run; this is all a lot of work though). The Homogeneous model has some advantageous in terms of practical work, though. First of all, it's very easy to get a break-down of run-scoring with it - though this version of the Markov is a piece of solid gold, as it will do all that stuff for you as well. Still, the closed form, symbolic nature of the solution that the Homogeneous model gives you is still useful for some particular manipulations you might need to do - for isntance, you can take partial derivatives to get PRECISE linear weights - or if you just really like being able to do it in equation form, and not haivng to punch the numbers in.

Various linear guys - these will do well, essentially by taking the empirical advantages to the extreme. They get all the stops possible out of typical lineup construction, talent distribution, pitching usage, maybe even particular managers' or players' idiosyncracies, by pulling data from really recent previous seasons. They pay the price for this 'cheating', though, if you get them in the wrong spot, by changing the data source very much at all - i.e. if it's trained on 2008-2011 data, it may predict 2012 really well, but give in 1985 and it will fall flat; furthermore, if you want to look and see what value you should give things, forget about it - these are not very predictive at all.

BaseRuns - BaseRuns is the reigning champion, except maybe simulators, of the run modelers that are out there. But while a part of me really really wants to love it based on the simplicity of the formula and the intuitive rightness that is there, it still has some issue. Namely, the B value here is determined empirically, just like in those linear models, and while that does have many accuracy advantages, it can also lead to ruin if you're looking at something very different from your training set - even if it just happens to be a team or pitcher (or umpiring crew or ballpark or...) that is very different from the average of what you have been looking at. So watch out!

In any case, you are going to be very limited by your projections, which are almost always going to be less accurate than any of this stuff. In the limit as the projections approach more perfect accuracy, though, the Markov/Homogeneous version of things (which can by approximated by simulator with enough trials with less difficulty in construction) is going to win out, and the linear ones will fall into the dust.

Re-Introducing the Homogeneous Model

Some time ago, I introduced what I call the Homogoneous model of run-scoring. However, I don't think this is the best explanation, and together with launching this new blog, I wanted to give it a fresh introduction.

The essential assumptions of the model are that every plate appearance has the same probabilistic distribution of outcomes as every other plate appearance; there are never any outs made on basepaths (i.e. no double plays, pick-offs, caught stealing, etc.); there is never any play of any kind which happens in the middle of a plate appearance (so no steals, passed balls, wild pitches, etc.), and that runners don't advance on outs (this one can be eliminated, though this would require roughly tenfold the work as went into the entire model so far). The model accepts nine parameters - rates for walks, singles, doubles, triples, homers, and outs (these must be constrained so as to add to one); as well as the probabilities that a runner on first will advance to third on a single, such a runner will score on a double, or that a runner from second will score on a single. Technically, it is also assumed that all runners advance at least as much as the batter on singles, doubles, and triples, and whilst this isn't necessarily always the case, in almost every application, the assumption is quite a safe one to make.

The model works on an inning basis. The basic premise for the model is that the weakest positive offenisve event is a walk, and four walks scores a run, five walks scores two, six walks scores three, etc. We can then mathematically derive that every team which only makes walks or outs will have (given an on-base percentage p) a chance of p^(n)*(1-p)^3*(1/2)*(n^2+3n+2) of scoring n-3 runs (for n>3). (The trick in getting the combinatorics right is that you know for sure that the final event of the inning was one of the outs; statisticians may recognize this as an instance of a negative binomial distribution looking for 3 failures). We can then get an overall run expectancy by taking the product of each of these probabilities with the number of runs being scored, and then summing from one run to infinity, which eventually leads us to (6*p^6 - 18*p^5 + 15 * p^4)/(1-p), which is the expression for the number of runs you score in an inning if all your positive events are walks, or in other words, the minimum number of runs you'll score at a given OBP.

To make the leap from this to a full distribution, given all the different possible offensive events, you simply have to realize that only the last three successes matter in an inning with three or more successes, as the previous ones are going to have been taken care of by the distribution from walks, as derived above. So you make an adjustment for the last three successes in an inning, and adjustments for the runs you score if you only get one or two successes in an inning, and you end up with the very long and complicated formula for run scoring given in the link at the top of this article.

Okay, so what are the strengths and weaknesses here? Well, first off, the assumptions are fairly hampering on the model. Steals and various during-a-PA basepath information is actually not such a big effect, as it ends up mostly cancelling out. Double plays, however, will bring the runscoring down a fair bit. But the bigger issue is every PA being the same as every other. This is just wrong. There are three reasons. The first is that the lineup is constructed in a way that makes certain sequences of events more or less likely than others. This probably helps the lineup slightly, though it depends somewhat on the actual construction. The second is that in different situations, different events are more or less likely to happen. With the bases loaded, for instance, the batter will be more apt to take a walk, and the pitcher will be less likely. Defensive positioning, mental approach, kind of swing, kind of pitch, aggressiveness - all of these things can change, and they can all have potentially very significant impacts. Exactly which side benefits out of this is something that empirical research would need to be done on. If this is out there at all, I'd be grateful for you sending it to me.

The final reason for the difference is that the strength of opposition is not constant, but rather it varies. So, if you're looking at an offensive lineup, the opposing pitchers will have different strengths, and if you're looking at a pitcher, the opposing offenses will have different strengths. Defense, park and weather factors, umpiring, and time-throug-the-order effects all play here as well. You also have to watch out for the expectations of the given situation, e.g. with platoon splits. In any case, all of these variations will serve to increase the run-scoring in comparison with the average case. The reason is that run-scoring is not linear. In fact, it's VERY non-linear. You can actually see this just from the on-base terms, where you have factors of OBP to the fourth, fifth, and sixth powers, and much more importantly, you are dividing by (1-p), which makes enormous impacts in making the curves go nearly vertical even on logarithmic scales near OBPs of 0 and 1. But the general point is, for any given average, the time you score less, you score rather significantly less, but the times you score more, you score WAY significantly more. And the overall effect is that if you add it all up, the mores are going to always overway the lesses, and this effect increases the higher your variance from the average.

A Priori Sabermetrics: An Explanation

The kind of Sabermetrics most people focus on is what I call “a posteriori sabermetrics” (or “empirical sabermetrics”). It uses data in order to construct statistical models in order to measure and predict performances in the game of baseball (or other sports, though I’m going to focus on baseball). It is based on established data, and it is also therefore dependent upon this data. This approach has a number of advantages. Data is fairly plentiful, and a number of statistical tests and models are increasingly common and easily done with recent advances in technology. It also has an advantage in how it focuses – it does not necessarily need to model every piece of how a final measure comes into place in order to be able to predict it. For instance, if you are looking at runs scored, you can focus on only a few statistics – maybe just the various different kinds of hits, walks, and outs – and not worry about things like lineup construction, ballpark, etc., and even with simple estimates, you can come up with a pretty good estimate, as all of the other things you are ignoring get taken care of as hidden variables which you essentially just copy over from past to future. This is particularly useful in that you don’t always know what all the pertinent factors are.

The problems you can run into with a priori sabermetrics come when things start to change. When you aren’t explicitly modeling something, a change in it is going to disrupt your statistic, potentially very greatly. You also get problems when you try to take your model beyond what it was developed on – for example, if you are trying to predict run scoring from a model built on leagues where OBP runs from .280 to .370, you have to be incredibly leery of applying this model to a team which has OBP over .400 or under .250 (or even when you get near the edges of your bounds at all, in some cases).

 

A priori sabermetrics, on the other hand, don’t require any data to build at all. They instead rely only on a knowledge of the rules, as well as almost always some assumptions and/or parameters. There are a couple of ways of doing a priori work. My most preferred method, possibly due to my deductive, scientific background, is to try to develop purely mathematical models. The nicest thing about these is that they can be absolutely and perfectly accurate, constrained of course by any assumptions and parameters. They also take essentially no time to run once they are computed. Finally, the equations you get here, being ‘real’ equations, and not just random artifacts or constructs, can be manipulated and adapted for other uses that you might have for them down the line, provided that your assumptions still hold in that new frame of observation.

Another, more popular kind of a priori sabermetrics is simulations. The nice thing about these is that they are really not very difficult to program – you can get a pretty good one going with a computer, a compiler, and several free hours. They can also be run to arbitrary decision – if you need more accuracy, you just run the simulator for more trials. Because they’re so easy to program, they’re also pretty easy to adapt, and they can be used in quite a wide variety of applications.

In either case, a priori sabermetrics, not being derived from data, are neither enslaved to it – they are applicable to ALL situations in which the assumptions hold and the required parameters are available. They are good for all kinds of crazy and wacky run environments. Perhaps their biggest triumph is when you are looking at major changes between the data you have ample supply of to look at and what you are trying to adapt it to – if you are trying to run sabermetrics on your local little league, a priori metrics are going to be far friendlier to you than most of what is being currently developed for the majors. But perhaps on a more practical level (man, don’t apply saber stats to the little league…), if you are trying to make a concerted change, a priori sabermetrics are going to be much better to you than empirical ones. For instance, if you want to look at having batters changing their approach in such a way that home runs and strikeouts both go way up, to ridiculous proportions, but things like singles and doubles go way down, you aren’t going to have any empirical basis for this change, and you need to rely on more purely theoretical a priori stuff. Of course, the big drawback of these kinds of statistics are that they are a slave to their assumptions and parameters – or lack thereof. If you are looking for run scoring as a function of ten factors, you can PERFECTLY model it to those ten factors, but you have absolutely no feel for any other factors you aren’t modeling. Where the a posteriori stats can buffer for this by assuming the same hidden mechanics will persist into the future as they’ve been in the past, here you have nothing.

Ultimately, the answer to these problems is to simply account for those hidden variables. But as you always have these parameters floating around, you are going to need at least some a posteriori research – which can actually include non-statistical things like scouting as well – in order to give you these inputs. But the further down the line this is, the more robust of a system you are developing. If you try to model runs straight off, you aren’t going to get as far as if you theoretically develop a system of how they are produced based on things like the rates at which basic events happen (e.g. walks, doubles, home runs, strikeouts, etc.), and then try to predict those basic statistics based off of your empirical research.