Thoughts on Player Value Part II

In the first post on Player Value, we discussed some of the basic types of baseball related data along with the strengths and weaknesses of each type.  If I was to summarize this post as succinctly as possible, I would say that primary data is the most reliable while the more calculations you have to make to produce a given statistic, the more opportunity for error is introduced.  On the other hand, primary data may be limited in what it tells you about a player's performance than calculated data which is why we often prefer calculated data despite the increase opportunity for error.

Offensive statistics have been around since practically the invention of the game, including fairly advanced and calculated measurements.  I recall reading about Slugging Percentage for the first time in the 1961 World Book Encyclopedia Yearbook!  On the other hand, defensive statistics have been extremely limited, probably due to difficulties in measurement.  For decades, defensive measurement was limited to Errors, Chances and Fielding Percentage, measures that I think we would all agree do not begin to quantify the contributions, positive or negative, of a player to defense.  Defense was largely in the eye of the beholder.  The conventional wisdom was that once you one a Gold Glove award for defensive excellence, you could pretty much count on winning it every year for the rest of your career.  You were also more likely to win a Gold Glove if you hit well too!

At the time Moneyball was written, statistics oriented GM"s like Billy Beane had come to totally discount defense in the formula for winning baseball.  Since they had no way to measure it, they preferred to write if off as a parameter to consider, rather than rely on scouting or "the smell test."  Their attitude was, "if you can't measure it, it doesn't exist."

Since then, at least two fairly sophisticated defensive measurement systems have become fairly well known, and now that they have something that can be expressed as a number, the Billy Beane's of the world have suddenly embraced defense as in important contributor to a player's value.  Possibly the best known of these systems is UZR or Ultimate Zone Rating as championed by a website known as Fangraphs.

I do not pretend to completely understand UZR.  I will try to summarize it here, but I highly suggest visiting the Fangraphs site linked to the left and clicking on the Glossary tab at the top of the page, then look for an article called UZR Primer.  That is the most complete explanation of the system that I know of, and it's from the site that champions it.

Basically, UZR is derived from the charted observations of knowledgeable people of each and every ball in play in MLB.  Judgements are made as to type of hit, degree of difficulty fielding it, location.  Frequency of plays made are then compared to historical norms based on multiple years of data.  Throws are also charted as well as limiting the progress of baserunners.  All this data is then quantified into a number which tells you how many outs above or below average a fielder achieves and this is then translated into a form of Win Shares to be added into an overall WAR score.

I think we would all agree that UZR tells us a lot more about a player's defense than the old errors, chances and fielding percentage.  As we learned in Part I of this series, though, the more variables and the more calculations involved in producing a statistic, the higher the probability of introducing error into the process.  The big question about UZR is not that it is measuring the right components of defense, but whether it is doing so accurately.  Fairly wide swings in UZR from year-to-year continue to be problematic for the statistic.

Forget for a minute, that the entire process is base on the subjective opinion of an observer.  The system also has significant structural problems in the calculations themselves.  One big problem with UZR is what I will call Effective Sample Size.  To illustrate this problem, let's look at Hunter Pence who grades out as a roughly average defensive RF, who plays the position exclusively and who plays every game, so his sample size should be as large as you can get for the position.  UZR has 6 degrees of difficulty in it's ratings.  I list them here with the expected percentage of successful plays in parentheses for each:  Impossible(0%), Remote(1-10%), Unlikely(10-40%), Even(40-60%), Likely(60-90%), Routine(90-100%).

Now, I will list Hunter Pence's percentage success rate for each category this year with the number of chances for each in parentheses:  Impossible-0%(56), Remote- 9.1%(11), Unlikely-22.2%(9), Even- 57.1%(7), Likely-100%(15), Routine- 99.2%(264!).

As you can see, the range of plays sorted by degree of difficulty is a reverse bell curve skewed heavily to the routine.  The vast majority of plays are going to be made close to 100% of the time or 0% of the time.  The types of plays that would be expected to separate a good fielder from a bad fielder, Remote, Unlikely, Even and Likely combine to make up a remarkably small sample size, which I would call the Effective Sample Size.  As few as 1 or 2 plays made or not made the Remote category can make a large difference in your final number.  Differences in UZR are derived from remarkably small Effective Sample Sizes!

This explains, at least in part why we see such large variations in UZR from year-to-year.  Add in the impact of injuries and the normal trajectory of aging and it can be almost impossible to factor out whether yearly swings in UZR are due to chance, injury, age, or skills.  It also calls into question whether even 3 years of data is gives you an accurate picture of a player whose physical ability may have eroded over that period of time.

Still, I would much rather know what a player's UZR is than simply his errors, chances and fielding percentage simply because it includes information about important factors in defense even if that information is flawed by the methodology.  On the other hand, I maintain that there is still a significant role for subjective observation, or the "Smell Test" in evaluating the defensive skills of baseball players due to flaws in the methodology of deriving UZR and other defensive metrics.