Working with Poor Data
I just read a lengthy article on the Seattle Mariners-focused blog U.S.S. Mariner called Evaluating Defense. It's a nice discussion on the current state of defensive statistics and their evaluation in baseball. Beyond that, it's a nice discussion on interpreting and making decisions based on multiple sources of not-so-great data.
On contradictory results:
[T]he age of defensive statistical analysis is still in its infancy, and as such, there is not a consensus system that is correct, or established as the industry standard. There are several systems built on solid theories that evaluate different parts of defensive prowess, and sometimes, these systems give widely contradictory results. So, what do we do then, if two systems, both well designed, can’t agree?
At this point, my preference is to take a prism perspective. All of the systems have strengths, and all have flaws. So I’d rather not take any of them at face value, but instead develop a general idea of a player’s abilities based upon as much good input as I can get.
On sample size:
The generally accepted principle in defensive statistics is that you need at least two years of data to generate any kind of real conclusion about a player’s abilities, and you’d prefer to have more. With [Yuniesky] Betancourt, we basically have 1/3 of one season. There are just way too many non-fielding factors that could influence the number over that period of time. Ball in play distribution is a huge factor in small sample defensive numbers, for instance. If Betancourt happened to receive more easy to field grounders than others, his number would be through the roof. If teams were whacking uncatchable balls into the hole, his rating would suffer, and because of the small time frame, the impact of a few extra balls here and there would be magnified greatly.
When it comes to defensive evaluations, you simply cannot ignore the issue of sample size. Limited data samples can be more misleading than informational. If you don’t have a big enough sample, ignore the data.... [W]e need to use the best available information we have, and in cases like [Betancourt's], that’s scouting reports.