Baseball Stats and BI Musings Part II: Data Quality

In Part I, I took a run at assessing a couple of the most popular baseball statics to see how they measured up as well-formed performance metrics. The other thought that has been running through my mind as I’ve been scoring my son’s baseball games has to do with data management and data quality.

Scoring a baseball game requires a couple of things:

• Making judgment calls as to what actually happened
• Capturing the right information on screwy plays where a lot of stuff happens (this happens a lot more in 9-year-old baseball games than it does in college or professional games)

The first item is one of the reasons why college and professional games have an “official scorekeeper.” There are some plays that are clearly fielding errors…but there are some that require a subjective assessment. And, even if there is clearly an error, it’s sometimes subjective as to whether it was a bad throw or a bad catch.

And, things can get a little complicated. For instance, if you look at this picture closely, you’ll be able to tell that my son is churning his 9-year-old legs as fast as he can (admittedly in pants that would fit most 12-year-olds) as he runs towards first base. And, yet, the catcher is standing right at home plate with the baseball, looking like he’s about to make a throw. What’s going on is either totally obvious to you — meaning you played baseball or have followed it with a decent level of interest — or it seems very bizarre. My son had just struck out. The rule in baseball is that, if a player strikes out AND the catcher drops the ball AND EITHER first base is unoccupied OR there are already two outs in the inning, the catcher needs to retrieve the ball and either tag the batter or throw the ball down to first base so the first baseman can tag first base. This is what’s called a “strike him out, throw him out.” You don’t see it very often in the major leagues or college, because catchers don’t drop that many balls. You see it quite a bit when the players are nine and ten years old.

Either way, my son had an official at bat with a strikeout, even if he made it to first base safely (if, for instance, the catcher overthrew first base). If that had happened (in this case, it didn’t), I would have needed to record a strikeout as well as an error on the catcher.

Sound complicated? It is, and it isn’t. Baseball has other semi-obscure rules — if a baserunner passes another baserunner, he is out. I didn’t learn that rule until I saw it happen to Baylor in the College World Series several years ago. So, scoring a baseball game correctly requires:

• Paying close attention to every play throughout the game
• Knowing the rules well
• Knowing how to quickly and accurately record both “normal” plays and oddball plays
• Being able to make the subjective calls quickly and effectively

I’ve never actually tried to verify this, but I am fairly certain that, if you take three run-of-the-mill scorekeepers and have them score the same game and then compare their results, you will get three slightly different versions of what happened. Yet, we view baseball stats and box scores as being completely black-and-white.

I worked with a data management guru at National Instruments who had a Mark Twain quote in her e-mail signature that said something to the tune of: “A man with one watch always knows what time it is. A man with two watches is never sure.” (I’ve tried to look up the exact wording and confirm that this indeed originated with Mark Twain in the past, and I didn’t have much luck.) This is an excellent point, and it applies to both baseball and business.

If we see a number that appears to be precise — 73 pitches, 10,327 visits to a web site, 2,342 leads — we equate precision with accuracy. It doesn’t cross our mind that a scorekeeper might have inadvertently clicked his pitch counter when the pitcher actually made a throw over to first base to try to pick off a runner. We ignore the fact that all data capture methods when it comes to web analytics are inherently noisy. We forget that sometimes our lead management processes break down and load a duplicate lead or miss a lead. We assume that the data that gets entered into our systems by humans gets entered by a robot rather than by a human — no judgment calls, no mental lapses. And that is simply not reality.

None of this is to say that we should throw out the data. At the end of the day, the ERAs that I calculate for the pitchers on my son’s team are going to be pretty close to the ERAs that another scorekeeper would calculate. Close enough. But, it’s easy to get caught up first in assuming that precise numbers are perfectly accurate, and, then, when something happens where you see a discrepancy, focussing on trying to get the “right” number rather than asking, “Is the difference material?”

The moral? Well…baseball is a great sport!

Oh, wait. There’s more. Don’t rely too much on your data. Don’t expect it to be perfect. Don’t focus on making it perfect. Make sure it’s “good enough” and go from there.