Type I vs. Type II Errors in Customer Data Management

[Update: See the comments at the end of the post. I might have my definitions backwards, but I need to do some digging to make sure I get the corrections right. The underlying point of the post — that, in customer data management, you’ve got to recognize that false positives and false negatives are not the same thing and situationally err on the side of one or the other — still holds. But, it’s not good to have the terminology reversed!]

Last week, I ended a post promising a future post on Type I vs. Type II errors when it comes to customer data management. I’ve found myself running into confusion on the distinction, with all customer data errors being treated as the same type, when they are not.

Let’s Start with Definitions

Wikipedia has a nice write-up that goes into much deeper detail, but here are my quick and crude definitions:

• Type I Error = α (Alpha) Error = False Positive = you mistakenly judge something to be true that, in fact, is false
• Type II Error = β (Beta) Error = False Negative = you mistakenly judge something to be false that, in fact, is true

It’s easy to get mired in statistics-speak and make a snap judgment that this distinction between types of data errors are solely of interest and use to statisticians. That’s not the case. Depending on the situation, one type can be critical while the other can be barely consequential.

Customer Data Example: Automated Customer Merges

Most companies of any size battle duplicates in their customer data. I have yet to work at or with a company where the sales force and call center staff don’t complain about the number of duplicates that exist in internal systems. In this situation, it’s pretty common to try to automate a deduplication routine of some sort (some would say that this is the biggest benefit of a customer data integration, or CDI, initiative).

Unfortunately, customer data is messy. For every “identical match” in the system, there are generally 5 to 10 “probable matches.” After all, if the data was truly identical, then the likelihood that a duplicate would have gotten created in the first place would have been greatly diminished! That means that the automated matching system has to apply various business rules to determine which matches are true matches and which ones are not — the system makes an educated guess. Many CDI systems apply a scoring system — the higher the score, the more likely the two records are a true match. The lower the score, the less likely. It’s then up to the user to establish the threshold above which the records will automatically be merged (and, possibly, a lower threshold that will trigger a manual review of the records by a human being).

Where that threshold gets set is purely a Type I vs. Type II error decision:

• The higher the threshold gets set, the greater the likelihood of a Type II error — a false negative where two records could have been merged because they were, in fact, duplicates, but they were not identified as such
• The lower the threshold gets set, the greater the likelihood of a Type I error — a false positive where two records get merged, even though, in reality, they represent two different customers

It would be wonderful if the matching logic was such that a threshold existed that eliminated all errors, but no such threshold will ever exist. The question then becomes: which type of error is “more” acceptable? That will influence where you set your threshold. It’s a situational question! For example:

• If your customer data includes extremely sensitive information (think medical records, social security numbers, credit card numbers), then you will want to err on the side of Type II / False Negative errors — deal with the messiness of more duplicates and put the burden on your call center / sales force to trigger merges if and only if they have confirmed the merge needs to happen through a manual inspection and/or an interaction directly with the customer
• If, on the other hand, you are only using the post-merged customer data to send marketing promotions to the customers, then the added cost of the direct mail may push you to err on the side of Type I / False Positive errors — some of your customers may miss out on some marketing promotions, but very few of them will receive multiple copies of the same direct mail piece, and your overall postage costs will be lower

This is a very real example — a single company may use the former mentality as its core matching logic, but then maintain a separate data store with more aggressive matching for purposes where Type I errors are not critical.

Type I and Type II in the News

A friend and former colleague shared a recent story in The Washington Post about the Social Security Administration running into trouble when trying to deny social security benefits to certain felons. It sounds like there were some technical gaffs, for sure — when people with minor offenses on their record got caught up in the denial of benefits. Overall, this reads like it was a case of an unacceptable number of Type I errors — people being denied benefits because they were mis-identified as “fleeing felons” (the article reads like some of the stories of people being banned from flying because they had the same name as someone on the terrorist watch list).

Both types of errors will occur. As important as being clear on which type are “better” given the situation (and developing your processes accordingly), ensuring that you have processes in place to fix the errors when they do occur and are identified is just as critical — something that the federal government was apparently missing in this case! The federal government? Lacking in customer service? Shocker!

So, there you have it — more thought than you ever wanted to see on Type I and Type II errors. I also promised in my earlier post that a future post would cover my observations on cognitive dissonance and the status quo when it comes to customer data management, but that’s going to have to wait for another day!