Now That's a Complaint.....

From Concurring Opinions (and elsewhere), a paper by Chris Hoofnagle "Measuring Identity Theft at Top Banks." Hoofnagle is asking the question: How does a consumer or regulator measure the incidence of identity theft from a financial institution? In an attempt to answer, Hoofnagle took the number of identity theft complaints collected by the FTC and matched them up with institutions listed on the complaint, with the intent of coming up with a score that could be used by consumers to judge how well the institution protects identity.

Call me crazy if I'm wrong, but Mr. Hofnagle seems to be pushing the data way beyond its utility.
Is a complaint to the FTC via a web form a reliable indicator of fraud controls at an institution? In my past experience as an investigator, I handled many cases of identity theft. I'd estimate that at least half, if not two thirds of the allegations of "identity theft" were not, in fact, identity theft. A suspicious charge on a bill, a bad skiptrace, or even a breach disclosure notice could result in complaint of "identity theft." Crime statistics that involve prosecutions of actual criminals may provide an underreported, but more reliable measure.

Hoofnagle mentions that he believes the number of FTC complaints may be low, due to historic underreporting of identity theft to criminal authorities. Again, according to my experience, which may be non-representative, I'd say that people will fill out a web form that belongs to the FTC sooner than they'd call the police. The FTC is more analogous to the Better Business Bureau than law enforcement.

I was going to write something about my frustration with the publicity that the FTC complaint statistics were receiving. Complaints are easy to count and a handy metric. But I don't think that they mean much without some evaluation of the validity of the complaint. That is, what is interesting is hard to find out.

Right before I read Hoofnagle's paper, I read this post from the Microsoft Security Development Lifecycle blog. The author makes the following statement regarding using vulnerability counts as a measure of software security:

"Measuring security is a real challenge, and while we may debate the
merits of vulnerability counts, right now it's the only concrete metric
we have."

I guess I'm saying that the only concrete metric one may have may be misleading, inaccurate, or irrelevant. Concrete isn't synonymous with valid. I may have issues with "metrics" but I love Metric. Need less, use less, we're asking for too much I guess, cause all we get is...