Monday, April 28, 2008

Cruel But Fair: The IT Auditor's Ball

There is no need to remind me how I dislike Las Vegas. As the woman walking away from the conference this afternoon said, "casinos are full of weird people." And she wasn't talking about her fellow information systems governance professionals.

Well, I'm almost live blogging the event (no wireless connectivity? 20 lbs of printed procedings? CACS is old school, baby!) from the IT Audit bloggers meetup (the attendees so far: me & a bottle of cheap scotch).
So what did I learn on my first day at the North American Computer Audit Control and Security Conference?

1. Dumb user jokes still get a laugh. The dumb user jokes need to end now. Really. It adds nothing, and only confirms everyone's opinion that security and audit people are arrogant and condescending. More on this later.

2. The "I am not a lawyer" defense to compliance. If something is too unpleasant, or unsavory, yet explicitly outlined in law and regulation, there is a tendency to punt the enforcement to legal. Cause, you don't want to practice law without a license. You know, cops aren't lawyers, either. Nonetheless they enforce the laws. This is an issue that can be solved, and likely has been, between auditors, security practitioners and lawyers.

3. The ice machine on the 13th floor of the Rio is broken. This is the thoughest lesson I've learned. But experience is a bitter and effective teacher.

4. Can gaussian distributions be helpful in analysis of breach disclosure? My butt was in the wrong seat to attend this talk, but the slides were curious (mostly because the color-coding in the pie charts didn't work in the B&W procedings). I would have been interested in hearing how that would work. I don't have the depth in stats to have flung anything at the presenter, but I may have had the guts to shout "HERETIC."

Soundtrack for today: "Raving & Drooling"

Wednesday, April 16, 2008

Metrics Gone Wrong: Horsepower at 100% Throttle


In the April issue of Bike magazine, Simon Hargreaves examines the myth of the dyno. The rise of the the Dynojet Dynamometer provided a cheap, standard way to measure motorcycle horsepower, allowing a common manner to rate the impact of your performance tweak. Roll your bike up to the rollers, and wind it up to full throttle. Moments later, the dyno spits out a pretty graph with torque and horsepower. (I recall a sweaty, restless July night at Texas World Speedway, the motorsport jewel of the Bryan/College Station where my buddy and I parked the VW camper van next to the dyno. Yosh pipes howling through 100% throttle get old after about the 15th carb rejetting, but the dyno truck's jam box pumping out interstitial "Give It Away" got old after the 5th round. )

None the less, Hargreaves cites the problem with a standard measure:

First, higher horsepower figures than the manufacturer next door sells more bikes than him, though - second - higher horsepower figures bring anti-biking legislation closer and closer, despite the fact that - third - accident figures aren't related to increased power, even though - fourth - the performance of your three 160hp models comfortably exceeds the ability of your customer to get anywhere near using it all without crashing.
The answer is measuring 40% and 20% throttle as well. The nebulous corner exit power that was measured only in sphincter tension or nebulous terms like "grunt" and "oomphus" is now a value that can be colored red, blue or green and plotted on a pretty graph. And a telling graph it is, as the GSX-R1000 appears to have dropped power at 20% throttle (to reduce highsideability) while maintaining the pornographic 160hp at top.

So, the top number, the easy number, the number of honorable tradition, means less and less once it is maxed. The tweaks underneath where there, and important. But you are stuck with your gut feeling until you plot it with a pretty blue line.

Monday, April 14, 2008

Metrics Gone Wrong: Body Count

From the Washington Post, and which also I heard on the radio this morning, the Colombian army finds a twisted method to meet their performance metrics:

But under intense pressure from Colombian military commanders to register combat kills, the army has in recent years also increasingly been killing poor farmers and passing them off as rebels slain in combat, government officials and human rights groups say. The tactic has touched off a fierce debate in the Defense Ministry between tradition-bound generals who favor an aggressive campaign that centers on body counts and reformers who say the army needs to develop other yardsticks to measure battlefield success.
This is the most extreme example of how a metric intended to track progress toward a goal becomes a measure of performance for the implementers. Focussed on the finger pointing at the moon, rather than the moon itself, the implementers manage the metric but undermine the goal. I don't believe this behavior is uncommon. I saw this sort of behavior in a past life as a fraud examiner. An individual forged a stack of documents, because he understood more documents were good for the company, their legitimacy only an inconvenience.