What metrics-driven games have a hard time measuring

Raph Koster's on a tear these days on the theory and practice of game design. Today, it's a fab little sermonette on why it's not right to sneer at data-driven, "free-to-play" games that use extensive instrumentation to make games that captivate players' attention without a lot of flair or imagination. But Koster has a codicil to his message embracing metrics-driven game design: there is much that is important about games that isn't captured by metrics:

And the more the audience divorces itself from we who make their entertainment, the more important it is that we be clear-eyed about what their tastes and behaviors actually are. And that, in turn, greatly undermines the value of "experts," — because we are in many ways, the most likely to be hidebound and unable to see past the blinkered assumptions precisely because we built them up with hard-won experience.

But! And it's a big but.

Sometimes, though, what works only works within the field of measurement. If it turns out one of those useless mewling babies was going to grow up to be Einstein, we would have been pretty dumb to toss him out when he was a sullen teenager (even if he did get good grades). A lot of things fall outside of the typical field of measurement.

* Anything that unfolds over a very long period of time. By the time you have true long-term data on a split-test, you've essentially chosen a path through inaction.
* Anything that lies in the realm of emotion is invisible — we can easily see results, but we cannot see, barring a focus group, the whys for a given action. (There are various measurement techniques, such as net promoter score, which try to get at this indirectly).
* Anything that is a short-term loss for a long-term gain. Many sorts of behaviors players might engage in may pay out when considered as a systemic aggregate, even though regarding them as a funnel may show them to be terrible. One example might be character customization — it's an extra step that likely costs some users in an F2P funnel, but it may also yield far greater revenue over time due to character customization.
* Anything that exists outside of the game proper, where it can be hard to tie cause and effect together. Examples include things like community development, the value of strategy websites built by players, etc.

Koster endeth the lesson with some constructive suggestions for fusing traditional and data-driven game design to get the best out of both.

Improving F2P

(Image: Metrics and UX, sitting in a tree, a Creative Commons Attribution Share-Alike (2.0) image from alabut's photostream)