Wednesday, May 11, 2005
Is it a tennis tournament or a Pride parade? Apparently, at this week's German Open in Hamburg, they play The Weather Girls' "It's Raining Men" when they have to put the roof covering on the stadium. Score another point for Germanic literalism.


Monday, May 09, 2005
After spending the past few years using software to analyze text, I know the limitations of this approach. Computers are fantastic at counting words, identifying unique items, locating where words occur near one another, and even at figuring out when a text contains 'non-standard usage'. What a computer cannot do is read and comprehend tone and sense. You can program it to look for keywords that might indicate a particular tone (i.e. "disgusting"= outrage or horror), but when it comes to parsing and taking away semantic meaning, computers are ill-suited to the task.

Which is why essay-evaluation software has always disturbed me. It doesn't actually provide an assessment that even comes close to what a human could offer, and it instead takes shortcuts to identifying characteristics that someone-- the programmer(s)-- thinks are indicators of good writing. Because software can't actually understand what is being written, there's no other choice but to create a list of hallmarks of excellent writing and then try to compare a sample text with that list. Imperfect, but it might be a helpful tool for honing and restructuring a piece of writing. It should never be used to actually assign a grade for a course, though. The example from this article is a perfect illustration of why this is the case-- it is possible to fool e-Rater software into assigning the highest score to a piece of text that matches its criteria list, but which is completely off-topic. By that same token, it should also be possible to fool this software the same way junk e-mail writers work around spam filters-- by examining what triggers the software to reject an e-mail and just avoiding those items. Do we really want the next generation of high school/college/graduate students to learn how to write in order to satisfy the impoverished definition of good writing imposed by a piece of software? I certainly don't.

Again, I think that the core of this problem is the quantitative/qualitative tension that exists in so many social sciences (and increasingly in natural and 'hard' sciences as well). Using a piece of software to gauge the merits of an essay is the same as using quantitative tools to evaluate something that is, at its very heart, a qualitative product. Is it really such a surprise then that adding the word 'chimpanzee' to a letter of reference is enough to fool a computer into thinking that it is reading a superior GMAT essay? No, because from a quantitative perspective, any text could be worthy of a 6. A qualitative perspective-- where meaning blossoms-- is necessary to grade an essay, and only a human can provide this. I really doubt this will ever change.














? britblogs # DIY Blogs