Steven Johnson (author of the fantastic Mind Wide Open and other books) has written a fascinating essay about his new creative process, which involves a suite of tools that store his notes and works in unstructured databases, and tease out and suggest subtly connected ideas, so that as he writes, his computer jams with him, suggesting neat tangents to his subjects. It's a great example of good computer-human interaction, where computers are used to programatically count and compare quantifiable elements (word and phrase frequencies) and human beings are used to pass judgement on the output of the computers. People are good at understanding and crap at counting; computers are just the reverse.
Johnson's piece is a thought-provoking look at how productivity software can really change the way that you work — that you think! Writing in the era of these tools is truly a different undertaking than the writing of old.
Think of all the documents you have on your machine that are longer than a thousand words: business plans, articles, ebooks, pdfs of product manuals, research notes, etc. When you're making an exploratory search through that information, you're not looking for the files that include the keywords you've identified; you're looking for specific sections of text — sometimes just a paragraph — that relate to the general theme of the search query. If I do a Google Desktop search for "Richard Dawkins" I'll get dozens of documents back, but then I have to go through and find all the sections inside those documents that are relevant to Dawkins, which saves me almost no time.
So the proper unit for this kind of exploratory, semantic search is not the file, but rather something else, something I don't quite have a word for: a chunk or cluster of text, something close to those little quotes that I've assembled in DevonThink. If I have an eBook of Manual DeLanda's on my hard drive, and I search for "urban ecosystem" I don't want the software to tell me that an entire book is related to my query. I want the software to tell me that these five separate paragraphs from this book are relevant. Until the tools can break out those smaller units on their own, I'll still be assembling my research library by hand in DevonThink.
(There's also an accompanying NYT editorial that Steven wrote, but I can't get into it since the Times so aggressively blocks bugmenot passwords. If you think that newspapers should have the right to positively identify their casual readers, you can create a login and read this. Not me, though.)
(Thanks, Steven!)
