Nearly a decade ago I spent a few wonderful months living in San Francisco. I got a short contract position at a web company in SoMa. It was an exciting time to be in the industry, still before the Tech Wreck. For a wide-eyed young graduate from Brisbane it was the funkiest place to work I could imagine. There was a games room with a pool table. You didn’t have to put coins in the softdrink machine. There were free massages for staff on Tuesdays. But my favourite part, geek that I am, was every Friday when people would bring their lunch, sit around on beanbags and listen to one of the team deliver a lecture or presentation on some aspect of technology or culture that interested them. That’s where I first heard about collaborative filtering.
In January 2000, recommendation engines were still building up a head of steam, but of course now they’re everywhere. Our daily experience of the internet is so completely overwhelmed by data – email, websites, youTube videos, music, widgets - that it’s almost impossible to self-impose any kind of filter. That’s where recommendation engines come in – they help people to find content they should (in theory) like based on their own past behaviour, or someone else’s past behaviour, or the nature of the content they are seeking.
The second one – recommendations based on someone else’s past behaviour , or collaborative filtering – is used by retailers like Amazon. 67% of people who bought that book also bought these other four books.
It makes sense to apply this to consumer models. Retailers want you to buy more things, so they make it easy for you to find content you’ll like.
But it occurs to me that the algorithms of recommendation engines could be used by publishers as well. How could, for example, this modelling provide a clearer picture for book publishers sifting through piles of unsolicited manuscripts? Publishers are always searching for the next big hit and often only have the last couple of successful books as a benchmark. But what if they could turn the data into more meaningful decision-making tools? Would it enable them to make more commercially accurate calls on manuscripts they are considering taking to market?
I guess unpublished authors try (most often unsuccessfully) to do this in their queries. “My young adult fantasy would sit comfortably next to Harry Potter…”
But it is an interesting thought-experiment to consider how recommendation engines could guide content publishing decisions not just consumer purchasing decisions. It may require manuscripts to be tagged or scored in some way, which ultimately could be as onerous as simply assessing them subjectively or qualitatively based on an editor’s experience and instinct. Perhaps it’s something that could be combined with a crowd sourcing platform to harness the power of a pool of readers or fans.
Kate Eltham is a writer and creative industries professional based in Brisbane, Australia. She is Chief Executive Officer of 
kate.eltham {at} gmail.com
{ 2 comments… read them below or add one }
Hmmm… something about this idea worries me… maybe I just don’t like the idea of my work being ‘commodified’ in this way. But practically, the problem is one of time lag. Amazon’s recommendation engine works because it’s immediate and up to date. But books take a long time to go from unsolicited ms to finished product, and in that time things can change – the zeitgeist, among others. Publishers are looking not only for something like what worked before, but also the new, the next big thing. For that they need their intuition, common sense and, yes, knowledge of what’s worked before. They also need to be prepared to take a risk if they like the book enough. But I can see a process like you’ve described being hijacked by the accountants to prevent any risk-taking in publishing at all!
Pamela: As a fiction writer I don’t find it very comforting as an idea either, and am least encouraged by the belief that most of the editors I’ve met would probably pour scorn on it, but it’s interesting to conduct the thought experiment!