Long-tails and beaten tracks : The Universal Mind

Flickr.com has turned into an amazingly good search engine… I noticed it the other day when I was looking for photos of Pomanders (and who ever does that?). There’s a group dedicated to them… or photos of them.

I read recently that Youtube is the second biggest search engine on the web (in a blog that I’ve now misplaced) by one of the funders of Twitter, who was saying that Twitter was a threat to google because it delivers more up to the minute information… a notion roundly bollocked by the hoards of commenters etc.

Twitter is a pretty good search engine for what people are talking about on any given subject right now though. Want to know what people think of IE6? Search for it on Twitter. Hating IE6 is an international language.

Facebook is an excellent search engine for people. I’ve found people that I thought I’d lost forever.

Digg/Reddit and co have fucked it up. They’re not good search engines for news.

Instructibles.com is rappidly turning into a really good site for searching for “how to build things”… and that is its destiny methinks… but it’s not quite there yet.

There are probably good sites for searching for music… MySpace? Lastfm? I imagine that the backward facing copyright analists have/will interfere with that one… I use youtube for music searches and they interfere with that all the time.

So there you go. Web 2.0 is turning into Search 2.0. Who’d a thunk it?

And in a way it makes sense, because although google is utter genius, it’s also a bit crap because (like digg, reddit etc) it’s an exercise in Rewarding Beaten-Paths. Sure the long tails are there… but if you want to find an honest review of a camera say, you have to wade through about a million pages of people trying to sell them. If you want authenticity, you need to go to Amazon (a search engine for not just things to buy, but reviews) or some other site that specialises in reviews (and there are a fair few of those, but I’ve not found them to be terribly convincing) … where the focus is small enough that the interests of long-tails are genuinely served.

It’s classic AI – and quite possibly/probably how wetware brains work… every time you click a link, you increase the likelihood that it will be clicked again. Beaten-Paths are rewarded, but beaten paths are not always right, and can be (and are) gamed.

It doesn’t always work. It’s not always fair… in fact the bigger it gets, the less likely it is to be fair. Oligarchies of Influence are formed. It’s almost like (and this is an advantage of twitter/facebook) that classic AI isn’t that good for delivering authenticity… for that you need small networks of small-degrees-of-separation of white-listed people. “People you know”, or “who know people you know” or “who no people who know people you know”, but not much beyond that. You’re not guaranteed an answer though.

Another problem – a concern I have with the way these things are shaping up, is that this system encourages monopoly. The major filtering systems of the internet are specifically, systemically architected to encourage monopoly. It’s not democratic… it’s sold as democratic, but it’s not democratic.

I’d be a lot happier (for example) with some sort of P2P version of Flickr… so once something is uploaded into the Universal mind, it doesn’t get lost… but it doesn’t have the lack of resilience inherent in monopolistic systems. We’ve already seen the drawbacks of monopoly with Youtube… embedded a video recently? Chances are it will have been pulled by WMG – and your site will now have a dead link.

I think any P2P system will need to run in parallel… a bit like the distributed SETI system… where chunks of the Universal Mind that live in precarious silos like Youtube or Flickr are automatically backed up onto individual PCs… I think there are already sites that try to backup every single youtube video onto their own servers… but this is kindof replacing one problem with… well, the same problem.

This is probably already being done, but I don’t know about it.

Here’s a picture of a penguin

Apropos of nothing.

I’m also increasingly seeing the web as (you guessed it) The Universal Mind. It’s becoming a creature in its own right, with its own patterns and behaviours, and its own emergent moralities… but more of that later.