Whether it’s user-generated content like YouTube, user-written and edited knowledgebases, like Wikipedia and Freebase, or user-centric Identity like OpenID and Information Cards, user-driven thinking is transforming our world. With VRM– Vendor Relationship Management–that revolution reaches the market, creating tools for individuals to get more value out of their relationships with Vendors. The goal is to create a user-driven market, where individuals engage with vendors on their own terms, creating mutually beneficial relationships that generate new value for everyone involved.

So what would it mean to apply user-driven thinking to Search? Traditional search is a mix of user-driven and vendor-centrism. While users can enter any query and be directed to content anywhere on the ‘net, we can’t share our search history with Search Providers of choice, nor do we have control over how our activities are tracked and utilized. There are few, if any, open standards for the searcher side of the experience and few options for moving beyond traditional query-response Search.

At the VRM Workshop 2008, we fleshed out some ideas, building on the thoughts introduced in my previous post, as well as ideas discussed at VRM 2008 in Munich and IIW2008a in Mountain View. What I love about the conversations at these unconferences is that they are so rich, literally creating value on a moment-to-moment basis. And these were no exception.

Here’s what has emerged so far regarding User-driven Search.

1. User-driven Search is bigger than query/response.

User-driven Search is more than what we type into the query box and the results we get back from Search Engines. It covers an entire set of activities that span the Internet, including searches entered at site-specific Search Providers like Expedia, the USPTO, and Circuit City, and all the web pages we visit in-between. It is inherently cross-silo—even non-silo—as it encompasses all of our online efforts around a given Search topic.

A recent Google/Comscore study found that the average Travel searcher takes 29 days from their first Search until their first online purchase. These advanced Searches don’t take place all at one Search Provider nor do they usually happen all in one sitting. Users need tools that empower them to manage these advanced, multi-site, multi-session Searches.

2. Users should be able to activate and deactivate Search and tracking easily and at will

With User-driven Search facilitating advanced searches across the entire scope of our online activity, users need to be able to turn it on and off at will. Sometimes we want help and are willing to share to get it. Other times, privacy is preferred. We need to be able to turn off the surveillance and just do our thing. Unfortunately, traditional search and advertising networks don’t let us do that in any reasonable way.

There are ways to disable Doubleclick’s tracking and we can tell Google to stop personalizing our search results—if we also turn off our Search History. Yet most people have no idea this is possible and even more aren’t technically comfortable enough to mess with cookies or custom preferences. We shouldn’t have to jump through hoops to disable tracking, because if that’s the case, the vast majority of users will simply not do it, and even those who do will often opt-out completely, which means there really isn’t any choice at all. The decision shouldn’t be between using advanced search features or being treated like a digital transient. We should be able to get advanced features just when we want them and simply turn them off when we don’t. That choice needs to be transparently obvious and easy and available right in the Search interface.

3. Compartmentalization

When treating Searches that span more than single queries, users need to be able to separate them into their natural topical breakdown, in whatever way makes sense. Collecting our entire search history and/or clickstream into a single attention datastore literally destroys the context that makes the Searches relevant.

Users need a way to collect their Search-related activities into categories that make sense for them. We’d like to keep our summer vacation search activity together, yet separate from our financial planning Search. We’d like to collect our home buying search activity and store that in a different place than the queries and discoveries related to our child’s Search for information about George Washington. User-driven Search must deal with more than query/response and yet not so much that it encompasses our entire attention stream. It must capture the sweet spot of user-defined collections at a scale suitable to each Search individually, as determined by each searcher.

4. Visability and Editability

For users to drive Search, we need to be able to see and edit the all of the information used to provide results. Hidden or unauthorized data or tracking of our clickstream allow current Search Providers and advertising networks to analyze and guesstimate what we are looking for, but they don’t provide any way for us to contribute. Not only are they hiding in my virtual closet surveilling me—often without permission—they are missing a great opportunity to simply ask me what I want. By making all Search activity visible, Providers can say “Here’s the data we are using to try to help you.” By making that editable, they add “Can you help us improve it?” User interface challenges aside, there is no reason Search Providers shouldn’t ask for feedback and input. It is guaranteed to improve the quality of their view and ultimately their Search results.

Currently, Google, and its DoubleClick division, track your entire search history and just about anywhere you might go online, yet you have no idea what information they have on you, except for Google’s Search History—and you certainly can’t edit it. So when you track something down on a lark, or someone else uses your machine, irrelevant data gets bundled into your history, only to clog up the machinery that is actually trying to help you. Buy a book on knots for your young cousin and Amazon will be recommending Boy Scout titles for months. This is sometimes referred to as the “Tivo thinks I’m gay” problem. If users have neither visibility nor control over the data used for recommendations, they can’t correct these types of errors. We must have both visibility into the data driving advertising and search results, and we must be able to edit it as well.

5. Selectable disclosure on users’ terms

Having gone to the trouble to coordinate and maintain a collection of data for their Searches, users should be empowered to share that data with any service capable of responding intelligently. Search is a fundamental part of how we navigate the web; it makes no sense to restrict Search activity to any one provider. Just as your Search might take you to dozens of websites, it is also possible that it will bring you to dozens of Search Providers, from Google and Yahoo! to Amazon and eBay, even to microSearch Providers like Circuit City or Schwab. As users navigate across the web, their Search should go with them, seamlessly disclosed to authorized Search Providers as easily as possible.

Today, Google serves as a locked-in data silo for most people’s search history. There’s no way to send that history to Yahoo! Or MSN Live or Amazon or eBay to see what they might be able to do for you. As technologies for personalized search results improve, the value of that search history will continue to increase. We need to be able to send select parts of our search to providers of choice and we need to be able to do it trivially. As easily as we go from one website to another, we should be able to send our Search to a new Search Provider.

And yet, if we are to facilitate the easy transfer of this data, we also need to protect users’ rights, even as we expose more secrets to more people.

Schwab, for example, could greatly improve the ease of finding appropriate offerings if they could review the relevant parts of the current Search instead of relying on you entering just the right queries and properly navigating their site architecture. But it is unlikely that users are going to want to give Schwab any information unless there’s an understanding about just exactly how that information will be used (and the ability to select just what information is sent). We generally don’t want companies to start sending us junk mail or calling us with sales offers just because our Search shows that we are in the market for one of their products. But, if we could be assured they would use our Search just to provide better results and perhaps to improve their offerings, we are far more likely to share that part of our Search that could help them help us. We want explicit agreement for data rights access and we want it before we give them any data, and when we want to select what we send so they get just the parts that make sense, and not any personal information we don’t want to share. A User-driven Search solution must not only allow users to send select portions of their Search wherever they want, it must allow them to set the terms for exactly what recipients can do once they get it.

6. Impulse from the user as a specific statement of Search Intent

Recommendation systems presume that an analysis of your history is the best way to discover what you might want now. The NetFlix recommendation challenge and Amazon recommendations feature both use this approach. Not only does this place the user in a passive mode, it also has no facility for users to state what they actually want, right now. People have widely varying interests and easily switch between tasks even in the middle of a Search. Our past transactions may paint an interesting picture of who we are, but it rarely describes what we want in any given moment. What we really want from NetFlix isn’t the “perfect” movie for someone with my viewing history, we want the movie that’s perfect for the mood or situation we’re in right now.

Search systems, on the other hand, rely on a specific “objet de Search” as a trigger for directing efforts. The object de Search is a keyword or other statement that explicitly represents the user’s intent in some way. At traditional search engines, the query serves this purpose, with the user essentially asking “what web pages have these words” in the hope that those words might be on the page that has what they are actually looking for. At structured Search Providers, like Expedia or Orbitz, the entries for departure, destination, date, and number of travelers in the combined form data comprise the objet de Search.

For User-driven Searches, we must move beyond the keywords and limited structured form fields to allow a more complex, more expressive statement of intent. This statement should include the entire range of Search activity for your given Search, including queries, Search Providers, clickthroughs, captures, and annotations. In short, it should bundle up the entire Search and present it to the Search Provider as an explicit statement of intent. This presentation must be independent of any data silo, unlimited by the offerings of any particular vendor. It should be a proactive statement of “Here’s what I’m looking for: here’s what I’ve found so far and where I’ve been. Got anything that might help?”

Most importantly, Search operates in the foreground, with an explicit impulse from the user. User-driven Search isn’t about background profiling and analysis to try to guess user intent. It requires an explicit means for users to state their intent in ways Search Providers can understand. Instead of predatorial “targeting” of users with particular demographic, psychographic, or behavioral profiles, User-driven Search operates exclusively on that objet de search, as the entire representation of user intent. No more guessing. No more secretive or unauthorized tracking. No more stereotypical clustering based on industrial-era models of consumer behavior. Instead, User-driven Search Providers respond directly to clear, unambiguous representations of confirmed user intent.

Towards an Open Standard

This is the kind of solution we are working on at SwitchBook. At the VRM Workshop 2008, I was excited to learn more about MatchMine from J Trent Adams; they are moving in a similar direction for media-based recommendations. There is currently no service we know of that fully delivers on the promise of User-driven Search, but I’m looking forward to working with Trent and others to develop the open standards and protocols to make it possible.

If you are interested in joining the conversation, send me an email. We’ll be setting up a listserv to talk on a more regular basis. All are welcome.

It is time to give users more control over Search.

At VRM2008 in Munich and at IIW in Mountain View, I started a conversation about User-Driven Search, the premise: what would it mean for users to truly drive their searches?

User-driven is a new term that came out of the VRM community riffing on the meaning of user-centric development and user-centric identity. User-centric is a nice term, but it could be construed as limiting. For example, user –centric definitely implies that that the user is at the center of attention and the focus of the architecture, but it doesn’t necessarily mean the user is in charge of the experience. That’s a key distinction.

Not just tuna salad

Adriana explains this difference between user-centric and user-driven as metaphorically the difference between buying ready-made tuna salad or picking and choosing your own ingredients and making the tuna salad yourself. When I first talked with Doc about user-driven instead of user-centric, Jim Carrey’s The Truman Show immediately sprang to mind: from birth, Truman is the protagonist in a huge reality show revolving around him… only he doesn’t know it. The climax of the show is Truman discovering the rest of the world and confronting his father/producer. Clearly the Truman Show is Truman-centric… but it is most definitely not Truman-driven.

It’s about impetus and authority

For me, user-driven means that the user provides the impetus and is the controlling authority throughout the transaction. Sure, sometimes there is negotiation or collaboration with others… the user isn’t omnipotent, after all. However, the user is in charge of creating his or her own experience. This fits with user-constructed or customized solutions, like the tuna salad recipe. However, it has implications far beyond the limits user-created or user-customizable architectures.

Is the user initiating the experience? Is the user’s moral authority the primary control throughout the system? Is the system transparent to users, enabling them to make their own informed decisions about what will be presented to them and how it is presented? Is it the user who is shaping the input, intermediary results, and final outcome? If so, then it is user-driven. If not, it isn’t.

When it comes to the tuna salad metaphor, this is the equivalent of the tuna salad being made when I ask for it and on my terms. Not before. And although I could choose to make the salad myself–that is definitely user-driven–it could also be made by someone else to my specifications… extra mayo and black pepper, no onions, thank you.

Search as user-driven

Google’s keyword query-response approach to Search is, of course, user-driven to some extent. Nothing happens until the user enters a query, users are free to enter any query, and the system responds with results tailored for exactly what the user queried. The user does shape the experience to a limited degree. And yet, it still provides only a slim façade of user control. There is no way to modulate the algorithm, no way to let Google know which results are good or bad, and no way to refine the search other than keyword guessing games. And, perhaps most importantly, there is no way to manage the search beyond the immediate query. For that, the user is dependent on other techniques: bookmarking, cut & paste, opening multiple windows or tabs, even printing to paper or PDF to keep track of good finds. Evolution in Search History management is starting in the right direction, but the ideas here have been rather uninspiring so far.

User-driven systems create value inherently

The limits on the user-driven aspects of Google are particularly ironic given that it is precisely the element of user control that creates Google’s greatest asset: focused attention. Google’s money making asset is the collection of user-specified queries, queries that explicitly state words related to the user’s interest and implicitly denote user intent. It is precisely because the individual enters a specific keyword that Google is able to sell targeted ads… at great profit and benefit to advertisers and searchers alike.

The query entered in the Search box gives Google a implicit statement of intent. That intention is the gold Google resells to advertisers. If Google didn’t let users drive that intention, if they looked more like a content site or “Internet portal”, they’d have a lot less intention to monetize.

If we can extend that control, if we can make search even more user-driven, if we can enable richer, more explicit, more user-driven expressions of Search intent, I believe we can create even more value for everyone involved: search companies, advertisers, searchers, even non-paying websites showing up in “organic” results.

What does it mean to have User-Driven Search?

At SwitchBook, we’ve been doing a lot of thinking about what User-Driven Search might mean. I like starting the conversation with a simple example: what would it mean if I could take my search history from one search provider to another? This “dataportability” example is just an initial notion of how Search might become more user driven.

So, what do you think of when you hear (or read) “User-Driven Search”?

I’ll be leading a session on this topic at the VRM Workshop next week. I hope you can join us.

This material is based, in part, upon work supported by the National Science Foundation under Grant No. 0740861. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).