Notes on User Driven Search

Whether it’s user-generated content like YouTube, user-written and edited knowledgebases, like Wikipedia and Freebase, or user-centric Identity like OpenID and Information Cards, user-driven thinking is transforming our world. With VRM– Vendor Relationship Management–that revolution reaches the market, creating tools for individuals to get more value out of their relationships with Vendors. The goal is to create a user-driven market, where individuals engage with vendors on their own terms, creating mutually beneficial relationships that generate new value for everyone involved.

So what would it mean to apply user-driven thinking to Search? Traditional search is a mix of user-driven and vendor-centrism. While users can enter any query and be directed to content anywhere on the ‘net, we can’t share our search history with Search Providers of choice, nor do we have control over how our activities are tracked and utilized. There are few, if any, open standards for the searcher side of the experience and few options for moving beyond traditional query-response Search.

At the VRM Workshop 2008, we fleshed out some ideas, building on the thoughts introduced in my previous post, as well as ideas discussed at VRM 2008 in Munich and IIW2008a in Mountain View. What I love about the conversations at these unconferences is that they are so rich, literally creating value on a moment-to-moment basis. And these were no exception.

Here’s what has emerged so far regarding User-driven Search.

1. User Driven Search is bigger than query/response.

Paris ResultsUser Driven Search is more than what we type into the query box and the results we get back from Search Engines. It covers an entire set of activities that span the Internet, including searches entered at site-specific Search Providers like Expedia, the USPTO, and Circuit City, and all the web pages we visit in-between. It is inherently cross-silo—even non-silo—as it encompasses all of our online efforts around a given Search topic.

A recent Google/Comscore study found that the average Travel searcher takes 29 days from their first query until their first online purchase. These advanced Searches don’t take place all at one Search Provider nor do they usually happen all in one sitting. Users need tools that empower them to manage these advanced, multi-site, multi-session Searches.

2. Users should be able to activate and deactivate Search and tracking easily and at will

With User-driven Search facilitating advanced searches across the entire scope of our online activity, users need to be able to turn it on and off at will. Sometimes we want help and are willing to share to get it. Other times, privacy is preferred. We need to be able to turn off the surveillance and just do our thing. Unfortunately, traditional search and advertising networks don’t let us do that in any reasonable way.

on off switchThere are ways to disable Doubleclick’s tracking and we can tell Google to stop personalizing our search results—if we also turn off our Search History. Yet most people have no idea this is possible and even more aren’t technically comfortable enough to mess with cookies or custom preferences. We shouldn’t have to jump through hoops to disable tracking, because if that’s the case, the vast majority of users will simply not do it, and even those who do will often opt-out completely, which means there really isn’t any choice at all. The decision shouldn’t be between using advanced search features or being treated like a digital transient. We should be able to get advanced features just when we want them and simply turn them off when we don’t. That choice needs to be transparently obvious and easy and available right in the Search interface.

3. Compartmentalization

forks and knivesWhen treating Searches that span more than single queries, users need to be able to separate them into their natural topical breakdown, in whatever way makes sense. Collecting our entire search history and/or clickstream into a single attention datastore literally destroys the context that makes the Searches relevant.

Users need a way to collect their Search-related activities into categories that make sense for them. We’d like to keep our summer vacation search activity together, yet separate from our financial planning Search. We’d like to collect our home buying search activity and store that in a different place than the queries and discoveries related to our child’s Search for information about George Washington. User-driven Search must deal with more than query/response and yet not so much that it encompasses our entire attention stream. It must capture the sweet spot of user-defined collections at a scale suitable to each Search individually, as determined by each searcher.

4. Visability and Editability

For users to drive Search, we need to be able to see and edit the all of the information used to provide results. Hidden or unauthorized data or tracking of our clickstream allow current Search Providers and advertising networks to analyze and guesstimate what we are looking for, but they don’t provide any way for us to contribute. Not only are they hiding in my virtual closet surveilling me—often without permission—they are missing a great opportunity to simply ask me what I want. By making all Search activity visible, Providers can say “Here’s the data we are using to try to help you.” By making that editable, they add “Can you help us improve it?” User interface challenges aside, there is no reason Search Providers shouldn’t ask for feedback and input. It is guaranteed to improve the quality of their view and ultimately their Search results.

Erasing errorCurrently, Google, and its DoubleClick division, track your entire search history and just about anywhere you might go online, yet you have no idea what information they have on you, except for Google’s Search History—and you certainly can’t edit it. So when you track something down on a lark, or someone else uses your machine, irrelevant data gets bundled into your history, only to clog up the machinery that is actually trying to help you. Buy a book on knots for your young cousin and Amazon will be recommending Boy Scout titles for months. This is sometimes referred to as the “Tivo thinks I’m gay” problem. If users have neither visibility nor control over the data used for recommendations, they can’t correct these types of errors. We must have both visibility into the data driving advertising and search results, and we must be able to edit it as well.

5. Selectable disclosure on users’ terms

Iconic courierHaving gone to the trouble to coordinate and maintain a collection of data for their Searches, users should be empowered to share that data with any service capable of responding intelligently. Search is a fundamental part of how we navigate the web; it makes no sense to restrict Search activity to any one provider. Just as your Search might take you to dozens of websites, it is also possible that it will bring you to dozens of Search Providers, from Google and Yahoo! to Amazon and eBay, even to microSearch Providers like Circuit City or Schwab. As users navigate across the web, their Search should go with them, seamlessly disclosed to authorized Search Providers as easily as possible.

Today, Google serves as a locked-in data silo for most people’s search history. There’s no way to send that history to Yahoo! Or MSN Live or Amazon or eBay to see what they might be able to do for you. As technologies for personalized search results improve, the value of that search history will continue to increase. We need to be able to send select parts of our search to providers of choice and we need to be able to do it trivially. As easily as we go from one website to another, we should be able to send our Search to a new Search Provider.

And yet, if we are to facilitate the easy transfer of this data, we also need to protect users’ rights, even as we expose more secrets to more people.

Schwab, for example, could greatly improve the ease of finding appropriate offerings if they could review the relevant parts of the current Search instead of relying on you entering just the right queries and properly navigating their site architecture. But it is unlikely that users are going to want to give Schwab any information unless there’s an understanding about just exactly how that information will be used (and the ability to select just what information is sent). We generally don’t want companies to start sending us junk mail or calling us with sales offers just because our Search shows that we are in the market for one of their products. But, if we could be assured they would use our Search just to provide better results and perhaps to improve their offerings, we are far more likely to share that part of our Search that could help them help us. We want explicit agreement for data rights access and we want it before we give them any data, and when we want to select what we send so they get just the parts that make sense, and not any personal information we don’t want to share. A User-driven Search solution must not only allow users to send select portions of their Search wherever they want, it must allow them to set the terms for exactly what recipients can do once they get it.

6. Impulse from the user as a specific statement of Search Intent

ordering from menuRecommendation systems presume that an analysis of your history is the best way to discover what you might want now. The NetFlix recommendation challenge and Amazon recommendations feature both use this approach. Not only does this place the user in a passive mode, it also has no facility for users to state what they actually want, right now. People have widely varying interests and easily switch between tasks even in the middle of a Search. Our past transactions may paint an interesting picture of who we are, but it rarely describes what we want in any given moment. What we really want from NetFlix isn’t the “perfect” movie for someone with my viewing history, we want the movie that’s perfect for the mood or situation we’re in right now.

Search systems, on the other hand, rely on a specific “objet de Search” as a trigger for directing efforts. The objet de Search is a keyword or other statement that explicitly represents the user’s intent in some way. At traditional search engines, the query serves this purpose, with the user essentially asking “what web pages have these words” in the hope that those words might be on the page that has what they are actually looking for. At structured Search Providers, like Expedia or Orbitz, the entries for departure, destination, date, and number of travelers in the combined form data comprise the objet de Search.

For User Driven Searches, we must move beyond the keywords and limited structured form fields to allow a more complex, more expressive statement of intent. This statement should include the entire range of Search activity for your given Search, including queries, Search Providers, clickthroughs, captures, and annotations. In short, it should bundle up the entire Search and present it to the Search Provider as an explicit statement of intent. This presentation must be independent of any data silo, unlimited by the offerings of any particular vendor. It should be a proactive statement of “Here’s what I’m looking for: here’s what I’ve found so far and where I’ve been. Got anything that might help?”

Most importantly, Search operates in the foreground, with an explicit impulse from the user. User-driven Search isn’t about background profiling and analysis to try to guess user intent. It requires an explicit means for users to state their intent in ways Search Providers can understand. Instead of predatorial “targeting” of users with particular demographic, psychographic, or behavioral profiles, User-driven Search operates exclusively on that objet de search, as the entire representation of user intent. No more guessing. No more secretive or unauthorized tracking. No more stereotypical clustering based on industrial-era models of consumer behavior. Instead, User-driven Search Providers respond directly to clear, unambiguous representations of confirmed user intent.

Towards an Open Standard

This is the kind of solution we are working on at SwitchBook. At the VRM Workshop 2008, I was excited to learn more about MatchMine from J Trent Adams; they are moving in a similar direction for media-based recommendations. There is currently no service we know of that fully delivers on the promise of User Driven Search, but I’m looking forward to working with Trent and others to develop the open standards and protocols to make it possible.

If you are interested in joining the conversation, send me an email. We’ll be setting up a listserv to talk on a more regular basis. All are welcome.

[Update 5/3/2009: "user-driven Search" to "User Driven Search"]

7 responses to “Notes on User Driven Search”

  1. alecmuffett

    I posted a comment on which precedes this; it sounds to me that what you are proposing above is “spy in the cab” meets “clippy the paperclip”, with promises that it won’t really leak stuff to marketeers unless you really want it to.

    “Honest. We swear.”

    Again in this scenario, exactly as before, the user is in the position of supplicant, or at best is having their little search-turds recycled into composty goodness which someone else profits from and/or sells on their behalf.

    How is this “User Driven” as-per the comment in the previous posting?

  2. alecmuffett

    If you think of Firefox’s browser history as spying on your activity, you’re right. And if you think of advanced features on the client side as “Clippy the paperclip”, you’re right again.

    Strawman alert! Strawman alert! :-) Actually I see a slavering corporate beast looking to make money off-of all my search history, to be desireous of spying upon me: “All that untapped intention- and attention-marketing information, it must be mine bwahahahahahaha!” So you see, what I am talking about is the mindset of those who wish to build a new business atop other peoples’ information.

    Currently, the entire web is watching where you surf and you have virtually no control and no way to leverage it for your own value.

    “All these people, producing this good stuff and not letting us use it to sell them stuff getting benefit from it – how tragic and uninformed they are, we must help them.”

    Giving users control over that information and providing an explicit means for direct improvements in their search results is user-driven at the very core. The whole point is to liberate users from the isolated data silos at Google and Doubleclick and NebuAd

    In what way would this be Liberation, precisely? For instance, AdBlockPlus is liberation in my book – unshackling me from crap I am otherwise forced to endure. Google goes to great lengths to ensure that the crap is me-centric, to be sure, through contextualisation and whatnot. But it is still crap.

    giving them a tool that lets them bundle their behavioral and explicit Search data into a representation that they can edit and release on their own terms.

    To whose actual benefit, and for what reason would I want to do this, given I am of the AdBlockPlus mindset outlined above?

    Having “little search-turds recycled into composty goodness” is precisely what is happening out there, on the Internet, currently, with essentially no input from the user. Your phrasing is perhaps belittling, but you’re missing the point. The way that we search, what we look for, what we find, what we keep, all of these comprise a better representation of what we are actually looking for. The paltry keyword started losing its glamor when Google started personalizing search results based on your Search History. Given that such tracking actually has the potential to improve our search experience, isn’t it time to give users control over it?

    Doubtless; you make a good case for (eg:) Google to refine its search algorithms to profile user-searches and start trying to second-guess the user; and I agree with an implicit proposition that the user should be able to download/access their search history. But given that I’ve just stated the concept in terms of “What Could Google Do?” I have to ask: how is any of this user-driven or VRM-ish?

    As you mention in the comment on the prior post, user-driven suggests that the value is, at least partially, provided directly by the user’s participation in the system, such as with BitTorrent.

    Yes, Bittorrent would not exist without the content which users provide; theoretically the Google search function *could* exist without users making search queries, although they might not continue to exist for very long…

    It is precisely the editability, compartmentalization, and selective disclosure that makes User-driven Search so valuable. Without the user to vet and refine the Search data,

    … for whose benefit? …

    all you really have is an endless jumble of attention data with no direction and no user feedback. Which is precisely what DoubleClick (now Google) and NebuAd work with.

    Yes. So you are proposing that users are masochists who will skim their own search histories for clues about their deepest darkest desires, and use that deep self-knowledge to request advertising for [whatever] – britney spears albums, maybe?

    Of course, if you’d prefer to believe that keywords are good enough, then feel free to opt-out of next generation search, which I realize you may already do. In contrast, most people, I believe, have Google’s Search History enabled

    Actually I have it switched-off for privacy reasons, plus I wonder if “most people” will have that, since the proposition implies that “most people” (say, more than 50% of internet users) have got a Google account, which I find hard to believe.

    and I expect they will continue to benefit from innovations in “composty goodness”. They just won’t have it in any user-driven way until we can extract that Search History in a meaningful way and re-use it on their own terms.

    I love the way you say “WE CAN EXTRACT” and “THEIR OWN TERMS” – who is your notion of “WE” in this context, Joe?

    Finally, “won’t really leak stuff” can be formally bound in access policy and data rights agreements. Of course, today, Google is free to hand over your search history or YouTube viewing history to whomever they please. There are neither legal nor contractual restraints to protect user’s privacy. However, with a fully implemented User-driven Search solution, explicit, contractually binding agreement to data rights policies can and will be required before releasing search data to potential vendors.

    See above comment regards “deep self knowledge”.

    I think you’d agree that technology-based DRM is a waste of time. But legally enforceable contracts with Search providers is another thing altogether.


    At the end of the day, if you want someone to help you with your search, you have to give them something to work with. The point of User-driven Search is to make that transaction seamless, painfree, entirely under user control, and bound to non-repudiatable data rights agreements.

    Yeah. I am just thinking that the added value you are here proposing atop search, does not address customer wants. The tasty ice-cream you are suggesting is also an effective floorwax.

    If you still think that misses the opportunity for User-driven Search, what would you suggest to make Search truly user-driven?

    I wouldn’t, in the same way I wouldn’t try to make a kosher bacon-double-cheeseburger. The result would either be not-kosher, or not-a-bacon-double-cheeseburger.

    [Continuing the thread from the other blog comment]

    Re: inherent, it is the value-creation of user-driven systems that is inherent. To the extent that systems refuse to be user-driven, they are leaving value on the table.

    So when you say “User-driven systems create value inherently” you mean that user-driven systems have the inherent property of creating-value; which given the definition that the user’s input creates the value of a user-driven system, is something of a tautology.

    As for user-driven search, that’s actual the term we used in our provisional patent filing in 2006. I liked Adriana’s distinctions around it, which her post credits to Bob Frankston. However, the user-as-chef remains, IMO, a fairly limited metaphor for user-driven systems. I think we might be in agreement on that, as I don’t believe “customization” is the end-all-be-all of user-driven, where it is at the core of Adriana’s usage.

    I would be very careful to avoid misrepresenting Adriana’s position as a quest for customisation; it’s vastly different and far larger than that.

    So, we may still disagree about what user-driven means, but I think we are in agreement that I’m talking about something substantially different than Adriana.

    Oh yeah. I am not sure that that is good news for you, alas :-/

    I agree that Google is partially user-driven, but it is the ways in which it is not user-driven that are the interesting things to talk about. Which was the point of the post.


    I’m not sure why Twitter is user driven.

    Imagine Twitter without users. It’s easy if you try. It would simply not exist.

    I certainly can’t easily take my Twitters with me. It’s a silo pure and simple. one in which they have demonstrated a willingness to shut down API features at will.

    Ah, but now you are conflating User-Driven with DP and perhaps VRM. It’s a different quality.

    BitTorrent is definitely user-driven That’s a good example of a standard protocol enabling users to create immediate value. And we don’t even have to write our own client to use it.

    …except that, for at least one person, they *did* have to write their own client; and in fact the abundance of clients suggests that many folk actually *did*.

  3. dropsafe : Federating the Stupidity of Search through VRM! (not)

    [...] [this posting continues a comment thread from Joe Andrieu’s blog] [...]

  4. ragegirrl

    Joe, you are quoting selectively and narrowing my definition of the concept to something not conveying its original meaning that was defined in more complex manner elsewhere in the same post:

    as well as here:

    and for completeness, also addressed this in comments on your other post:

  5. - Future of Search Engine Marketing Trust Broken

    [...] They don’t admit it in what they say but they do admit it in what they DO.  Search marketing is failing — it can’t help but fail given the increase in Web users and the content we’re all creating (’social media’).  Search engines are struggling to respond by “getting personal” (personalized search). [...]

Leave a Reply

You must be logged in to post a comment.

I Support The Standard Label!A simple way for websites to describe in clear, consistent terms what they do with the data we share.
Join the announcement list for up-to-date info.