Whether it’s user-generated content like YouTube, user-written and edited knowledgebases, like Wikipedia and Freebase, or user-centric Identity like OpenID and Information Cards, user-driven thinking is transforming our world. With VRM— Vendor Relationship Management–that revolution reaches the market, creating tools for individuals to get more value out of their relationships with Vendors. The goal is to create a user-driven market, where individuals engage with vendors on their own terms, creating mutually beneficial relationships that generate new value for everyone involved.
So what would it mean to apply user-driven thinking to Search? Traditional search is a mix of user-driven and vendor-centrism. While users can enter any query and be directed to content anywhere on the ‘net, we can’t share our search history with Search Providers of choice, nor do we have control over how our activities are tracked and utilized. There are few, if any, open standards for the searcher side of the experience and few options for moving beyond traditional query-response Search.
At the VRM Workshop 2008, we fleshed out some ideas, building on the thoughts introduced in my previous post, as well as ideas discussed at VRM 2008 in Munich and IIW2008a in Mountain View. What I love about the conversations at these unconferences is that they are so rich, literally creating value on a moment-to-moment basis. And these were no exception.
Here’s what has emerged so far regarding User-driven Search.
1. User Driven Search is bigger than query/response.
User Driven Search is more than what we type into the query box and the results we get back from Search Engines. It covers an entire set of activities that span the Internet, including searches entered at site-specific Search Providers like Expedia, the USPTO, and Circuit City, and all the web pages we visit in-between. It is inherently cross-silo—even non-silo—as it encompasses all of our online efforts around a given Search topic.
A recent Google/Comscore study found that the average Travel searcher takes 29 days from their first query until their first online purchase. These advanced Searches don’t take place all at one Search Provider nor do they usually happen all in one sitting. Users need tools that empower them to manage these advanced, multi-site, multi-session Searches.
2. Users should be able to activate and deactivate Search and tracking easily and at will
With User-driven Search facilitating advanced searches across the entire scope of our online activity, users need to be able to turn it on and off at will. Sometimes we want help and are willing to share to get it. Other times, privacy is preferred. We need to be able to turn off the surveillance and just do our thing. Unfortunately, traditional search and advertising networks don’t let us do that in any reasonable way.
There are ways to disable Doubleclick’s tracking and we can tell Google to stop personalizing our search results—if we also turn off our Search History. Yet most people have no idea this is possible and even more aren’t technically comfortable enough to mess with cookies or custom preferences. We shouldn’t have to jump through hoops to disable tracking, because if that’s the case, the vast majority of users will simply not do it, and even those who do will often opt-out completely, which means there really isn’t any choice at all. The decision shouldn’t be between using advanced search features or being treated like a digital transient. We should be able to get advanced features just when we want them and simply turn them off when we don’t. That choice needs to be transparently obvious and easy and available right in the Search interface.
3. Compartmentalization
When treating Searches that span more than single queries, users need to be able to separate them into their natural topical breakdown, in whatever way makes sense. Collecting our entire search history and/or clickstream into a single attention datastore literally destroys the context that makes the Searches relevant.
Users need a way to collect their Search-related activities into categories that make sense for them. We’d like to keep our summer vacation search activity together, yet separate from our financial planning Search. We’d like to collect our home buying search activity and store that in a different place than the queries and discoveries related to our child’s Search for information about George Washington. User-driven Search must deal with more than query/response and yet not so much that it encompasses our entire attention stream. It must capture the sweet spot of user-defined collections at a scale suitable to each Search individually, as determined by each searcher.
4. Visability and Editability
For users to drive Search, we need to be able to see and edit the all of the information used to provide results. Hidden or unauthorized data or tracking of our clickstream allow current Search Providers and advertising networks to analyze and guesstimate what we are looking for, but they don’t provide any way for us to contribute. Not only are they hiding in my virtual closet surveilling me—often without permission—they are missing a great opportunity to simply ask me what I want. By making all Search activity visible, Providers can say “Here’s the data we are using to try to help you.” By making that editable, they add “Can you help us improve it?” User interface challenges aside, there is no reason Search Providers shouldn’t ask for feedback and input. It is guaranteed to improve the quality of their view and ultimately their Search results.
Currently, Google, and its DoubleClick division, track your entire search history and just about anywhere you might go online, yet you have no idea what information they have on you, except for Google’s Search History—and you certainly can’t edit it. So when you track something down on a lark, or someone else uses your machine, irrelevant data gets bundled into your history, only to clog up the machinery that is actually trying to help you. Buy a book on knots for your young cousin and Amazon will be recommending Boy Scout titles for months. This is sometimes referred to as the “Tivo thinks I’m gay” problem. If users have neither visibility nor control over the data used for recommendations, they can’t correct these types of errors. We must have both visibility into the data driving advertising and search results, and we must be able to edit it as well.
5. Selectable disclosure on users’ terms
Having gone to the trouble to coordinate and maintain a collection of data for their Searches, users should be empowered to share that data with any service capable of responding intelligently. Search is a fundamental part of how we navigate the web; it makes no sense to restrict Search activity to any one provider. Just as your Search might take you to dozens of websites, it is also possible that it will bring you to dozens of Search Providers, from Google and Yahoo! to Amazon and eBay, even to microSearch Providers like Circuit City or Schwab. As users navigate across the web, their Search should go with them, seamlessly disclosed to authorized Search Providers as easily as possible.
Today, Google serves as a locked-in data silo for most people’s search history. There’s no way to send that history to Yahoo! Or MSN Live or Amazon or eBay to see what they might be able to do for you. As technologies for personalized search results improve, the value of that search history will continue to increase. We need to be able to send select parts of our search to providers of choice and we need to be able to do it trivially. As easily as we go from one website to another, we should be able to send our Search to a new Search Provider.
And yet, if we are to facilitate the easy transfer of this data, we also need to protect users’ rights, even as we expose more secrets to more people.
Schwab, for example, could greatly improve the ease of finding appropriate offerings if they could review the relevant parts of the current Search instead of relying on you entering just the right queries and properly navigating their site architecture. But it is unlikely that users are going to want to give Schwab any information unless there’s an understanding about just exactly how that information will be used (and the ability to select just what information is sent). We generally don’t want companies to start sending us junk mail or calling us with sales offers just because our Search shows that we are in the market for one of their products. But, if we could be assured they would use our Search just to provide better results and perhaps to improve their offerings, we are far more likely to share that part of our Search that could help them help us. We want explicit agreement for data rights access and we want it before we give them any data, and when we want to select what we send so they get just the parts that make sense, and not any personal information we don’t want to share. A User-driven Search solution must not only allow users to send select portions of their Search wherever they want, it must allow them to set the terms for exactly what recipients can do once they get it.
6. Impulse from the user as a specific statement of Search Intent
Recommendation systems presume that an analysis of your history is the best way to discover what you might want now. The NetFlix recommendation challenge and Amazon recommendations feature both use this approach. Not only does this place the user in a passive mode, it also has no facility for users to state what they actually want, right now. People have widely varying interests and easily switch between tasks even in the middle of a Search. Our past transactions may paint an interesting picture of who we are, but it rarely describes what we want in any given moment. What we really want from NetFlix isn’t the “perfect” movie for someone with my viewing history, we want the movie that’s perfect for the mood or situation we’re in right now.
Search systems, on the other hand, rely on a specific “objet de Search” as a trigger for directing efforts. The objet de Search is a keyword or other statement that explicitly represents the user’s intent in some way. At traditional search engines, the query serves this purpose, with the user essentially asking “what web pages have these words” in the hope that those words might be on the page that has what they are actually looking for. At structured Search Providers, like Expedia or Orbitz, the entries for departure, destination, date, and number of travelers in the combined form data comprise the objet de Search.
For User Driven Searches, we must move beyond the keywords and limited structured form fields to allow a more complex, more expressive statement of intent. This statement should include the entire range of Search activity for your given Search, including queries, Search Providers, clickthroughs, captures, and annotations. In short, it should bundle up the entire Search and present it to the Search Provider as an explicit statement of intent. This presentation must be independent of any data silo, unlimited by the offerings of any particular vendor. It should be a proactive statement of “Here’s what I’m looking for: here’s what I’ve found so far and where I’ve been. Got anything that might help?”
Most importantly, Search operates in the foreground, with an explicit impulse from the user. User-driven Search isn’t about background profiling and analysis to try to guess user intent. It requires an explicit means for users to state their intent in ways Search Providers can understand. Instead of predatorial “targeting” of users with particular demographic, psychographic, or behavioral profiles, User-driven Search operates exclusively on that objet de search, as the entire representation of user intent. No more guessing. No more secretive or unauthorized tracking. No more stereotypical clustering based on industrial-era models of consumer behavior. Instead, User-driven Search Providers respond directly to clear, unambiguous representations of confirmed user intent.
Towards an Open Standard
This is the kind of solution we are working on at SwitchBook. At the VRM Workshop 2008, I was excited to learn more about MatchMine from J Trent Adams; they are moving in a similar direction for media-based recommendations. There is currently no service we know of that fully delivers on the promise of User Driven Search, but I’m looking forward to working with Trent and others to develop the open standards and protocols to make it possible.
If you are interested in joining the conversation, send me an email. We’ll be setting up a listserv to talk on a more regular basis. All are welcome.
[Update 5/3/2009: “user-driven Search” to “User Driven Search”]
I posted a comment on https://blog.joeandrieu.com/2008/07/12/towards-user-driven-search/ which precedes this; it sounds to me that what you are proposing above is “spy in the cab” meets “clippy the paperclip”, with promises that it won’t really leak stuff to marketeers unless you really want it to.
“Honest. We swear.”
Again in this scenario, exactly as before, the user is in the position of supplicant, or at best is having their little search-turds recycled into composty goodness which someone else profits from and/or sells on their behalf.
How is this “User Driven” as-per the comment in the previous posting?
If you think of Firefox’s browser history as spying on your activity, you’re right. And if you think of advanced features on the client side as “Clippy the paperclip”, you’re right again.
Currently, the entire web is watching where you surf and you have virtually no control and no way to leverage it for your own value. Giving users control over that information and providing an explicit means for direct improvements in their search results is user-driven at the very core. The whole point is to liberate users from the isolated data silos at Google and Doubleclick and NebuAd, giving them a tool that lets them bundle their behavioral and explicit Search data into a representation that they can edit and release on their own terms.
Having “little search-turds recycled into composty goodness” is precisely what is happening out there, on the Internet, currently, with essentially no input from the user. Your phrasing is perhaps belittling, but you’re missing the point. The way that we search, what we look for, what we find, what we keep, all of these comprise a better representation of what we are actually looking for. The paltry keyword started losing its glamor when Google started personalizing search results based on your Search History. Given that such tracking actually has the potential to improve our search experience, isn’t it time to give users control over it?
As you mention in the comment on the prior post, user-driven suggests that the value is, at least partially, provided directly by the user’s participation in the system, such as with BitTorrent. It is precisely the editability, compartmentalization, and selective disclosure that makes User-driven Search so valuable. Without the user to vet and refine the Search data, all you really have is an endless jumble of attention data with no direction and no user feedback. Which is precisely what DoubleClick (now Google) and NebuAd work with.
Of course, if you’d prefer to believe that keywords are good enough, then feel free to opt-out of next generation search, which I realize you may already do. In contrast, most people, I believe, have Google’s Search History enabled and I expect they will continue to benefit from innovations in “composty goodness”. They just won’t have it in any user-driven way until we can extract that Search History in a meaningful way and re-use it on their own terms.
Finally, “won’t really leak stuff” can be formally bound in access policy and data rights agreements. Of course, today, Google is free to hand over your search history or YouTube viewing history to whomever they please. There are neither legal nor contractual restraints to protect user’s privacy. However, with a fully implemented User-driven Search solution, explicit, contractually binding agreement to data rights policies can and will be required before releasing search data to potential vendors. I think you’d agree that technology-based DRM is a waste of time. But legally enforceable contracts with Search providers is another thing altogether.
At the end of the day, if you want someone to help you with your search, you have to give them something to work with. The point of User-driven Search is to make that transaction seamless, painfree, entirely under user control, and bound to non-repudiatable data rights agreements.
If you still think that misses the opportunity for User-driven Search, what would you suggest to make Search truly user-driven?
Strawman alert! Strawman alert! 🙂 Actually I see a slavering corporate beast looking to make money off-of all my search history, to be desireous of spying upon me: “All that untapped intention- and attention-marketing information, it must be mine bwahahahahahaha!” So you see, what I am talking about is the mindset of those who wish to build a new business atop other peoples’ information.
“All these people, producing this good stuff and not
letting us use it to sell them stuffgetting benefit from it – how tragic and uninformed they are, we must help them.”In what way would this be Liberation, precisely? For instance, AdBlockPlus is liberation in my book – unshackling me from crap I am otherwise forced to endure. Google goes to great lengths to ensure that the crap is me-centric, to be sure, through contextualisation and whatnot. But it is still crap.
To whose actual benefit, and for what reason would I want to do this, given I am of the AdBlockPlus mindset outlined above?
Doubtless; you make a good case for (eg:) Google to refine its search algorithms to profile user-searches and start trying to second-guess the user; and I agree with an implicit proposition that the user should be able to download/access their search history. But given that I’ve just stated the concept in terms of “What Could Google Do?” I have to ask: how is any of this user-driven or VRM-ish?
Yes, Bittorrent would not exist without the content which users provide; theoretically the Google search function *could* exist without users making search queries, although they might not continue to exist for very long…
… for whose benefit? …
Yes. So you are proposing that users are masochists who will skim their own search histories for clues about their deepest darkest desires, and use that deep self-knowledge to request advertising for [whatever] – britney spears albums, maybe?
Actually I have it switched-off for privacy reasons, plus I wonder if “most people” will have that, since the proposition implies that “most people” (say, more than 50% of internet users) have got a Google account, which I find hard to believe.
I love the way you say “WE CAN EXTRACT” and “THEIR OWN TERMS” – who is your notion of “WE” in this context, Joe?
See above comment regards “deep self knowledge”.
Mmmm…
Yeah. I am just thinking that the added value you are here proposing atop search, does not address customer wants. The tasty ice-cream you are suggesting is also an effective floorwax.
I wouldn’t, in the same way I wouldn’t try to make a kosher bacon-double-cheeseburger. The result would either be not-kosher, or not-a-bacon-double-cheeseburger.
[Continuing the thread from the other blog comment]
So when you say “User-driven systems create value inherently” you mean that user-driven systems have the inherent property of creating-value; which given the definition that the user’s input creates the value of a user-driven system, is something of a tautology.
I would be very careful to avoid misrepresenting Adriana’s position as a quest for customisation; it’s vastly different and far larger than that.
Oh yeah. I am not sure that that is good news for you, alas :-/
Mmmm…
Imagine Twitter without users. It’s easy if you try. It would simply not exist.
Ah, but now you are conflating User-Driven with DP and perhaps VRM. It’s a different quality.
…except that, for at least one person, they *did* have to write their own client; and in fact the abundance of clients suggests that many folk actually *did*.
I’m not suggesting that it must be anyone’s, I’m suggesting that it is invaluable and therefore users would benefit from tools to manage it. You’re disposition suggests that building a business is evil and therefore any quid-pro-quo for helping a user manage their searches is also inherently evil. Which is pure rubbish.
If you are satisfied with your ability to search the web in its current form, User-driven Search offers no value for you. For those of us who find the trillions of results on Google to be a bit unwieldy, there is plenty of room for improvement.
AdBlockPlus is a violation of your social contract and terms of service. You may not like that your website of choice has chosen a business model that offends you, but it doesn’t give you the right to access their content without addressing the quid-pro-quo they clearly expect you to participate in. I expect you don’t find that a compelling argument, but that’s ok. Hacking your way around capitalism isn’t innovative, it’s just passe.
In contrast, giving you a way to take control of your search related data definitely liberates you from the lock-in at Google.
Regardless of your predilection for hacking your way past the TOS, the reason you would want to bundle your search data and send it to a vendor is for better search results. Your question is essentially the same as asking why you would want to tell Google what you are looking for online?! My Goodness! Then someone would know what you’re doing!! And the answer is the same: because if you don’t tell someone about your search, nobody can help you find stuff.
Because it isn’t about Google. It’s about giving individuals the power that is, in a limited fashion, currently locked up in Google’s silo. Why should only Google have access to your Search history? Why shouldn’t you be able to edit it and move it around as you Search? Why shouldn’t you be able to send it to eBay or Amazon or Schwab or the USPTO and have them use it as an alternative entry point for search results and navigation? That’s why it is user-driven. Because it puts the power to manage your Search in your hands.
Actually, the types of statistical analysis and recommendations that are likely to be made from User-driven Search data would not at all be possible if there were no users.
If you don’t appreciate the benefit, you don’t need to be a user. Contrast that with the ad networks’ tracking of your online behavior or Phorm or NebuAd’s deep packet sniffing to give ISPs behavioral tracking capability. User-driven Search only works if the search results are better. How hard is that to follow? Better search results is a pretty straightforward benefit.
No, what I propose is that Search results can be improved if users have direct control over and contribute to the information used to generate results. No self-knowledge is required. Just activate a tool that helps organize your searches, and choose which Search Providers you trust. The suggestion that the only valuable output of a search is advertising is just bait. Keep it for the cock fights.
For the pedant in you, let me clarify the proposition: “most people who use Google accounts probably have their Search History activated” Further, I would suggest that a large enough number of people have cookie-activated accounts at the largest Search providers that it just may be “most” people using those search services are getting personalized search results today.
That’s definitely a typo. Of course if it is “We” then it is “our”. We is “we the users” of search services.
Again, if you are happy with the search results you get today, then don’t use User-driven Search. But I have a feeling a lot of folks will be happy to see improvements.
In fact, take a look at this research done by Microsoft: https://blog.joeandrieu.com/2008/03/18/microsofts-searchbar-an-integrated-tool-for-complex-searches.
There is plenty of room to improve Search and it is going to start by combining and manipulating exactly the sort of data I’ve been talking about.
Again, if you find Search too offensive for your sense of privacy, don’t Search online. However, I would bet that you have actually told Google what you are looking for, sacrificing your dearly loved privacy for a few mere scraps of search results.
No, I didn’t define user-driven systems that way. You appear to be. I think that’s a nearly useless definition. The user-as-chef metaphor had more legs than that. However, it is a reasonable observation: If you put users in charge, they will create value. User-driven systems are about putting users in charge. It doesn’t mean they have to create the value, just that they can.
Your care is noted. I would request similar care in understanding my words. I said “I don’t believe ‘customization’ is the end-all-be-all of user-driven, where it is at the core of Adriana’s usage” and I stand by that.
Quoting Adriana:
The metaphor clearly points to the user customizing the salad as they make it, determining the proportions, skipping or adding ingredients. This is the core argument she makes to explain how “users add value” to the system. It’s clear, accurate, but limiting.
Furthermore, I’d argue that simply “adding value to the process”–the core test of a user-driven system in Adriana’s post–is insufficient to deem anything user-driven. It is easy to let users add value. AOL let users post in forums and send email to each other and chat. Did that make AOL user-driven? No. The ability of users to control the content and functionality on our own websites is what makes the World Wide Web user-driven. And precisely why AOL never was.
That doesn’t make it user-driven. It makes it user-dependent. Until a system hits the “A” in NEA systems, it can’t be truly user driven. That “A”, you may recall, stands for “Anyone can improve it.” Twitter is just another pretty silo.
DP is a requirement for user-driven systems. If you can’t get your data out, it violates control and authority and therefore is not user-driven.
Clearly we disagree on what user-driven means. That’s fine. It’s even healthy. We’ll let the market sift through the details. Meanwhile, I’ll note the difference and continue to clarify what I mean by user-driven, because of course, the user-driven media I work in gives me the power to do exactly that.
Pingback: dropsafe : Federating the Stupidity of Search through VRM! (not)
Joe, you are quoting selectively and narrowing my definition of the concept to something not conveying its original meaning that was defined in more complex manner elsewhere in the same post:
http://www.mediainfluencer.net/2008/04/two-tales-of-user-centricities/
as well as here:
http://www.mediainfluencer.net/2008/05/from-misapprehensions-to-alternatives/
and for completeness, also addressed this in comments on your other post:
https://blog.joeandrieu.com/2008/07/12/towards-user-driven-search/#comment-1512
Pingback: JeffMolander.com - Future of Search Engine Marketing Trust Broken