Whether it’s user-generated content like YouTube, user-written and edited knowledgebases, like Wikipedia and Freebase, or user-centric Identity like OpenID and Information Cards, user-driven thinking is transforming our world. With VRM– Vendor Relationship Management–that revolution reaches the market, creating tools for individuals to get more value out of their relationships with Vendors. The goal is to create a user-driven market, where individuals engage with vendors on their own terms, creating mutually beneficial relationships that generate new value for everyone involved.

So what would it mean to apply user-driven thinking to Search? Traditional search is a mix of user-driven and vendor-centrism. While users can enter any query and be directed to content anywhere on the ‘net, we can’t share our search history with Search Providers of choice, nor do we have control over how our activities are tracked and utilized. There are few, if any, open standards for the searcher side of the experience and few options for moving beyond traditional query-response Search.

At the VRM Workshop 2008, we fleshed out some ideas, building on the thoughts introduced in my previous post, as well as ideas discussed at VRM 2008 in Munich and IIW2008a in Mountain View. What I love about the conversations at these unconferences is that they are so rich, literally creating value on a moment-to-moment basis. And these were no exception.

Here’s what has emerged so far regarding User-driven Search.

1. User-driven Search is bigger than query/response.

User-driven Search is more than what we type into the query box and the results we get back from Search Engines. It covers an entire set of activities that span the Internet, including searches entered at site-specific Search Providers like Expedia, the USPTO, and Circuit City, and all the web pages we visit in-between. It is inherently cross-silo—even non-silo—as it encompasses all of our online efforts around a given Search topic.

A recent Google/Comscore study found that the average Travel searcher takes 29 days from their first Search until their first online purchase. These advanced Searches don’t take place all at one Search Provider nor do they usually happen all in one sitting. Users need tools that empower them to manage these advanced, multi-site, multi-session Searches.

2. Users should be able to activate and deactivate Search and tracking easily and at will

With User-driven Search facilitating advanced searches across the entire scope of our online activity, users need to be able to turn it on and off at will. Sometimes we want help and are willing to share to get it. Other times, privacy is preferred. We need to be able to turn off the surveillance and just do our thing. Unfortunately, traditional search and advertising networks don’t let us do that in any reasonable way.

There are ways to disable Doubleclick’s tracking and we can tell Google to stop personalizing our search results—if we also turn off our Search History. Yet most people have no idea this is possible and even more aren’t technically comfortable enough to mess with cookies or custom preferences. We shouldn’t have to jump through hoops to disable tracking, because if that’s the case, the vast majority of users will simply not do it, and even those who do will often opt-out completely, which means there really isn’t any choice at all. The decision shouldn’t be between using advanced search features or being treated like a digital transient. We should be able to get advanced features just when we want them and simply turn them off when we don’t. That choice needs to be transparently obvious and easy and available right in the Search interface.

3. Compartmentalization

When treating Searches that span more than single queries, users need to be able to separate them into their natural topical breakdown, in whatever way makes sense. Collecting our entire search history and/or clickstream into a single attention datastore literally destroys the context that makes the Searches relevant.

Users need a way to collect their Search-related activities into categories that make sense for them. We’d like to keep our summer vacation search activity together, yet separate from our financial planning Search. We’d like to collect our home buying search activity and store that in a different place than the queries and discoveries related to our child’s Search for information about George Washington. User-driven Search must deal with more than query/response and yet not so much that it encompasses our entire attention stream. It must capture the sweet spot of user-defined collections at a scale suitable to each Search individually, as determined by each searcher.

4. Visability and Editability

For users to drive Search, we need to be able to see and edit the all of the information used to provide results. Hidden or unauthorized data or tracking of our clickstream allow current Search Providers and advertising networks to analyze and guesstimate what we are looking for, but they don’t provide any way for us to contribute. Not only are they hiding in my virtual closet surveilling me—often without permission—they are missing a great opportunity to simply ask me what I want. By making all Search activity visible, Providers can say “Here’s the data we are using to try to help you.” By making that editable, they add “Can you help us improve it?” User interface challenges aside, there is no reason Search Providers shouldn’t ask for feedback and input. It is guaranteed to improve the quality of their view and ultimately their Search results.

Currently, Google, and its DoubleClick division, track your entire search history and just about anywhere you might go online, yet you have no idea what information they have on you, except for Google’s Search History—and you certainly can’t edit it. So when you track something down on a lark, or someone else uses your machine, irrelevant data gets bundled into your history, only to clog up the machinery that is actually trying to help you. Buy a book on knots for your young cousin and Amazon will be recommending Boy Scout titles for months. This is sometimes referred to as the “Tivo thinks I’m gay” problem. If users have neither visibility nor control over the data used for recommendations, they can’t correct these types of errors. We must have both visibility into the data driving advertising and search results, and we must be able to edit it as well.

5. Selectable disclosure on users’ terms

Having gone to the trouble to coordinate and maintain a collection of data for their Searches, users should be empowered to share that data with any service capable of responding intelligently. Search is a fundamental part of how we navigate the web; it makes no sense to restrict Search activity to any one provider. Just as your Search might take you to dozens of websites, it is also possible that it will bring you to dozens of Search Providers, from Google and Yahoo! to Amazon and eBay, even to microSearch Providers like Circuit City or Schwab. As users navigate across the web, their Search should go with them, seamlessly disclosed to authorized Search Providers as easily as possible.

Today, Google serves as a locked-in data silo for most people’s search history. There’s no way to send that history to Yahoo! Or MSN Live or Amazon or eBay to see what they might be able to do for you. As technologies for personalized search results improve, the value of that search history will continue to increase. We need to be able to send select parts of our search to providers of choice and we need to be able to do it trivially. As easily as we go from one website to another, we should be able to send our Search to a new Search Provider.

And yet, if we are to facilitate the easy transfer of this data, we also need to protect users’ rights, even as we expose more secrets to more people.

Schwab, for example, could greatly improve the ease of finding appropriate offerings if they could review the relevant parts of the current Search instead of relying on you entering just the right queries and properly navigating their site architecture. But it is unlikely that users are going to want to give Schwab any information unless there’s an understanding about just exactly how that information will be used (and the ability to select just what information is sent). We generally don’t want companies to start sending us junk mail or calling us with sales offers just because our Search shows that we are in the market for one of their products. But, if we could be assured they would use our Search just to provide better results and perhaps to improve their offerings, we are far more likely to share that part of our Search that could help them help us. We want explicit agreement for data rights access and we want it before we give them any data, and when we want to select what we send so they get just the parts that make sense, and not any personal information we don’t want to share. A User-driven Search solution must not only allow users to send select portions of their Search wherever they want, it must allow them to set the terms for exactly what recipients can do once they get it.

6. Impulse from the user as a specific statement of Search Intent

Recommendation systems presume that an analysis of your history is the best way to discover what you might want now. The NetFlix recommendation challenge and Amazon recommendations feature both use this approach. Not only does this place the user in a passive mode, it also has no facility for users to state what they actually want, right now. People have widely varying interests and easily switch between tasks even in the middle of a Search. Our past transactions may paint an interesting picture of who we are, but it rarely describes what we want in any given moment. What we really want from NetFlix isn’t the “perfect” movie for someone with my viewing history, we want the movie that’s perfect for the mood or situation we’re in right now.

Search systems, on the other hand, rely on a specific “objet de Search” as a trigger for directing efforts. The object de Search is a keyword or other statement that explicitly represents the user’s intent in some way. At traditional search engines, the query serves this purpose, with the user essentially asking “what web pages have these words” in the hope that those words might be on the page that has what they are actually looking for. At structured Search Providers, like Expedia or Orbitz, the entries for departure, destination, date, and number of travelers in the combined form data comprise the objet de Search.

For User-driven Searches, we must move beyond the keywords and limited structured form fields to allow a more complex, more expressive statement of intent. This statement should include the entire range of Search activity for your given Search, including queries, Search Providers, clickthroughs, captures, and annotations. In short, it should bundle up the entire Search and present it to the Search Provider as an explicit statement of intent. This presentation must be independent of any data silo, unlimited by the offerings of any particular vendor. It should be a proactive statement of “Here’s what I’m looking for: here’s what I’ve found so far and where I’ve been. Got anything that might help?”

Most importantly, Search operates in the foreground, with an explicit impulse from the user. User-driven Search isn’t about background profiling and analysis to try to guess user intent. It requires an explicit means for users to state their intent in ways Search Providers can understand. Instead of predatorial “targeting” of users with particular demographic, psychographic, or behavioral profiles, User-driven Search operates exclusively on that objet de search, as the entire representation of user intent. No more guessing. No more secretive or unauthorized tracking. No more stereotypical clustering based on industrial-era models of consumer behavior. Instead, User-driven Search Providers respond directly to clear, unambiguous representations of confirmed user intent.

Towards an Open Standard

This is the kind of solution we are working on at SwitchBook. At the VRM Workshop 2008, I was excited to learn more about MatchMine from J Trent Adams; they are moving in a similar direction for media-based recommendations. There is currently no service we know of that fully delivers on the promise of User-driven Search, but I’m looking forward to working with Trent and others to develop the open standards and protocols to make it possible.

If you are interested in joining the conversation, send me an email. We’ll be setting up a listserv to talk on a more regular basis. All are welcome.

“Social Graph” is not just a singular noun.

“The Social Graph” is a popular misnomer that has plagued the social networking portability conversation ever since Brad Fitzpatrick catalyzed the blogosphere with a vision about the Global Social Graph.

But in fact, “The Social Graph” has little real value outside of computer science elegance. Nobody but Big Brother, the TSA, the CIA, and [insert surveillance agency of your jurisdiction here], actually want that single, monolithic view of all the relationships in the world. That’s The Social Graph.

In contrast, my social graph is hugely valuable to me. Your social graph matters to you. And it might be interesting to discover where our graph (plural) overlap. But neither of us actually care about The Social Graph.

At the VRM Workshop 2008, here at Harvard’s Berkman Center for Internet and Society, it came out that “social graph” is actually plural.

Like fish.

The Social Graph is a misleading distraction, a handy buzzword we can all slip into our cocktail conversations. But the real value is in the personal, independent social graph we all have. Plural.

If you think about it, that’s the only way you can really make sense of it in our user-centric, user-driven world.

It is time to give users more control over Search.

At VRM2008 in Munich and at IIW in Mountain View, I started a conversation about User-Driven Search, the premise: what would it mean for users to truly drive their searches?

User-driven is a new term that came out of the VRM community riffing on the meaning of user-centric development and user-centric identity. User-centric is a nice term, but it could be construed as limiting. For example, user –centric definitely implies that that the user is at the center of attention and the focus of the architecture, but it doesn’t necessarily mean the user is in charge of the experience. That’s a key distinction.

Not just tuna salad

Adriana explains this difference between user-centric and user-driven as metaphorically the difference between buying ready-made tuna salad or picking and choosing your own ingredients and making the tuna salad yourself. When I first talked with Doc about user-driven instead of user-centric, Jim Carrey’s The Truman Show immediately sprang to mind: from birth, Truman is the protagonist in a huge reality show revolving around him… only he doesn’t know it. The climax of the show is Truman discovering the rest of the world and confronting his father/producer. Clearly the Truman Show is Truman-centric… but it is most definitely not Truman-driven.

It’s about impetus and authority

For me, user-driven means that the user provides the impetus and is the controlling authority throughout the transaction. Sure, sometimes there is negotiation or collaboration with others… the user isn’t omnipotent, after all. However, the user is in charge of creating his or her own experience. This fits with user-constructed or customized solutions, like the tuna salad recipe. However, it has implications far beyond the limits user-created or user-customizable architectures.

Is the user initiating the experience? Is the user’s moral authority the primary control throughout the system? Is the system transparent to users, enabling them to make their own informed decisions about what will be presented to them and how it is presented? Is it the user who is shaping the input, intermediary results, and final outcome? If so, then it is user-driven. If not, it isn’t.

When it comes to the tuna salad metaphor, this is the equivalent of the tuna salad being made when I ask for it and on my terms. Not before. And although I could choose to make the salad myself–that is definitely user-driven–it could also be made by someone else to my specifications… extra mayo and black pepper, no onions, thank you.

Search as user-driven

Google’s keyword query-response approach to Search is, of course, user-driven to some extent. Nothing happens until the user enters a query, users are free to enter any query, and the system responds with results tailored for exactly what the user queried. The user does shape the experience to a limited degree. And yet, it still provides only a slim façade of user control. There is no way to modulate the algorithm, no way to let Google know which results are good or bad, and no way to refine the search other than keyword guessing games. And, perhaps most importantly, there is no way to manage the search beyond the immediate query. For that, the user is dependent on other techniques: bookmarking, cut & paste, opening multiple windows or tabs, even printing to paper or PDF to keep track of good finds. Evolution in Search History management is starting in the right direction, but the ideas here have been rather uninspiring so far.

User-driven systems create value inherently

The limits on the user-driven aspects of Google are particularly ironic given that it is precisely the element of user control that creates Google’s greatest asset: focused attention. Google’s money making asset is the collection of user-specified queries, queries that explicitly state words related to the user’s interest and implicitly denote user intent. It is precisely because the individual enters a specific keyword that Google is able to sell targeted ads… at great profit and benefit to advertisers and searchers alike.

The query entered in the Search box gives Google a implicit statement of intent. That intention is the gold Google resells to advertisers. If Google didn’t let users drive that intention, if they looked more like a content site or “Internet portal”, they’d have a lot less intention to monetize.

If we can extend that control, if we can make search even more user-driven, if we can enable richer, more explicit, more user-driven expressions of Search intent, I believe we can create even more value for everyone involved: search companies, advertisers, searchers, even non-paying websites showing up in “organic” results.

What does it mean to have User-Driven Search?

At SwitchBook, we’ve been doing a lot of thinking about what User-Driven Search might mean. I like starting the conversation with a simple example: what would it mean if I could take my search history from one search provider to another? This “dataportability” example is just an initial notion of how Search might become more user driven.

So, what do you think of when you hear (or read) “User-Driven Search”?

I’ll be leading a session on this topic at the VRM Workshop next week. I hope you can join us.

This material is based, in part, upon work supported by the National Science Foundation under Grant No. 0740861. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

If you’re going to bet your company on a platform, pick the open one.

That was my advice last month at the Caltech/MIT Enterprise Forum on Platforms. It turned into a lively debate (you can check out the audio for the June 8 2008 event), almost inevitably pitting me against Peter Coffee of Salesforce.com, with Marc Canter rallying the forces of small business and openness in his inimitable style: caustic, irreverent, and sourcing no small amount of passion.

One small side effect of the debate was that, at times, it slipped unavoidably into a referendum on Salesforce.com rather than an insightful discussion of the merits of open verses closed systems and how companies can reasonably navigate their choices. Pulling back from that focus on Salesforce left more than a few unanswered issues. Peter Coffee contacted me after the event and asked to keep the conversation going, which sounds like a great idea.

So, here’s a reprise of my presentation on June 14.

Open Platforms and Standards

Level 4 Platforms FTW

This article is about open platforms and what an entrepreneur should think about when choosing a platform for their business.

Two Questions

When choosing a platform, you’ll want to consider two major questions:

• Will it do what you need?
• Will it last long enough for you to do it?

The first is a matter of features and functionality, and must be evaluated on a case-by-case business for each business. Since I don’t know your business, I won’t spend much more time on this question.

The second question is about longevity. Will the platform itself be available as long as your business needs to use it? Will it be stable and robust enough on a moment-to-moment basis? Is there a self-sustaining community of developers and integrators to help you continue to help your company adjust as business needs change over time? In other words, will the platform continue to be able to provide value for your business in the long term?

What it means to be Open

In talking about open platforms, I want to be clear what I mean by open. An open platform, for the sake of this article (and as far as I’m concerned, for the purposes of understanding the revolution of open systems), is one that adheres to the principles of N.E.A:

• No one owns it
• Everyone can use it
• Anyone can improve it

If a single entity or group owns the platform, it isn’t open. If there are barriers preventing users from accessing or developing on the platform, it isn’t open. If you can’t, with reasonable effort, improve the platform itself, it isn’t open.

Every platform is open to some degree. After all, that is the point: platforms open proprietary systems so that third-party developers can innovate and create value beyond the scope, resources, or expertise of the platform creators. But truly open platforms allow anyone to improve the actual platform, not just develop within its constraints. For example, SSL, the secure sockets layer used to secure web access, was developed in the early days of the world wide web when a small group of innovators figured out a way to automatically encrypt information transmitted between a web browser and a website. They didn’t need to negotiate a contract or get permission, they simply implemented their solution and made it available to everyone, changing the platform of the World Wide Web itself.

When you bet on a closed platform, you are betting that the future of your company fits into the future as designed by the platform owner. When betting on an open platform, you bet that someone, somewhere, will be able to evolve the platform to meet your continuing needs.

NEA is a concept from World of Ends. What the Internet Is and How to Stop Mistaking It for Something Else by Doc Searls and David Weinberger, written October 2003.

Marc Andreessen’s Three Platforms

In September 2007, Marc Andreessen, one of the enablers of the World Wide Web, wrote an article titled the The Three Platforms You Meet On the Internet. I’m going to revisit those platforms visually, and introduce a fourth kind of platform that Marc missed. Because the text may be hard to read, here’s a legend of the elements present in all four graphics.

User Interface

Developer Application

Platform Enabler

Platform Service Provider

With that, let’s look at Marc’s three levels of platforms.

Level 1 Access API

Level 1 Platforms allow third party developers to access the platform via a well-defined and documented API, typically using HTTP or SOAP. This allows developers to create their own applications, running anywhere the developer chooses, with access to the data and services running on the platform. Twitter, eBay, PayPal, Flickr, and Del.icio.us all support this type of interaction.

Level 2 Plug-in API

Level 2 Platforms allow third party developers to create their application anywhere, with specific, limited ways to affect the user interface running on the platform. The key shift here is that the platform provider controls the overall user experience, but allows the developer a way to create value within their interface. Photoshop, FaceBook, Firefox, Internet Explorer, and Outlook are all applications which act as Level 2 platforms.

Level 3 Run-time Environment

Level 3 Platforms let develpers create applications that actually run on the platform. That means the code written by third-parties executes directly in the platform context. The overall user experience is still controlled by the platform, but compared to Level 2 platforms, Level 3 applications typically customize the user interface more extensively. Salesforce, Ning, OpenSocial, Windows, Java, EC2, and Google Apps are all Level 3 platforms.

Common Assumption

All three levels share a common assumption. You might notice it if you consider the three levels together:

All three levels assume a single platform service provider: Just one pink triangle. Of course, we can assume the number of developers is unlimited… that’s the point of a platform. However, all of the systems described above are built to enable access to one single platform, a platform owned and controlled by its creator.

When Marc wrote his article, I emailed him and asked “What about the World Wide Web?” It doesn’t fit any of his models, yet is clearly one of the worlds most widespread, most successful, and most relevant platforms in history. On the World Wide Web, there is no central server, no platform provider. It is a different animal altogether. I call it a Level 4 Platform.

Level 4 Open Protocols

Level 4 platforms allow developers to build applications anywhere–on a website, on your desktop, even on your cell phone–and those applications can talk to any number of platform providers without restriction, using standard open protocols. Many of us have heard of the most successful protocols: SMTP, POP, HTTP, HTML, TCP/IP, RSS, but most users know these by the applications they enable: email, the World Wide Web, the Internet, blogs.

Level 4 platforms are truly open, even as each piece of technology is provided by separate for-profit companies. It wasn’t until the World Wide Web opened online services to literally any company with a modicum of technical capability that the world began to enjoy the power of ubiquitous interactive services. As long as CompuServe and AOL and AT&T controlled tightly integrated one-stop-shop subscription services, the number of third-party developers was inherently limited. To launch a new service on AOL, you had to negotiate your own contract, all the while knowing that AOL could already be working on something similar and would never allow you to compete directly.

The web changed all that, allowing literally thousands upon thousands of entrepreneurs to explore their own (perhaps crazy) ideas of how to create value for people. No longer limited or burdened by a central platform provider, the pace of innovation exploded. Certainly, most of those ideas were crazy, but we couldn’t have known which were which without trying them first. The open nature of the web as a platform allowed exactly that.

The whole point of a platform is to encourage third party developers, so that everyone gets more value–value which is inherently beyond the purview of its creators. As long as a platform is closed to some degree, it is limiting the possibility for innovation. Limiting the innovation limits value to users. And limiting value to users limits the success of the platform.

Choose Open When Possible

The only guarantee for longevity is if you have options.

• Multiple Service Providers allow you to switch if you need to. You never know if your provider will continue to meet your needs. They may even start to compete with you. The freedom to move to another standards-compliant provider gives you control. Just like you have over your web hosting company and email provider.
• Source Code allows you, or developers working on your behalf, to improve, fix, simply maintain the platform your business depends on. If an open platform breaks, source code access keeps you from depending on the platform provider for a fix. Just like you can with Apache, Linux, Joomla, or Drupal.
• Intellectual Property Use Rights assure users and third-party developers that you are free from fear about changing licensing terms and future lawsuits.

You can’t always choose an open platform. Sometimes it is worth it to develop on a proprietary platform because it helps you get to market faster, with more features. Depending on your needs, you might be better off building on a closed platform. But, if you can, I recommend choosing an open platform whenever possible.

Note on VRM

In case folks are wondering why I am talking about Level 4 Platforms, it is because that is precisely what we are working to build over at Project VRM, where I am the Chair of the Standards Committee.

VRM is the conceptual reciprocal of CRM, Customer Relationship Management. Instead of large-scale enterprise software that helps big companies extract more profit from every customer, VRM is about tools for individuals to create more value in their relationships with vendors.

• Our mission is to enable both buyers and sellers to build mutually beneficial relationships, where everyone benefits from the zero-distance network that is the Internet.
• Our approach is open standards and open source development.
• Our goal is a level 4 platform of the kind described above for user-driven online commerce.

If you can make it, you are welcome to join us at the 2008 VRM Workshop next week at Havard’s Berkman Center for Internet and Society.

Bonus Link: Michael Cote on Platforms As A Service and Lock-in with Force.com

Pignerol Antoine recently asked some questions about VRM and I thought I’d answer them publicly.

Is VRM really different from social CRM ?

Yes, although exactly how depends on how you define social CRM. Based on my understanding, I would suggest that VRM is first and foremost about providing value for the user with any vendor, as opposed to using social networking tools with a particular vendor. VRM is vendor agnostic and silo-adverse. The goal is to catalyze the development of tools for individuals through protocols and standards that let them work with any vendor seamlessly, without loss of functionality or services.

Does VRM work with a CRM ?

Sure. A CRM is a company-centric system. Every company should pay attention to its customers and CRM is currently the best-of-class thinking on the enterprise-side for how to do that. Different VRM services act on behalf of the individual, yet still require connecting to enterprise systems. For things to be seamless, VRM services should marry into CRM services for fulfillment.

Can VRM be implemented in all kinds of business?

Yes. Any business can support VRM services and be compliant with general VRM principles. Ultimately, it will be as easy for a small company to be VRM compliant as it is for a small company to run a blog or a wiki today. That takes some level of technical sophistication, but it is within grasp of any company that wants to invest a small amount of effort using freely available open source tools. Eventually, VRM will be available in the same way.

What’s needed for VRM to work ?

We need to work through electronic marketplace issues from customers’ perspectives, with attention to the full power of relationships, finding consistent ways to create new value through the network. For the Standards Committee, that means a public conversation starting with users and requirements. Once that is vetted in an open source manner, we can explore particular implementations. We believe that with a well defined, high quality requirements specification, service providers will emerge to deliver those services.

As customers are looking for lower prices, don’t you think that Personal RFPs are gonna cost more for customers (because they are personalized offers) ?

Two things here. First, I don’t think customers are just looking for lower prices. They are looking for better value.

http://blog.joeandrieu.com/2008/03/07/pricing-markets-and-demand-vrm-style/

One of my favorite examples of this is Shopatron’s business where they sell everything at 100% manufacturer suggested retail price, no discounts, no rebates:

http://blog.joeandrieu.com/2007/01/19/shopatron-redefines-vendor-relationships/

Second, the personal RFP is designed to eliminate transaction costs in the marketplace. Currently, product and vendor discovery is slow, expensive, and uncertain. That means buyers waste time and vendors waste advertising and lead generation dollars seeking the right match between needs and solutions. Any time transaction costs are reduced, you have an opportunity for better prices.

At the same time, Vendors will be discovering ways to provide more value to customers and the net result could easily be that customers will end up paying more for enhanced services or products. Ideally, this will mean that commodity products continue to drop in price while value-added customizations are welcomed by buyers and voluntarily paid for at a premium over the commodities.

What do you expect from VRM?

I expect it will take longer and be more work than any of us would prefer. However, I think that the concepts behind VRM, and hopefully our work developing standards and catalyzing working solutions, will enable a fundamental shift in the marketplace. As Doc Searls has said more than once, the industrial revolution is over: industry won. There is an incredibly powerful legacy of using computers and networks to help companies make more money (and create more value as they do so). Unfortunately, companies tend to think for themselves first, often to the detriment of overall economic benefit.

I see a world where every individual is engaged and empowered to get the most out of their relationships with vendors–vendors of all sizes. In that world, not only are individuals and vendors each getting and creating more value directly, the entire economy is operating at a higher efficiency as less money is spent on wasted advertising and product development and more is spent on fulfilling verified demand. This would supercharge Adam Smith’s invisible hand and provide a significant increase in aggregate global wealth for everyone. It takes the benefits of the zero-distance network and extends it efficiently into the domain of user-driven commerce.

Bart Stevens recently suggested a breakdown on the potential economic impact of VRM, based largely on a post by Steve Rubel arguing that $1B is wasted in online advertising today.

First, I anticipate the Personal Datastore to become a design pattern that underlies other VRM services, rather than a service by itself. In fact, a PD isn’t really a PD unless it enables VRM services explicitly… Personal Datastores aren’t just online storage like Amazon’s S3.

Second, I think the $1 Billion number is far too small. Steve is only estimating the CPM costs for display ads that are literally missed by users during eye tracking studies. That’s an intriguing number because those ads truly are wasted… there isn’t even any brand exposure because the ads are not even seen. It’s like paying for ads in a magazine that is never opened by a real reader.

On the other hand, there are still plenty of ads that are seen by the wrong people and CPC ads that are clicked on by the wrong people. Note that for the “right” people, those ads arguably generate useful brand exposure, so they aren’t wasted.

When advertising starts with the advertiser, it inherently wastes money, as it inevitably buys placement in ineffective or misaligned media. By now it is an old chestnut that advertisers waste half their budget–they just don’t know which half. Sometimes advertising is an investment in exploring potential markets… the goal is the data gained in the test marketing, which isn’t entirely a waste. Other times advertising is educational outreach where the goal isn’t so much to trigger a sale, but instead to introduce people to new products and services. Sometimes this is called demand generation. And that still leaves a vast amount of waste, buying media (offline or online) that just doesn’t perform or create any value. The potential savings in these areas is not only missing from Rubel’s analysis, I’d wager it is far more than $1 billion.

The huge potential of VRM is to turn these models inside-out, by providing a scalable pipeline directly into the product development and sales divisions of capable firms. Instead of Vendors guessing what people want, VRM services can cost-effectively tell Vendors what people truly do want. If the product is available, the sales team can enable purchase and delivery. If the product doesn’t exist, the Vendor can create it if demand is sufficient.

This new paradigm is exactly the shift from Attention to Intention that Doc and I have been advocating. The Attention game is the world of traditional advertising, where the industrial manufacturer competes in mass media to get the attention of the right consumers in order to generate demand for their products and services. Given that attention, they seduce, cajole, and entertain in hopes of winning new sales.

The Intention game, on the other hand, starts with explicit requests from the user to fulfill actual demand. Sometimes that intention will be nascent, needing further exploration and discovery. But eventually, for the segment of the population that finds something they want or need, that intention shifts from educating oneself about available options to seeking specific satisfaction, that is, buying a solution. Because intention starts with the user’s commitment to take the relationship to the next level, it immediately takes a vast amount of guesswork and wasted advertising out of the equation.

This guesswork and wasted advertising is probably closer to $100 billion/year, but that’s just my gut feeling. And that number only addresses the loss side of the equation, that is, the money we save by not wasting product development and advertising dollars. It ignores the value of products and services that today languish as innumerable missed opportunities–missed because companies have no way to efficiently gauge true market demand. There are undoubtedly services and products that exist–or could be profitably offered today–which fail to reach customers because we don’t have a suitable mechanism for connecting the right customers with the right companies. This potential to close the gap between potential sales and unmet demand, is simply too large to estimate.

The Cost-Per-Action/Pay-for-Performance business model of Affiliate Marketing is likely to continue to transform the ad industry, significantly reducing billions in unnecessary expenses, including the $1B wasted on unseen display ads in Rubel’s analysis.

It won’t be until we transform explicit intent into new offerings and new sales that we unleash the vast potential that is VRM.

Phil Whitehouse recently served up some nuggets to stimulate conversations at next week’s VRM2008 in Munich.

I’ve been thinking a lot about VRM lately. Not so much about what it means, but rather the mechanism of how it can work.

If you’re new to VRM, it can be summarised like this: it’s the reciprocal of CRM. Rather than being bombarded with advertising, much of which is irrelevant, and the rest irritating, wouldn’t it be nice if you could just tell vendors what you want, on your terms? Without even going to the trouble of looking for them? If they’re willing and able to respond, they do so. Everyone else goes on their merry way. It’s all about sharing the data you want with the people you want.

Some examples from Doc Searls (Cluetrain Manifesto dude), who heads up the VRM project, include:
I want to:

- Buy a power convertor near St.Paul’s in the next three hours, at any price
- Buy a stroller for twins near Highway 70 in Kansas today for under $300
- Buy an Apple laptop with a 500gb HDD and weighs under five pounds, as soon as it comes out
- Buy a double decaf cappuccino at the next exit on this highway
(You can see more examples presented by Doc on this photo)

There are a few big problems that need solving. Filtering is one (both on the outbound request and the way back in), targeting is another (how do you choose which vendor to share your data with?), organisation is a third (by what mechanism do customers agree to share their data, and in what form, while retaining control over it?).
…

don’t know much about establishing standards. My erstwhile colleague Paul Downey, on the other hand, represents BT at the W3C and thus knows a bit about standards. He sez this will be a hard problem to crack, and he’s probably right. Big question: to what extent would we, the customers, allow brokers to help create this standard?

My view is this problem needs to be overcome before VRM can move forward, regardless of whether brokers are involved.

…

Good stuff. As the chair of the Standards Committee for Project VRM, it might be obvious that I think we need to create some standards.

At the end of the day, interoperability requires either standards or one-to-one interoperability engineering. The user-centric Identity movement has grown like crazy in the last few years largely because a hybrid of these approaches have been used, as OpenID, Higgins, CardSpace, and Liberty (among others) took their 1.0 products and figured out how to make them work together, leveraging standards like WS* and SAML as they did so. The nice thing about standards is that once they are in place, they reduce an O(n^2) problem, where every software vendor has to coordinate with every other vendor, to an O(n) problem where each software vendor coordinates to the standard.

The problem with standards is they are slow to develop. But once you have some apps and some standards at the 1.0 level, the efforts towards interoperability can get serious traction, like they did with the user-centric Identity movement.

I’m hoping we can engender a similar development cycle with VRM. We need both working applications and formal standards and specifications, especially with regard to data formats and communications protocols.

I’ll diplomatically disagree and agree with Bart (read his comment on the original post) regarding leaving standards to others. On the one hand, we should leverage existing work as much as possible. For example, I see Higgins and XRI playing a major enabling role for us. On the other hand, while the Dataportability and Higgins guys are doing great work, they are not necessarily solving the problems VRM has set out to tackle, namely reinventing the marketplace on behalf of individuals while creating more value for vendors.

As an example, the Dataportability movement has framed the problem in terms of Data and Portability. This brings to mind exporting and importing “my” data from vendor to vendor. That’s a start toward liberating users from vendor silos. However, I think the real win is in user-centric services, where the location of the “data” is essentially irrelevant–even as it is hosted under the control of the user–and all user-authorized vendors can access the data through approved services.

That’s the idea behind the Personal Address Manager, which we’ll be discussing in Munich. Your actual postal address isn’t that much of a problem from a dataportability perspective. It’s just a few lines to enter and no real need to “export” it from some vendor’s silo. However, when you change your address, it would be nice for the new address to automatically propagate to those authorized to get it. Or, for more sophisticated vendors, to have the address provided on demand, so that they never send postal mail to the wrong address. Such a service would be automatically discoverable by vendors using the Identity layer to authenticate and authorize exactly who gets it.

I see the job of VRM as working through these scenarios from the user’s perspective and ensuring the development of enough standards and technology for a complete implementation.

In any case, I’m looking forward to seeing Phil and Bart at VRM2008. There’s plenty of room to continue this conversation. Join us if you can; it should be fun. =)

From Yahoo News:

Majority Uncomfortable with Websites Customizing Content Based Visitors Personal Profiles

Level of Comfort Increases When Privacy Safeguards Introduced

ROCHESTER, N.Y.–(BUSINESS WIRE)–A majority of U.S. adults are skeptical about the practice of websites using information about a person’s online activity to customize website content. However, after being introduced to four potential recommendations for improving websites privacy and security polices, U.S. adults become somewhat more comfortable with the websites use of personal information.

Good stuff, although one should read closely to understand exactly what users dislike. Customization isn’t the problem… it’s the unauthorized invasion of privacy. The questions asked by Harris were rather leading. It would be interesting to see what people say to “if asked, would you allow a Search engine to provide enhanced results based on your behavior.” My understanding is most people do opt-in to the advanced features of Google desktop, which asks essentially the same question at install time. People don’t like surreptitious activities, but if you ask up front, it’s much easier for folks to say yes.

Here’s yours truly with Trent Adams and Steve Greenberg of Dataportability, talking about VRM. Also in the podcast: dataportability news and Kaliya Hamlin on the Data Sharing Summit.

Kudos to Assemblyman Richard L. Brodsky in the NY State Assembly for taking on GoogleClick and the rest of the back-end invisible online tracking services.

The NYT reports A Push to Limit the Tracking of Web Surfers’ Clicks:

AFTER reading about how Internet companies like Google, Microsoft and Yahoo collect information about people online and use it for targeted advertising, one New York assemblyman said there ought to be a law.

 

Michelle V. Agins/The New York Times

Assemblyman Richard L. Brodsky, the sponsor of a New York bill to limit how companies collect data on computer users.

So he drafted a bill, now gathering support in Albany, that would make it a crime — punishable by a fine to be determined — for certain Web companies to use personal information about consumers for advertising without their consent.

And because it would be extraordinarily difficult for the companies that collect such data to adhere to stricter rules for people in New York alone, these companies would probably have to adjust their rules everywhere, effectively turning the New York legislation into national law.

“Should these companies be able to sell or use what’s essentially private data without permission? The easy answer is absolutely not,” said the assemblyman who sponsored the bill, Richard L. Brodsky, a Democrat who has represented part of Westchester County since 1982.

…

“A law like this essentially takes some of the gold away from marketers,” said Joseph Turow, a professor at the Annenberg School for Communication at the University of Pennsylvania. “But it’s the right thing to do. Consumers have no idea how much information is being collected about them, and the advertising industry should have to deal with that.”

This is an absolute no brainer.

If you don’t have permission, don’t track users.

In the physical world, we have developed fairly robust rules of social etiquette and even laws regulating this sort of behavior. Can you imagine how creepy it would be if some stranger surreptitiously followed you around, noting where you go, what products you buy, even what sections in the supermarket you lingered in? Yech! Get that stalker away from me.

And yet, that is exactly what most (if not all) online ad networks do to maximize their ability to sell high margin ads targeting Internet users. It makes sense. If they can tell from your clickstream behavior that you are likely looking for a new car, then they can create a lot of value by showing you new car ads. Value for advertisers and value for you… after all, you ARE in the market for a new car, right?

When it works, it’s cool. But what about when you don’t want it working? When you want a little discretion as you window shop? When you’d like some privacy? Unfortunately, it doesn’t work that way. Users can’t tell when the ad networks are, or are not, tracking–it is all invisible to them–and there’s no equivalent of a “do not call list” to turn off such tracking.

The right answer is to move toward user-centric advertising, where the user is explicitly in control of all the data used to offer them ads and can even limit the types of ads shown. This resonates with Esther Dyson’s testimony to the Federal Trade Commission at their Townhall on behavioral targeting and her subsequent article at Huffington Post, where she proposed using a “Disclosure 2.0″ approach to this issue.

If advertisers and ad networks can create real value with behavioral tracking and targeting, then full disclosure–and even full user control–will only enhance trust and deepen the relationships between businesses and customers. The long term value of a customer depends on building viable, healthy relationships. Relationships depend on trust. By engaging openly and responsively with their customers–with full disclosure and as much user control as possible–companies can craft entirely new, more trusting and more profitable relationships while customers feel more comfortable about their personal boundaries, have increased confidence in their vendors, and get easier access to better products and services.

Advertisers need to get this. Ad networks and search engines need to support it. And it may be that regulators need to enforce it.

[This is precisely the sort of payoff for vendors that Project VRM is working towards.]