It is time to give users more control over Search.

At VRM2008 in Munich and at IIW in Mountain View, I started a conversation about User-Driven Search, the premise: what would it mean for users to truly drive their searches?

User-driven is a new term that came out of the VRM community riffing on the meaning of user-centric development and user-centric identity. User-centric is a nice term, but it could be construed as limiting. For example, user –centric definitely implies that that the user is at the center of attention and the focus of the architecture, but it doesn’t necessarily mean the user is in charge of the experience. That’s a key distinction.

Not just tuna salad

Adriana explains this difference between user-centric and user-driven as metaphorically the difference between buying ready-made tuna salad or picking and choosing your own ingredients and making the tuna salad yourself. When I first talked with Doc about user-driven instead of user-centric, Jim Carrey’s The Truman Show immediately sprang to mind: from birth, Truman is the protagonist in a huge reality show revolving around him… only he doesn’t know it. The climax of the show is Truman discovering the rest of the world and confronting his father/producer. Clearly the Truman Show is Truman-centric… but it is most definitely not Truman-driven.

It’s about impetus and authority

For me, user-driven means that the user provides the impetus and is the controlling authority throughout the transaction. Sure, sometimes there is negotiation or collaboration with others… the user isn’t omnipotent, after all. However, the user is in charge of creating his or her own experience. This fits with user-constructed or customized solutions, like the tuna salad recipe. However, it has implications far beyond the limits user-created or user-customizable architectures.

Is the user initiating the experience? Is the user’s moral authority the primary control throughout the system? Is the system transparent to users, enabling them to make their own informed decisions about what will be presented to them and how it is presented? Is it the user who is shaping the input, intermediary results, and final outcome? If so, then it is user-driven. If not, it isn’t.

When it comes to the tuna salad metaphor, this is the equivalent of the tuna salad being made when I ask for it and on my terms. Not before. And although I could choose to make the salad myself–that is definitely user-driven–it could also be made by someone else to my specifications… extra mayo and black pepper, no onions, thank you.

Search as user-driven

Google’s keyword query-response approach to Search is, of course, user-driven to some extent. Nothing happens until the user enters a query, users are free to enter any query, and the system responds with results tailored for exactly what the user queried. The user does shape the experience to a limited degree. And yet, it still provides only a slim façade of user control. There is no way to modulate the algorithm, no way to let Google know which results are good or bad, and no way to refine the search other than keyword guessing games. And, perhaps most importantly, there is no way to manage the search beyond the immediate query. For that, the user is dependent on other techniques: bookmarking, cut & paste, opening multiple windows or tabs, even printing to paper or PDF to keep track of good finds. Evolution in Search History management is starting in the right direction, but the ideas here have been rather uninspiring so far.

User-driven systems create value inherently

The limits on the user-driven aspects of Google are particularly ironic given that it is precisely the element of user control that creates Google’s greatest asset: focused attention. Google’s money making asset is the collection of user-specified queries, queries that explicitly state words related to the user’s interest and implicitly denote user intent. It is precisely because the individual enters a specific keyword that Google is able to sell targeted ads… at great profit and benefit to advertisers and searchers alike.

The query entered in the Search box gives Google a implicit statement of intent. That intention is the gold Google resells to advertisers. If Google didn’t let users drive that intention, if they looked more like a content site or “Internet portal”, they’d have a lot less intention to monetize.

If we can extend that control, if we can make search even more user-driven, if we can enable richer, more explicit, more user-driven expressions of Search intent, I believe we can create even more value for everyone involved: search companies, advertisers, searchers, even non-paying websites showing up in “organic” results.

What does it mean to have User-Driven Search?

At SwitchBook, we’ve been doing a lot of thinking about what User-Driven Search might mean. I like starting the conversation with a simple example: what would it mean if I could take my search history from one search provider to another? This “dataportability” example is just an initial notion of how Search might become more user driven.

So, what do you think of when you hear (or read) “User-Driven Search”?

I’ll be leading a session on this topic at the VRM Workshop next week. I hope you can join us.

This material is based, in part, upon work supported by the National Science Foundation under Grant No. 0740861. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

Bart Stevens recently suggested a breakdown on the potential economic impact of VRM, based largely on a post by Steve Rubel arguing that $1B is wasted in online advertising today.

First, I anticipate the Personal Datastore to become a design pattern that underlies other VRM services, rather than a service by itself. In fact, a PD isn’t really a PD unless it enables VRM services explicitly… Personal Datastores aren’t just online storage like Amazon’s S3.

Second, I think the $1 Billion number is far too small. Steve is only estimating the CPM costs for display ads that are literally missed by users during eye tracking studies. That’s an intriguing number because those ads truly are wasted… there isn’t even any brand exposure because the ads are not even seen. It’s like paying for ads in a magazine that is never opened by a real reader.

On the other hand, there are still plenty of ads that are seen by the wrong people and CPC ads that are clicked on by the wrong people. Note that for the “right” people, those ads arguably generate useful brand exposure, so they aren’t wasted.

When advertising starts with the advertiser, it inherently wastes money, as it inevitably buys placement in ineffective or misaligned media. By now it is an old chestnut that advertisers waste half their budget–they just don’t know which half. Sometimes advertising is an investment in exploring potential markets… the goal is the data gained in the test marketing, which isn’t entirely a waste. Other times advertising is educational outreach where the goal isn’t so much to trigger a sale, but instead to introduce people to new products and services. Sometimes this is called demand generation. And that still leaves a vast amount of waste, buying media (offline or online) that just doesn’t perform or create any value. The potential savings in these areas is not only missing from Rubel’s analysis, I’d wager it is far more than $1 billion.

The huge potential of VRM is to turn these models inside-out, by providing a scalable pipeline directly into the product development and sales divisions of capable firms. Instead of Vendors guessing what people want, VRM services can cost-effectively tell Vendors what people truly do want. If the product is available, the sales team can enable purchase and delivery. If the product doesn’t exist, the Vendor can create it if demand is sufficient.

This new paradigm is exactly the shift from Attention to Intention that Doc and I have been advocating. The Attention game is the world of traditional advertising, where the industrial manufacturer competes in mass media to get the attention of the right consumers in order to generate demand for their products and services. Given that attention, they seduce, cajole, and entertain in hopes of winning new sales.

The Intention game, on the other hand, starts with explicit requests from the user to fulfill actual demand. Sometimes that intention will be nascent, needing further exploration and discovery. But eventually, for the segment of the population that finds something they want or need, that intention shifts from educating oneself about available options to seeking specific satisfaction, that is, buying a solution. Because intention starts with the user’s commitment to take the relationship to the next level, it immediately takes a vast amount of guesswork and wasted advertising out of the equation.

This guesswork and wasted advertising is probably closer to $100 billion/year, but that’s just my gut feeling. And that number only addresses the loss side of the equation, that is, the money we save by not wasting product development and advertising dollars. It ignores the value of products and services that today languish as innumerable missed opportunities–missed because companies have no way to efficiently gauge true market demand. There are undoubtedly services and products that exist–or could be profitably offered today–which fail to reach customers because we don’t have a suitable mechanism for connecting the right customers with the right companies. This potential to close the gap between potential sales and unmet demand, is simply too large to estimate.

The Cost-Per-Action/Pay-for-Performance business model of Affiliate Marketing is likely to continue to transform the ad industry, significantly reducing billions in unnecessary expenses, including the $1B wasted on unseen display ads in Rubel’s analysis.

It won’t be until we transform explicit intent into new offerings and new sales that we unleash the vast potential that is VRM.

Alex Iskold of Blue Organizer asks “What is the Killer App?” for the Semantic Web in an article that nicely condenses the current best of class in the major contending promises of what Tim Berner’s Lee has recently dubbed the Giant Global Graph:

• Natural Language Understanding
  • No longer a need for cryptic “Googlese” to get the computer to give you want you want.
• The Genie in the Bottle
  • The magically perfect assistant who can answer any question or satisfy any need you might have.
• Semantic Knowledge Bases
  • Structured databases that have deep understanding of the meaning behind the data, rather than just the characters and numbers used to represent the data. Think Freebase and Twine.
• Semantic Search
  • Natural language understanding driving search results, so you can ask questions like “What clubs does Tiger use?” rather than Googleses keyword queries. Hakia, Powerset, and Cognition are all in this space.
• Social Graph
  • The pan-Internet interconnection of you and all your friends, associates, family, etc. LinkedIn, Plaxo, MySpace, FaceBook.
• Shortcuts
  • Intra-page shortcuts that augment a web page’s content to enhance a user’s browsing experience. This includes SnapShots from Snap, BlueOrganizer and SmartLinks from Alex’s own AdaptiveBlue, Shortcuts from Yahoo!, and In-text search from Lingospot.

It’s nice walk through the space and particularly interesting how Alex responds to the current state-of-the-art in each. I’ll summarize here, so I can respond in turn (check out the full article for Alex’s actual statements):

• Natural Language Understanding
  • Huge, hairy problem. No solution in site.
• The Genie in the Bottle
  • Even harder. Needs magic that isn’t even conceptually well understood.
• Semantic Knowledge Bases
  • More detailed data is good, but does it really help users? Not emotionally catalytic enough for people to actually get excited and jump on board.
• Semantic Search
  • Doesn’t look like the killer app so far, because none of the “semantic” approaches seem to improve much on Google.
• Social Graph
  • This is just a subset of the semantic web and therefore not its killer app.
• Shortcuts
  • An up and coming category, these embedded shortcuts remove search as the killer navigation online. However, it is still young, misunderstood, and also lacks emotional umph.

First, the most intriguing item is that Alex is candid enough to be critical of the category in which he places his own company’s flagship product. Perhaps AdaptiveBlue has turned the corner on their conceptualization of the market and are rapidly, fiercely developing their next innovation, their next rev, the thing that just might become the killer app of the Semanic Web. That makes me curious, indeed.

Second, I like the break down, but naturally have some slightly different opinions. SwitchBook is still largely in stealth mode–we have yet to publish much on what we are doing even though we are relatively open in face-to-face meetings. However, from my posts here you can guess that it involves search, user-centrism, and particularly the principles underlying VRM.

So let’s look at Alex’s breakdown again:

Natural Language Understanding

Definitely a huge problem. Not only do you have to deal with the incredible elasticity of language, once you’ve mapped the natural language into some sort of internal representation, you still have to figure out what the heck you are going to do with it.

In other words, “understanding” is context specific not just in terms of words having different meaning in different places–Jaguar could mean a car, a cat, or an operating system depending on whose brochure or website you find it on–but it also has different meaning based on what you (as a system, as a service) are going to do with that understanding.

• Are you going to return web pages that contain Jaguar with the same meaning?
• Are you going to offer alternatives to the term Jaguar, like a thesaurus?
• Are you going to translate Jaguar into other languages?
• Are you going to sell Jaguar compatible products?
• Are you going to reason over the threats and opportunities of Jaguars?

All of these require fundamentally different internal representations of the “understanding” of the natural language from the user.

As Jaron Lanier will tell you, language is an interface by which people remotely control the world outside their mind. We use it to communicate with others to get what we want and to understand how to respond to others (which is basically figuring out how to eventually get others to give us what we want). As such, its primary use, its raison-d’etre, is to influence the world around us. So, what we really want isn’t to understand the language, but to understand (1) what a speaker wants and (2) how to influence the world.

It turns out, people are incredibly adaptive at both of these tasks. Language is just one of the interfaces we use and we are capable of learning entirely new tools quickly when they demonstrate a more efficient, more effective way to get what we want. The humble spreadsheet is one of my favorite examples of this. I believe that more people “program” in MS Excel than in any classic programming language: we write mini-programs using functions like sum() and average() and put data in and look at the results. Who would’ve thought that entry-level clerical workers, accountants, and soccer moms around the world would’ve learned to program? And yet, they do. In my opinion, Excel is probably the most widely used programming environment in the corporate world.

Could you imagine trying to replace that with Natural Language? I can only imagine that a natural language version of Excel would be more convoluted and harder to use, but maybe that’s just because I lack imagination.

The Genie in the Bottle

This is more interesting. I agree that this goal is arbitrarily far away–no one will crack this nut entirely until we have both omniscience and omnipotence programmed into our software (and that is essentially never). However, by understanding clearly exactly what the Genie would do if he or she could, then you have a starting point for building innovative solutions.

Consider the development of online virtual worlds. Many people also said that the fictional Star Trek holodeck is arbitrarily far into the future, that, like the Genie, it requires so much advanced technology as to effectively be magic. And yet, Janet Murray’s Hamlet on the Holodeck gave us a realistic assessment of the current state of the art and how we might eventually get there. Sure, we are still arbitrarily far away from the uber virtual experience of the Holodeck. But Second Life, World of Warcraft, and Grand Theft Auto have all broken incredible ground in making a simpler, more feasible version of that experience available today to tens of millions of people.

So, what we can learn from the Genie is how to think about the “perfect” Search service. Imagine for a moment the absolutely perfect search service. Think bigger than natural language search. Think bigger than talking to your computer and getting what you want.

The perfect Search is when you only just barely have to indicate your intention and your search result appears. Somehow, magically, the system just knows what you want and when you are ready to actually act on that desire, the system has already brought your desire to you. No more running to the vending machine to get a soda from an arbitrarily limited selection in fixed volume and vendor-mandated packaging. The system knows you are getting thirsty, knows what you want (not just from history information but even from sensing your current blood-sugar and taste craving) and how you want it, and the moment you commit to getting that soda, it appears at your desk–perhaps even without you knowing exactly which soda you wanted today. All of this done discretely, unobtrusively, privately, and with the utmost discretion so neighbors or co-workers don’t see what you’d rather they don’t. The action, ultimately, is always driven by your committed intention. Not your attention, not some statistically predicted estimate of your desire, but your actual, expressed commitment to realize a particular desire. Express an intention and magically, it is fulfilled.

That’s the Genie.

While it isn’t yet available, bits and pieces of it are becoming available, just as online text MUDs and World of Warcraft are bits and pieces of 30 years working towards the ultimate virtual reality. By placing the committed intention of the user at the core of value creation, at the heart of the system design, I believe the Genie provides an almost the ideal model for conceptualizing the Holy Grail of Search.

Semantic Knowledge Bases

Essentially, I agree with Alex, this is a technology looking for a problem. “Better” data and more “powerful” ways to interact with and reason over that data should provide better results and is, therefore, a Good thing–assuming there are no other costs. Unfortunately, the semantic web has significant transitional and ongoing costs to turn the free-form, anyone-can-post-anything World Wide Web, into a system where participating as a first class of citizen requires using RDF or microformats or some other arcane technology to transform formerly arbitrary scribblings–and marketing and online stores and customer service and media outlets and whatever–into semantically structured information. It requires an imposition of structure that is inherently limiting and counter to the user-centric architecture of the open web.

Nobody wants to pay that cost unless the immediate value to them is obviously much greater. And so far, the value is uncertain and far into the future.

Semantic Search

Alex suggests that because none of the semantic search companies is better than Google that semantic search isn’t the killer app. Well, Google uses a lot of semantics in its Search. Most users just don’t know it. They’ve used Latent Semantic Indexing for years and AdSense is all about wicked smart semantic analysis of web page content for matching ads from the Google ad universe. In fact, one of the more interesting semantic tricks Google does is one you can see for yourself. Try typing “jaguar” (or some other ambiguous term) into Google’s query box.

You’ll find that alternative meanings of “jaguar” all show up in the early results. Jaguar as a car. Jaguar as the cat. Even the Jaguar quantum chemistry package from Schrodinger, which has no reason being in the top ten at Google. Google does this because it knows that from the limited query box, it can’t figure out which Jaguar you really mean. But it also knows that users will filter out the misses and get excited about the hits. They design for the “Ah-hah” moment. As long as one in ten (or so) results matches the user’s intended meaning of Jaguar, then Google gets credit for finding the “right” jaguar. Brilliant.

So, I argue that any search that isn’t semantic is a dinosaur waiting for the undertaker. Maybe it isn’t a killer app as a distinct service, but it is already an integral part of the #1 killer app of the Web, Search.

Social Graph

On this one, Alex fails to explain clearly enough why he doesn’t like it. Any killer app is going to be a “subset” of the entire market. Email isn’t the totality of the Internet, but it is the killer app that first broke down the isolated IT networks and marched like Sherman all the way through to the consumer market to give the sexier World Wide Web a fighting chance at establishing the Internet as much a fundamental part of the civilized world as electricity, running water, and paved roads.

Actually, I think the social graph might be the killer app of the Semantic Web. It doesn’t deliver the full value of the Semantic Web, but it provides such immediate, obvious value for so many people that once the privacy controls are worked out, many many people are going to be surfing the Semantic Web without knowing it as they seamlessly mingle across their social internetwork through the former silos of Facebook, MySpace, Plaxo, and others. If it can be a killer app without people giving it credit, then the Social Graph is definitely a contender.

Shortcuts

This is absolutely illuminating. I like AdaptiveBlue’s product a lot, and others in this category have potential. However, I usually find the disjoint interactions confusing. Shortcuts, by nature, interfere with the “normal” web experience and are inherently intrusive. I happen to have Snap installed on my machine and I’m still surprised and often annoyed when it pops-up “previews” of links I’m doodling my cursor on.

I do that… I doodle mouse and doodle click. I have the same problem at the New York Times’ website, actually. They allow you to look up the meaning of any word on a page just by double-clicking on the word. Problem is, I doodle-click meaninglessly, sort of a virtual twiddling of my thumbs as I browse. And -whoops- I just triggered a new page download I don’t really want. It is a mess.

So, shortcuts have a long way to go to be less intrusive and to find the right “intuitive” connection with the user. Ultimately, I am a huge fan of augmenting the traditional “browse”-based experience of the web, rather than replacing it wholesale. People like the web. They like their services. They like the freedom of going anywhere that supports http and html. And yet, many of those websites don’t have the technical wherewithal to get “semantic”.

BlueOrganizer does a nice job, for example, of connecting IMDB listings of movies with NetFlix so it is easy for you to go from the Internet’s unofficial authority on movies to the leading movie-on-demand service. All without NetFlix or IMDB needing to do anything. That sort of user centrism is critical to the next evolution of the web and it’s the semantics of what is already on the web pages that make that possible. Shortcuts are just one effort to do something with that semantic data. Perhaps as they grow up, they will become more useful to more people.

Closing

Again, despite my initial hopes, I have written WAY too much, which is a pathological flaw I seem to have. Thanks for hanging in there.

My point in responding to Alex’s post is simply this: any killer app needs to start and end with the User. This is so true it has become a software development truism that everybody knows is important, but few know how to translate into their feature development schedule. Technology alone–like Natural Language Understanding–will never be a killer app. Only when someone figures out how to make it electric for users–exciting and immediate and so obviously valuable–can any innovation become a killer app.

With all due respect to the folks who love this term, the Semantic Web is one of those bundling concepts that is about as useful as the term “Electric Appliance.” It is useful in describing a category of product, but completely useless in helping retailers make decisions about what products people are going to buy this season. Until companies move beyond that catch all descriptor into product discussions that connect with what users already understand and want… none of the “semantic” offerings can possibly breakthrough to being a true killer app.

Over at On the Pod, Duncan Riley hosts a rambling but intriguing chat about APML and its relationship with the latest Social Network developments:

Ashley Angell, Chris Saad (Faraday Media) and Jon Cianciullo (Cluztr) join Duncan Riley to discuss APML, OpenAuth, OpenSocial and how we are moving towards open data sharing.

Even with the audio problems, it is worth listening to the whole bit. There is a great, concise analysis of the real value and limitations of OpenSocial, but my ears perked up especially when Chris endorses VRM and, more importantly, connects the success of APML to its role in providing user control over Attention data.

I have said several times that Attention efforts are missing the big win, digerati conversations about the “Attention Economy” notwithstanding. Getting your attention without providing value is annoying. We will soon see the impact of that as people begin to react to Facebook’s latest innovations in advertising.

It sounds redundant to say it, but the real goal for people is to realize our Intentions. That’s what “intention” means. We put our energy and will into realizing our intentions. Attention just happens to be how we filter out the signal from the noise. It does not inherently translate to value for anyone. In fact, distracting your attention is a key skill of politicians, magicians, and con-men everywhere. So, it isn’t about “Attention.” It’s about “Intention.”

The opportunity, then, for service providers and software vendors is to provide tools for individuals to manage their Intention. Solve that while facilitating vendors’ goals–because many, but not all, Intention activities are directly monetizable in a transaction–and you have a service or product that can generate serious value for everyone involved.

That’s the promise of VRM.

As I’ve mentioned before, VRM–or Vendor Relationship Management–is at the core of SwitchBook’s approach to tools for Complex Search. Our involvement in that effort has transformed how we think about Search, advertising, and online marketplaces.

VRM’s mandate is straightforward: Enable buyers and sellers to build mutually beneficial relationships. The vast majority of online buyer/seller relationships include Complex Searches prior to a transaction and the bigger the transaction amount, the more effort that goes into the Search and therefore, the more important and useful the tools provided to individuals. We see a direct link between providing people control over their online Searches and enabling them to have richer, more rewarding relationships with vendors. To us that means simplifying how people realize their Intentions online by connecting them with the right resources more efficiently and more credibly.

What is intriguing about the podcast is the endorsement of VRM and the related commitment to empowering user choice through APML tools. Whatever words we use to describe it–”Attention” or “Intention”–increased user control is definitely part of the solution.

As Attention becomes more and more shaped to be responsive to user choice rather than a smart database of people’s online behavior, and the more it empowers both explicit expressions of interest and the implicit meaning we can glean from people’s clickstreams, the closer Attention comes to Intention.

To be fair, much of what we are doing with Intention incorporates a lot of Attention data. So, while there are still some key distinctions, it is good to see APML folks talking about VRM in a way that suggests a fruitful convergence is not so far away.

On the Social Network Interoperability list, Danny Ayers recently pointed to a great post, “The World is Now Closed” by Dan Brickley, with the following quote:

[[from Dan Brickly:] So what am I getting at here? I guess it’s just that we need these big social sites to move away from making teen-talk claims about how the world is - “Sally (now) loves John” - and instead become reflectors for the things people are saying, “Sally announces that she’s in love with John”; “John says that he used to work for Microsoft” versus “John worked for Microsoft 2004-2006?; “Stanford University says Sally was awarded a PhD in 2008?. Today’s young internet users are growing up fast, and the Web around them needs also to mature.]

This is fascinating. It belies an underlying hubris of much thinking in both AI and the semantic web. We often imagine that it is somehow possible to map out, understand, or process some sort of “objective” set of facts. Computer Science practically conspires to force this world view on its practitioners. When programming, we not only start with assumptions about data, we must concretize those assumptions so our algorithms have something to transform from input to output. “Fuzzy logic” and neural nets embrace ambiguity, but computer science on the whole lives in a world of clearly defined inputs and outputs. It literally forces one to think in terms of objective data.

But in the real world, nothing is that simple. Was Princess Diana murdered? Is OJ guilty? Is DNA evidence conclusive? These are legal examples, where ambiguity is argued to death in court so contestants can eventually move on with the rest of their lives, but what about love, betrayal, politics, or discrimination? Does he really love her? Did your business partner always plan to stab you in the back or is he actually still acting in what he believes is in the best interest of the company? Were there weapons of mass destruction? Did race or gender influence your hiring decision?

Answers to these kinds of questions can’t be reduced to facts. They can only be reduced to “good enough” approximations of facts.

This is particularly apparent, for example, in Freebase, a socially maintained structured “factual” semantic database which came out of Applied Minds and at least in part from the brilliant mind of Danny Hillis. Freebase is like Wikipedia on crack. Delightfully ambitious, it has set out to leverage the social editing power of wikis to construct a semantically and computationally accessible knowledgebase of everything worth talking about.

If we ignore for a minute that Wikipedia–and all similar social constructs–can never be perfectly accurate and instead accept that they can be exceptionally useful, then we can begin to see the allure of a socially edited and maintained database of facts such that a computer could query or reason over embedded topics. It’s a great idea and hopefully will create enough value by solving enough of the problem.

And yet, one can see in its “factual” hubris, the beginning of its fundamental limitations. Take for example the “type” associated with living people. There is a different distinct type for deceased people. There was a fair amount of discussion about this, but apparently rather than allow “people” to be either living or dead, it made more sense to separate the two types. Ok. It’s often easy to tell if people are really dead. But what if it isn’t? What if someone, like Steve Fosset, is lost and presumed dead? (That’s my presumption, anyway.) What about Amelia Earhart? What if an individual is brain-dead but still breathing? Do you wait for a definitive statement from a coroner? What if there is no body? The “factual” paradigm requires someone–or the collective someone of social editing–to make the call about whether or not someone is categorized as a living person or a deceased one.

And I have barely scraped the surface on religious “facts”. Both Freebase and Wikipedia (which is often used as the source used in Freebase)  address this in part by shifting from “fact” mode into contextualized statements or claims. See Jesus and Mohammad entries in Freebase. Coincidentally, at the time of this writing the Wikipedia entry on Mohammad is locked to editing because of disputes. It is the nature of the most interesting topics to generate disputes, and yet these same disputes prevent us from asserting any sort of singular claim with any honesty.

The solution used is in Wikipedia is to state that so-and-so religion claims certain things, for instance, about Jesus or Mohammad, and cite a source for those claims (and implicitly listing the editor who entered those claims). It is not clear yet how much these semantics will be captured in the underlying data structure at Freebase.

Generally, these factual databases and modeling systems (such as certain unified schema proposed by some proponents of the semantic web) implicitly require someone to distinguish what is fact from what is not, and often do so without clarifying the asserted “fact” is really a “claim”, although the editing history at least allows you to know who made the claim. The systemic requirement that somebody decides what is “true” is patriarchal, Apollonian, and unrealistic. It enforces a top-down view on the world, even though we know as a matter of practical experience that there are many, many viable and interesting and rewarding competing world views. And yet, the architectural assumptions of Wikipedia are clearly making it difficult to come to terms with appropriate language to present “facts” about Mohammad.

Whether or not there is a classic objective reality in the Ayn Rand sense is irrelevant from a systems development perspective. What’s important is that there are numberless different and competing views of the world, stored in people’s heads, in corporate data silos, and soon coordinated in individual personal data stores. No one system can ever assimilate, aggregate, and accommodate all of those distinct datasets into a unified whole. Trying to do so is a fool’s errand and designing your systems to count on it a recipe for an unscalable system.

Instead, what is important, in my not so humble opinion, is that the interfaces between as many sources as possible allow for fluid, low-transaction-cost, accurate engagement across the network, no matter who you are or who they are, moderated by appropriate rights management and identity access control, so each of us can seamlessly access the datasphere as broadly as we have the right to, as easily as if each data store were our own. Consider how most web browsers can access (mostly) all web pages. That ubiquitous access to different data fuels Wikipedia’s editorial preference for citing accessible web pages whenever making claims. That’s a profoundly simple and powerful model for engaging the world’s diverse data and communications needs. We just need to upgrade to sharable semantic interfaces and proper access mechanisms. Brickley’s comment on claims verses facts highlights a critical system requirement: the acceptance of ambiguity.

Clearly this is the kind of thinking that fuels much of my interest in VRM. Vendor Relationship Management still requires much gestation and care before it can truly be judged as a widely useful effort. But what it does in this crazy world where each data silo has divergent data and every vendor wants to own it all, is redefine the working context so that we can focus on what each individual actually knows and needs, which at least for that individual, for that customer, for that “monetizable opportunity,” is actually quite likely to be “right.” And since it is “right” for that closest dataset to that individual, it is likely to be right in a way that might create value for someone who can respond to those needs and for the person whose needs get addressed. We are working by focusing on the interface between these distributed systems, on the protocols that make networked semi-automated vendor-customer relationships work, not on any presumptions of fact or a globally rigorous index or model of all the world’s information.

Hence the incredible resonance of Dan Brickley’s observation about the relative value of “claims” verses “facts”. We can’t really know if a fact is true, generally, but we can convince ourselve that a given person or company or entity has asserted a claim. And by connecting the claim to an a particular person or company, anyone relying on that claim can decide on their own whether or not to trust that entity or keep checking the facts. For most of us, most of the time, a handle for consistent claims is enough to weave together a shared set of expectations and understandings, which we can use in the face of a philosophically intractable inability to discern the “objective” truth.

Some of this is, of course, old-hat to those folks coming from the Identity world, where they already speak of “claims” and “assertions” rather than facts. And as such, VRM gladly claims that heritage and common sensibility. If you think about it, it makes sense in a vendor relationship. Who really cares what the “factual” price of an item is when you can find a credible vendor willing to offer that same item at a better price. That’s all about claims at the interface between the buyer and seller and all about how we, as individuals, relate with vendors.

The upshot: systems that represent claims of fact made by specific entities will be more robust and more useful than systems that simply represent claims of fact. And that you can design on.

In my previous post on VRM’s Personal Datastores, I discussed how we can decentralize information services by focusing on the user as the point of integration. Not only would that give the user direct control over their personal data–to the cheers of privacy advocates everywhere–it would provide a more robust, reliable, and scalable approach for important VRM use cases, including personal health care data, media consumption histories (and licenses), personal RFPs, and more.

Three replies sum up the curious or critical responses I’ve had:

• Matthias Gutfeldt: “But how do we do it?”
• William Hayes: “Any idea when someone will step forward to provide the user information mgmt service?”
• Dave Weinberger: “How does VRM (or Joe’s vision of it) differ from federated identity schemes in which the user has control over her personal info? “

I’ll generalize these into two focused questions:

1. What is different with VRM and Personal Datastores?
2. What will it take to implement them?

VRM’s Personal Datastores (PDs) are a new inflection of the familiar paradigm shift of decentralization

The answer to the first question lies in the recent advances in user-centric identity and the upcoming access-rights infrastructure built into XRI and XDI.

Limited versions of user-centric data stores have been around for decades. The PC revolution followed the same paradigm shift: put the applications and data on the user side instead of a central mainframe. The Internet itself echoes that user-centric view of the world, especially when you consider businesses as users and online services as vendors. Any architecture that moves data control from a centralized vendor to the decentralized user resonates with the user-centrism of Personal Datastores.

What decentralized systems don’t necessarily have–and what PDs add–is structured third-party read access to that data store.

Internet Email as Personal Data Store

POP, IMAP, SMTP–essentially all store and forward email architectures–allow independent third parties to input data into a simple user data store. That’s the point. An Internet-based email service that can’t accept email from anyone on the net, isn’t really Internet email. However, with email, there isn’t a way for outside third parties, such as vendors, to access that data store. The privacy reasons behind this are self-evident. Most people don’t want neighbors or “vendors” reading their email.

Blogs as Personal Data Store

Blogs, on the other hand, offer both input and output of personal data and move a little bit further along the spectrum towards a Personal Data store. Blogs are primarily output mechanisms; users write posts and those posts are published to the world. Comments provide an input mechanism from the “cloud” of arbitrary Internet users, giving blogs a limited input and output capability for what is essentially a publicly accessible personal data store.

The access rights management on blogs, however, leaves much to be desired–and is far from enabling many core VRM scenarios.

Access Limitations

Most blogs are simply available to the public–or occasionally to a limited “internal” audience by restricting access to the web page. A VRM data store should have extremely fine grained access privileges, including by “identity class” so that, for example, all legitimate Travel Agencies could access a personal RFP for travel or certified medical doctors who have registered an emergency medical situation warrant could access a personal medical history. These sorts of restricted rights mechanisms require not only the emerging user-centric identity technology, they require an institutional infrastructure capable of reliably authenticating “travel agencies” and “certified medical doctors” who have “registered a warrant”. Ultimately, a Personal Datastore must not only store the requisite data, it must provide secure and effective access to the right vendors and individuals, and refuse access to all others.

Input Limitations

Second, the ability to “input” into a blog data store exists in the form of comments, but it is limited. Sometimes this privilege is restricted by identity (using TypeKey, OpenID, or an InfoCard for example), but not always, and access is usually restricted in a simple way: for example, any InfoCard user can post a comment on Kim Cameron’s blog on any post. This is a good start towards identity-based access rights management, but most sites have minimal distinctions between different classes of users and different data sets. Of course, blogs are for blogging, so they don’t have a need for sophisticated access functionality. However, when you treat markets as conversations, VRM needs to enable conversations between users and buyers. That implies that users can, for example:

• Input data to their Datastore
  • RFPs (requests for proposals)
  • Customer Interaction Data
    • service calls
    • RMA requests
    • bug reports
    • reviews
  • Personal Health Updates
    • Symptom Reports
    • Doctor Visits
    • Medication Log
    • Exercise Log
• Amend/Revise Data
• Reply to Vendors (securely, privately)
• Manage data access
  • publish subsets of the data to specific vendors
  • publish to sets of vendors

It also implies that Vendors can

• Access subsets of the data
• Add new data to the Datastore
  • Prescriptions
  • Proposals in response to RFPs
• Update subsets of data (securely and with reciprocal privacy)
  • RMAs
  • Customer Service History
  • Revised/Updated Proposals

This also implies that data stored by the vendor should be protected from edits by other vendors and even users (although outright deletion must remain an option). It would be a mess if users could edit responses to RFPs or prescriptions directly. Rather, the integrity of the system requires a mechanism to assure that the data is what the original author intended it to be.

This level of access functionality is essentially non-existent in blogs. Other online markets provide some elements of these features, but none that I know of place complete control in the hands of the user, enabling any vendor (approved by the user) to participate. eBay is a good start; it definitely democratizes the vendor/buyer marketplace. However, you don’t control the types of buyers/vendors who can access your listings and you still need to use eBay to culminate the transaction. It will probably be a challenge for eBay to find ways to make money while putting the transaction context in the Personal Datastore.

However, it is worth noting that CompuServe and AOL faced the same conundrum with the Internet, struggling to learn how to profitably transition from a closed-loop system to an Internet that let anyone access and publish anything. Ultimately separate business models worked quite well with access providers (Earthlink, AT&T, etc.) and service providers (Google, Amazon, etc.). Meanwhile, AOL is still struggling to define its business.

VRM and Personal Datastores will likely create a similar segmentation of silo & service businesses into specialized vendors of Personal Datastore Services and those focused services that leverage PDs to deliver new value.

Blogs + ID access = Personal Data Store?

Blogs with Identity-based access privileges start looking like a workable Personal Datastore for some VRM use cases. Consider posting a Personal RFP (request for proposals) to your blog, with appropriate tags (travel, ready-to-buy, hotel, airfare, car), pinging a pingmarket service like Technorati, and receiving offers via comments to that post. If access to the post–and the ability to reply–were seamlessly moderated by a credentialing service (so only authentic “travel agents” could respond), then we start to have a system that could work. Vendors who subscribe to RSS feeds from the pingmarket see sales opportunities right on their desktop, not unlike Shopatron’s manufacturer-to-retail online distribution service.

This architecture highlights two additional requirements: first, how can we trust the claims of the user? Second, how can we (automatically) understand the requests (and claims) of the user?

Validating User Claims

VRM relies upon users making claims of various types:

• intention and interest
  • In the market for a new car
  • Buying a plane ticket
  • Looking for a home
• affiliations
  • AAA Member
  • Retired military
  • US Citizen
  • Member of the California Bar
  • credit card #
  • employment
• facts
  • Address
  • Age
  • Gender
  • Income
• certificates
  • Licensed to drive by California DMV
  • Insured to drive by BBB
  • Security clearance by US Federal Government
  • Credit rating by Equifax

Many vendors avoid wasting time with unqualified leads, including competitors and window shoppers, as well as individuals who can’t legally purchase the product because they are underage or excluded due to export control laws. In addition, the reputational history of the user enables Vendors to focus resources on the most promising buyers. Buyers with no history or with negative history don’t deserve the same VIP treatment that proven, reliable buyers deserve. We see this on eBay, but have no clear way to leverage our eBay reputation with other vendors. A VRM system would allow reputations of this nature to arise explicitly from multiple reputation vendors, and incorporate our transaction history across multiple marketplaces.

It will be a while before our institutions implement these kinds of authentication services, but it is already happening with the earliest “adopters” (apologies to Geoffrey Moore, in his terminology, these guys are all “hobbyists” even when multinational corporations). Sun Microsystems, for example, now validates employee claims so that third party Vendors can rely on that validation for providing services. With Microsoft’s CardSpace technology built into .Net and Vista and soon Active Directory… plus OpenID and Higgins open source solutions, the Internet identity infrastructure is somewere akin to the World-Wide-Web was in 1993 or 1994. Which is to say, about to seriously explode into corporate and mainstream consciousness.

So, to answer the first question, what is different with VRM’s PDs is incredibly fine grained control over both the data and who accesses it. Today’s federation systems actually move in the opposite direction by allowing a wider and wider system of vendors to access personal data with absolutely no control by the user. The result is, frankly, a culture where people hesitate to provide full or accurate information because of fears of what vendors will do with it.

Standardized VRM Data Types and Protocols

For VRM to scale beyond human moderated interactions–the kind enabled by Shopatron where retailers personally check the Shopatron website to select orders to “bid” on–we need a solution for automated understanding of user data. No, I don’t mean some mammoth Artificial Intelligence, natural-language-processing, all-knowing, all-dancing automated salesman. What we need a standardized way for people to make claims such that Vendors can understand them. This means a cross-Vendor open standard for structuring VRM data, including claims, RFPs, personal health records, etc. In some ways, this parallels the work being done by many many folks building the semantic web. If the original data is presented in a structured, commonly understandable format, then programs can have a reasonable expectation of “understanding” it in a useful way.

Currently, Vendors typically have their own internal data structures and formats. This makes it hard to move data from one system to another. Yet, that is exactly the power of the Personal Datastore, serving as the point of integration between multiple vendors, no matter who is sourcing the original data. So, if Amazon, BlockBuster, and NetFlix all want read/write access to my PD to better understand my media consumption history–and provide better recommendations based on that understanding–then all three need to be able to store the data in a mutually understandable way.

This is a huge problem. We are essentially talking about reversing the damage done at the Tower of Babel, of integrating a formal representation of all possible data.

HUGE.

PROBLEM.

Except if we look at it a bit differently. Taken at 30,000 feet, VRM’s PDs seems to offer a secure, universally accessible and universally understandable read-write data store. That sounds great. It also sounds like an insurmountable problem. However, by breaking the data types down into cohesive use cases–at 1,000 feet–we can start to package the PDs in a way that is implementable, scales with use, and provides high-quality understandable data to individuals and Vendors.

First, think of the PD as a fungible store of any kind of data. Built smart read/write access to that data using user-centric identity systems with third-party credentialing for “identity class”-based usage. The VRM is about vendor-customer relationship data, but once the infrastructure is in place, truly any structured data makes sense. (Unstructured data just acts like another ftp or web repository.)

Second, take real-world integration problems and solve them with relatively small, focused data formats and get Vendors to support those formats. For example, a standard media-history record that any vendor can read and write into our PD. Or a standardized RFP format, potentially with an extensible RFP type so that custom, structured information can be embedded in Airfare RFPs, retail goods RFPs, or service RFPs. By tackling real-world problems and working with a handful of real-world vendors, shared data formats that provide immediate value can be developed in a realistic timeframe. By solving each of these in an open-standard, open community fashion, a library of VRM data formats will start to emerge hand-in-hand with the VRM protocols that manage the creation, distribution, and consumption of that data.

You might think of it like MIME for vendor-consumer interactions. MIME is the Multi-media Internet Mail Extension. It was designed to allow email attachments of files like word documents and images. It also allows webservers to specify the type of a file being downloaded by the browser. In both of these cases, the underlying access protocols of SMTP/POP and HTTP don’t need to know anything about what is in the MIME attachment. Instead, applications use the MIME type to do the right thing once the data arrives. In the same way, a PD should provide an identity-based fungible data store where rich data formats of different types can be intelligently stored, accessed, and managed.

The result is a system that scales by adding new open standard data-types to the open data store, just like email and the World Wide Web scaled to support images, Flash, audio, and movies.

Access Rights and Responsibilities

Now that we have a technological infrastructure–and conceptually an institutional infrastructure for validating Identities and claims–we are still missing perhaps the most critical piece of the puzzle: the legal infrastructure.

In today’s internetworked world, there is astonishingly little control over data other than denying access. If a Vendor knows who I am because they bought my name and information from some mailing list company, they can–and do–bombard me with junk mail. They share it with other divisions or sell it to third parties. Some Vendors do have reasonable privacy policies, and I would be remiss not to give a tip of the hat to eTrust, which has done much to advocate in this area.

However, with the Personal Datastore we are talking about a massive restructuring of the scale and type of information that will be made available to vendors, and making it available at incredibly low marginal costs. Not only will Vendors need a viable system for appropriate use of that data, users will need to be assured that the data they put in their PDs is protected in a rigorous way, minimizing user exposure to spam, unwanted solicitations, fraud, stalking, and identity theft. Having personal data–such as your address–in your PD must feel and be at least as secure as entering that same information at the culmination of an online purchase.

Interfaces and Phishing

There are two parts to this problem. The first is the user interface for how a user securely manages their private or semi-private personal data. Largely this has to do with minimizing phishing attacks while assuring the user can feel comfortable with the correct vendors. Kim Cameron often discusses this topic and it remains one of the biggest security risks for user-centric Identity systems. However, VRM and the PD don’t address this problem, nor do I see them ever doing so. As with the rest of the user-centric Identity movement, VRM will build upon the work of others.

Access Rights Management

The second problem is controlling what happens to the data after it leaves the PD. Or, to put it another way, providing restricted use licenses to Vendors who access your data.

I’ve consistently attacked the language of the AttentionTrust and others when they discuss users’ rights in regard to our “attention data”. Many people assert that we, as individuals, own the Attention Data sprinkled around the Internet as we “spend” our attention at various places. I have yet to receive a satisfactory answer to my queries about what it means to “own” that attention data, as it seems ludicrous to me to assert ownership over things like website access logs at YouTube or our transaction history at Amazon. Clearly, the Vendors own that data at least as much as we do.

However, when we store data in our Personal Datastore, we do own it. What the AttentionTrust and APML get right is that by collating our Attention Data in a data store on our computers–or on computers under our control–we are creating a data resource that we do in fact own and control. It doesn’t make sense to then give up that control just to get a better ad from Amazon, does it?

It might. If Amazon were to legally commit to using that data only for presenting that ad. In general, it isn’t usually the immediate use of personal data that we find annoying. What annoys us is the indiscriminant use, propagation, or application of that data out of context and for unexpected uses. I don’t mind telling the bartender what beer I’d like–otherwise she’d have a hard time serving me–but it would be annoying if that choice was broadcast on a loudspeaker “Joe Andrieu orders a Guinness” and posted to the bar’s blog the next morning. Rules of etiquette reinforce these sorts of expectations in real-world society. We call it “discretion”. But until there is something formally restricting Vendors who access a Personal Datastore, we can expect them to use all information as widely and as creatively as they can profitably do so. The consequence is that many users will refuse to expose authentic data, undermining the whole system.

At the same time, we can’t expect every vendor to read, evaluate, and agree to a custom twenty-page licensing agreement for each Personal Datastore they want to access. Instead, what we need is a handful of simple, standard access rights contracts or terms that can legally bind Vendors who access our PDs. Fortunately XRI and XDI have this sort of access rights architecture built-in. However, the actual rights contracts which would use those access protocols remain to be written.

Here are a few rights that users might want to be able to secure for their data, as well as some privileges they could provide to vendors:

1. Reciprocity–That vendors who access a particular type of data also agree to reciprocally provide updates to that data. For example, I might let Amazon access my media history records if they agree to update it with my past and future media purchases at Amazon.
2. Non-propagation–No further distribution of the data beyond the specific services authorized. No reselling to third-parties. No re-use by other divisions.
3. Non-persistence–No retention of the data beyond the session of the current transaction. For example, an emergency room physician can access my personal medical history while I’m under his or her care, but he or she can’t store that data on any internal systems.
4. Anonymous Persistence–Data can be retained, but only if it is suitably anonymized and disassociated from the individual user.
5. Editable Persistence–Data may be retained by the vendor, but it must be editable and deletable by the user.
6. Anonymized Analytic Rights–Vendor has the right to query the PD at a later point for business or operational analysis, as long as that analysis ensures anonymity after the fact.*

*One of the main reasons companies retain detailed customer data is to analyze it for business improvement. Perhaps the product is doing particularly well in certain areas or particular demographics. Perhaps certain customers are having a particularly hard time with certain product features. I expect that many of the largest Vendors will be unable to support non-persistence or anonymous persistence unless they are allowed some way to incorporate rich user data during analysis time. One benefit of the PDs as a source for Vendor analysis is that if performed with non-propagation and non-persistence, it can provide secure, private access to a much broader source of customer data than customers are willing to give and Vendors are able to capture. By shifting this datamining from Vendor silos to Personal Datastores, not only would Vendors get richer, more timely, and more accurate information for their data mining, individuals would gain explicit control over the use of personal data which is currently entirely under Vendor control in their private silos.

We are already seeing quite a bit of activity by the large search Vendors to modify their own data retention policies to be more user friendly:

http://www.mercurynews.com/business/ci_6449050?nclick_check=1

What makes VRM and Personal Datastores different?

In summary, VRM is applying user-centrism to vendor/customer relationships in a way not possible (or not worth the effort) before user-centric identity platforms emerged:

1. Fine grained user control
   • the data
   • who can access the data (and how)
   • access rights and responsibilities for those who do access the data
2. Third-party validation of claims
3. Standardized meta-data schema for sharing data with multiple vendors

What will it take to implement VRM and Personal Datastores?

So, knowing what is different with VRM makes it pretty clear what we’ll need to achieve it:

1. A service and/or software infrastructure capable of offering users control over fungible, identity-accessed data stores. On the open source software side, look to OpenID, the Higgins Project, and Sun Microsystems (amid many others). I’m not yet aware of anyone offering PDs “hosting” services, but I believe we should see PD capable products within a year or two.
2. Institutional infrastructure of organizations providing claim verification services using the user-centric Identity Provider (IDP) architecture. The Department of Motor Vehicles, AAA, Equifax, eBay, National Association of Realtors. All are capable of providing added value for their licensed drivers, members, consumers, users, and Realtors, respectively, by being an IDP and enabling Relying Parties (RPs) to provide new services or capabilities based on validated user claims.
3. Legally binding, and tractable access rights agreements. Before users are comfortable sharing personal data automagically, we’ll need to wrap clear and enforceable rights agreements around our data stores.
4. Open standards and protocols for the exchange of use-case specific data. Within communities of interest, we need to forge common schemas so Vendors can easily work with data provided by others. There isn’t really a general solution to this problem; however, with focused “vertical” approaches, we can define how we share specific personal information across the Internet.

Closing

This is a long post. If you’ve made it this far, thanks for staying with me. =) VRM is an incredibly rich vein for innovation and I hope I have succeeded in exploring that a bit with you. As a focal point for new services, software, and initiatives, VRM provides both a clear moral framework–it’s about empowering the user in their engagement with vendors–and a clear bounds on our scope: reinvent vendor-customer relationships.

There is a lot of fertile ground for VRM… and a lot of ways that the same underlying technology can be used and applied to use cases outside the vendor-relationship context. Those applications will emerge and be delightful surprises as Doc Searls and the VRM crew rally and clarify the vision we see for a VRM future–and create the technology to bring VRM to market.

Hopefully this article has given some depth and concreteness to a few VRM concepts that have been in recent discussions, especially the Personal Datastore. The entire movement is still early in its gestation. No doubt, the ideas presented here will evolve to become something different than what we have started with and I encourage and invite you to join us in evolving VRM. This is an open community effort and we welcome your input.

Jeremy Zawodny of Yahoo!, recently asked Where to buy unlocked GSM mobile phones? perhaps unknowingly following in the footsteps of VRM (Vendor Relationship Management) pioneer Doc Searls.

I’m a big fan of Jeremy’s blog, even if that is where he introduced me to the most time consuming addiction I’ve had in years (thanks a lot Jeremy). It is a nice balance of professional and personal posts, useful enough for folks in the search business and authentic enough to feel like you know Jeremy and even come to like him. In short, an enjoyable and successful blog.

In his recent post, he asked:

I may be in the market for a relatively cheap quad-band GSM phone that can be used in various foreign countries. Nothing fancy. I just need something I can drop a SIM card into and make calls and maybe send a few text messages.

The trouble is that I have no idea where to buy one and what to look out for. I’ve seen some available on Amazon.com and, as expected, they seem to be all over eBay. Plus there are various on-line stores that seem to specialize in selling unlocked GSM phones to people in the United States. But I don’t know which are trustworthy.

So if you wanted a handy little Motorola or Nokia GSM phone, where would you buy it on-line? And, if you happen to have a preference, which phone(s) would you look at or try to avoid?

In case you’re wondering, I’ve been a Verizon (CDMA) user for a few years now and have no plans to change that. I’m just looking for something that’d be handy when I and others travel.

Amazing. This is functionally identical to Doc’s VRM Inquiry looking for a new phone back in November 2006:

Okay, I’m gonna ditch my Verizon Wireless account, and the silo’d-to-hell Palm 700p that goes with it (and not with other providers). The Palm is nice in some ways, but its too heavy, too crashy, too lousy at too many things I depend on. (Dialing, for example.)

So here’s my VRM intention-market gesture of the day…

I want a phone that is GSM-based (so it works overseas as well as in the U.S.), works across as much of the U.S. and Canada as possible (Verizon has been a disappointment in this respect), has a GPS, and has an easy-to-use UI. I don’t care about PDA functions, ringtones (I like the old Western Electric bell ring, though), or camera functions. I like keys that are easy to read and use, and an address book that’s easy to synchronize with a computer. It would be nice, for personal reasons (I work for Linux Journal), if it ran on Linux. I’d rather it not (for the same reason) run on Windows. Mostly I just want it to be a good GSM phone with a GPS. And I’m willing to let the GPS function slide, just to get a good phone.

And I’m ready to buy it.

So, who gets my money, and for what? (The Nokia E62 has been recommended, but I’m not sure the phone keys aren’t too small.)

Consider what these posts have in common:

• posted on high-traffic blogs
• written by sophisticated Internet power users
• requests for cell phone and cell phone vendors
• considerable detail about what they want and don’t want in terms of features
• lack of certainty about which product meets their needs best
• lack of certainty about where to buy that product
• a request for help from their readership
• customers ready, willing, and able to buy, right now

What’s fascinating to me is that both of these posts are clear VRM gestures well before any sort of VRM infrastructure is built to handle it. Doc’s worked. And Jeremy’s probably will also, because he has a large enough, friendly enough, and sophisticated enough readership to help him find a phone that fits his needs.

I know that two VRM RFPs don’t make a trend, but for them both to show up in my feed reader and be for GSM phones by sophisticated guys who know as well as most how to research and shop online… I believe it is a bellwether of things to come.

I’ve written a few times about the VRM personal RFP. From my view it is one of the “low hanging fruits” of the VRM world. Shopatron has already demonstrated a viable approach for turning confirmed orders over to a streamlined marketplace of vendors. Now both Doc and Jeremy are showing that even without that streamlined marketplace, for some purchases, just blogging a psuedo-RFP is better than the alternatives.

Many people have raised criticisms about personal RFPs.

1. People don’t want to do all that work
2. People don’t want to wait for responses
3. You can’t change vendor behavior
4. It’s too complicated
5. Vendors can’t handle open-ended RFPs
6. There’s a catch-22: without orders, no vendors. without vendors, no orders.

On the surface, these all seem like valid obstacles. But each one is either untrue or irrelevant.

First, for some purchases, people already do an incredible amount of work. I’ve interviewed a TV buyer who spent 18 months gradually, slowly researching his options before finally purchasing a new 42″ plasma TV. And it took a full three months after he was financially committed to buying it to actually settle on the model and vendor. Another interviewee spend 28 hours over four days researching travel and work options for teaching abroad. She knew what she was looking for, had plenty of experience, and found that this was the kind of research she needed and was willing to do to plan and create her trip.

Clearly, people are willing to put a lot of work into these kinds of purchases. These are complex searches. And it seems clear to me that there are a lot of these kinds of searches that could be made easier with the right tools, including VRM personal RFPs.

Second, there is no reason that a personal RFP requires any more waiting that submitting an HTTP request. Sure, some models of RFP fulfillment have a certain amount of delay, especially if vendors need a human to process the order. I don’t know the reason for the delays at Priceline, but at Shopatron this delay is effectively hidden from the user. Shopatron’s approach works because they have a fixed set of products and vendors–and a fixed price–where they can assure timely delivery: the delay gets eaten up in the shipping and handling rather than in the purchasing.

Once more scalable VRM protocols are defined, vendors will be able to participate in real-time. If Hertz, Avis, and Enterprise are all in the VRM marketspace, there isn’t any reason they couldn’t automate responses to RFPs for rental cars. It’s not automated yet, but then again, neither were any of the direct to consumer shopping services in 1993. The web changed all that, with realtime responses for product inquiries, orders, tracking, returns, and more. So, count on a large number of VRM RFPs creating responses in seconds, especially in well-established markets like travel and real estate.

Third, vendor’s behaviors change all the time. The World-Wide Web changed a lot of things. And so has Shopatron. While new innovations always face challenges the more they disrupt current practices, there are always vendors willing to be first movers if the payoff potential is large enough. There will also be vendors who will fight tooth and nail before they do business any different than what got them where they are today. That’s just part of the challenge. We not only have to find the right vendors, we have to find the right solution, the right way to communicate it, and the right way for them to say “yes.” That’s called sales. Fortunately, I can think of nothing more motivating to a vendor than a confirmed, qualified, and capable customer.

Fourth, it is becoming increasingly clear that people demand solutions to complicated problems. Of course, we want it to be as simple as possible, but we are well beyond the early days of the world wide web where people were learning how to click on hyperlinks. People have sophisticated needs and while the simplicity of the Google search box made a huge difference in how we surf the web, even Google knows they have to move beyond that. We know that getting the right answer can’t happen magically. We are decades away (at least) from mind-reading computers that distill basic needs into real-world products and services. Some how, some way, we must express our intent so that people can respond, even better if computers can respond automatically. The challenge is figuring out how to package RFPs like Doc’s and Jeremy’s without losing the ease, flexibility, and specific vagueness, boundaries that were clear but within which many options would work. The technical challenge is significant, but I don’t believe it is too complicated for users. Instead, I suggest that the solutions we’ve seen so far might be too complicated for users. Sort of like the Internet before hyperlinks.

Fifth, while many vendors can’t handle open-ended RFPs, that doesn’t mean the entire system fails. It simply means that that vendor can only respond to limited RFPs. An unnamed car rental agency responded to Doc’s presentation of VRM with the certainty that they could never respond to all the varied options that people ask for. Their inventory management systems simply don’t have that kind of flexibility. However, that doesn’t mean that they couldn’t respond to those RFPs that fit within their parameters or even respond to partial hits with proposals that clearly specify their limitations. When the waiter apologizes because the restaurant is out of your favorite beer, do you leave the restaurant? Not usually. A VRM RFP needs to enable a sales conversation to take place, one that affords vendors a chance to suggest partial RFPs based on existing capability and capacity. High-end vendors who specialize in custom, high margin requests, would be free to propose a solution that meets the full RFP. Ultimately, the user selects which vendor gets the order.

Sixth, this kind of catch-22 is actually a network effect. It is true that in the beginning there won’t necessarily be a lot of users to attract vendors and that as vendors start to participate, there will be limits on the types of products and services available. This is a catch-22. However, as new vendors join, it becomes increasingly attractive to customers. And the more customers, the more attractive it becomes to vendors. And when this is delivered in an open-standard, open-network environment, the runaway tipping point can happen quickly and in many different markets at once. So, rather than complain about the catch-22, find those seed markets where vendors and customers can readily see the value, and build services to connect people with vendors. There will be early adopters. In the right markets, those adopters will trigger a network effect that catalyzes the entire marketspace, just as the World-Wide Web grew from academia to technology markets to technologists to eventual mainstream adoption.

I don’t know if Jeremy realized he was making a VRM gesture or if he would even consider his post an “RFP,” but perhaps he will see this post and think a bit about how VRM is addressing something fundamentally new, and yet, incredibly close to what we people already need.

The $64,000 question: why didn’t Jeremy just search Yahoo!?

There’s a lot of room to go in search… and NONE of the current search providers–not just Yahoo!–could have answered Doc or Jeremy’s inquiry more effectively than their blog posts. In fixing that problem lies the hope of VRM personal RFPs.

There’s been a lot said about the new Attention Economy and a lot of interest in generating value from “Attention Data.” The enthusiasm bubbles up from excitement about a new abundance to reach the conclusion that Attention is now the scarce resource that smart people will use to create wealth in our post-mass-media, post-mass-production world.

The problem starts with the excitement about a “general abundance” in our newly digital world which Wired famously referred to as The Long Boom in 1997, but we eventually realized was just the Big Bubble. The problem of the New Attention Economy then expands beyond simply a new “Economics of Abundance” with a hubristic assertion that Attention is the secret ingredient for future success, complete with calls to invent an entire new “Attention Economics.”

Unfortunately, Attention is only limited because time is limited. Its value is ephemeral at best and annoying at worst.

We each have just as much Attention in any given day as we have conscious, waking hours. It’s just a function of time. Simultaneously, Attention, once earned, has fairly lightweight value. It can be earned by a distracting but ineffective advertisement and does not in itself indicate anything other than a first-order stimulus response. Put flaming letters flying across the sky, I’ll probably look up at it and pay Attention for a moment. But that doesn’t mean I’ll do anything in response.

So Attention not only presents nothing unique in terms of scarcity, its value is intrinsically frail. That isn’t an exciting basis for a new economy.

Abe Burmesiter over at Abstract Dynamics has a different take on why there is no new “Attention Economy” (worth reading in its entirety):

Somewhere on the edge of academia circulates the idea that economics is defined as the “study of how human beings allocate scarce resources”. It’s a definition that doesn’t show up in most dictionaries, but it has a stubborn persistence…

It’s a curious distortion to make economics strictly a study of scarcity, and like the textbook chaos theory case it starts out as a rather minor disruption. Scarcity is after all essential to the generation of price and value, and economists hold those processes dear to their hearts. There is of course more to economics than just studying scarcity, but it’s not exactly an alien concept. What’s curious is what happens when non economists start latching onto the distortion, what’s curious is when scarcity meets attention from three different directions.

Michael Goldhaber, Richard Lanham and Georg Franck, all more or less independently converged on a phrase, “the economics of attention” in the past decade or so. At the core of their thought (which varies widely in quality) is the observation that in a time where information is becoming, in Goldhaber’s terms, “superabundant” what is scarce is attention…

The first irony is that if economics was really just to be about the distribution of scarce resources it wouldn’t even be about money. For money is about as far from a scarce resource as there is. It can be printed out by any government and by a skilled counterfeiter too. Or it can be generated by any group or organization with enough clout… Money is anything but scarce. The problem is not there is not enough, but that it circulates with a damaging inequality.

Goldhaber and Lanham though don’t seem really want economics to be about money anyways though. They’d much rather refocus it all around attention. It’s an act of overstatement that probably does them far more harm than good. They get to make exaggerated statements about the need for a new economics, perhaps it makes their observations seem bigger, but it also makes it far easier to ignore them. They might want it all to be about attention, but quality and accuracy still have a bit of value left in them. Someone is going to make a career pulling attention scarcity into the wider economic stream of thought, but it just wont have the extreme ramifications the attention lovers vest into it.

In short, thinking about Attention may be intriguing, but it isn’t going to be the foundation for a new economics. It isn’t going to reinvent how we do business or how we work as a society.

So, despite others’ feverish efforts to the contrary, I’ll go on the record as stating unequivocally, 2007 will not be the year of the Attention Economy. Neither will 2008, 2009 or 2010.

For my money, I’m betting on Intention, VRM, and ComplexSearch.

What do you get when you turn proprietary data silos inside out?

Users in control.

Doc Searls has been advocating VRM for a while (here too). What’s nice about his thinking — in addition to the open source/open standards approach we’d expect from a senior editor at the Linux Journal — is that he’s working the problem through the entire technological spectrum:

I don’t think VRM should be confined to a browser, either. I think this is something that should work through a cell phone, a card, or any other device or representation that works for the individual.

Not only are the vendor’s silos being turned inside out, so are the technology and network providers’.

My mindset has been stuck in the browser, perhaps with an accompanying helper application that does nice things for users, but still basically software on a personal computer. At the core, SwitchBook’s innovation is useful in larger contexts, but it won’t start out that way. Our strategy in simple:

1. Make it work with current search habits.
2. Augment IE and Firefox.
3. Expand to other OSes and browsers as quickly as possible.
4. Push the underlying API and data format as an open standard.
5. Open the tool for customization as widely as possible.
6. Open source the code for “built-in” customization

But are we going to take the time now to make sure it works in cell phones or datacards or iPods or anything other than a computer? There just isn’t enough bandwidth for that in a bootstrapping startup.

Fortunately, Doc’s VRM work as a Fellow at the Berkman Center gives him the freedom to invest in a solution of that breadth. A VRM solution that is bigger than any one company, technology, platform, or medium. Say goodbye to the silos.

It has also given me a fresh way to think about Complex Search. Much of VRM — as I understand it — is designed to be automagic. Specify your needs, receive bids from selected/qualified vendors using a tool that makes it easy to manage those relationships. But before one can specify needs, most people need to spend time discovering their needs. For all but the simplest purchases, that’s a Complex Search.

For example, take Doc’s latest VRM “Gesture”

I want a phone that is GSM-based (so it works overseas as well as in the U.S.), works across as much of the U.S. and Canada as possible (Verizon has been a disappointment in this respect), has a GPS, and has an easy-to-use UI. I don’t care about PDA functions, ringtones (I like the old Western Electric bell ring, though), or camera functions. I like keys that are easy to read and use, and an address book that’s easy to synchronize with a computer. It would be nice, for personal reasons (I work for Linux Journal), if it ran on Linux. I’d rather it not (for the same reason) run on Windows. Mostly I just want it to be a good GSM phone with a GPS. And I’m willing to let the GPS function slide, just to get a good phone.

That’s a mouthful. Doc is famous enough in the blogoverse to get feedback without the VRM infrastructure. He may not have a vendor make an offer directly (although a smart vendor would seriously consider sponsoring Doc), but he’ll probably get enough direction from peers to narrow down his vendor choices. With a fully operating VRM, the fulfilment side of that gesture will be streamlined and automated so that any vendor who wants to can cost-effectively make Doc a competitive offer, perhaps even a bundled package that leverages their unique value-add. That will take a lot of work, but the potential value to everyone in the transaction is clear.

Compare that to the broad, politicized, unfocused brush strokes of the AttentionTrust and you can see why I think the AttentionTrust goals are still too blurry and ambiguous to generate much success. VRM is working with Intention. It is highly focused. Its output is clear. The benefit to users and vendors is evident. AttentionTrust is stuck thinking about everything, all the time, and only online, then mashing that into some anonymized goulash from which magic is supposed to emerge. Bah humbug. I’ll believe it when I see it.

I think Doc is on to something, though. The Internet so radically drops the costs of so many different modes of communication, it will continue to restructure our society for another couple of decades, at least. Most of the success to date has been based on one-to-many marketplaces, such as Amazon or many-<aggregated-as-one>-to-many marketplaces such as eBay. VRM lets us create inverted “many-to-one” markets. Markets of one. Make your gesture, create a market. That’s powerful.

And yet, Doc’s gesture — as every request for bids must — also contains a treasure trove in the form of Doc’s requirements, a wealth of needs Doc learned the hard way. He’s a power user with heavy demands and he pushes technology to its limits. He is fed up with his current options and, having experimented enough, he knows just what wants. But he’s lucky to have that experience. Most people have no idea what the deciding factors could or should be for the products they want to buy. (Can you say megapixel?) Doc is anything but a typical consumer.

Consider what it was like when the web started taking off in 1994/5/6. At that time, I was out selling Internet marketing services and helping companies figure out what to do online. Overwhelmingly, time and again, smart, capable, professional people asked “How much does a website cost?” Well, what kind of website do you want? Their question was inherently non-sensical, but people didn’t understand that yet.

First, you have to figure out what you want, then, and only then, can you send out an RFP to get bids on it. Sure, you scale your RFP based on what your budget is — and unless you have deep pockets, it pays to be prudent in what you include in your request — but at the end of the day, only a detailed specification provides enough direction for vendors to submit a bid. The result of these conversations was often a small strategy and/or requirements engineering contract to distill their needs into just such an RFP.

So how does that work with VRM? How do people develop enough expertise and understanding of their needs so they can present a request like Docs? How does VRM work for regular folk?

In short, they search. They explore. They learn.

From friends. By reading reviews. Going to various manufacturer’s and vendor’s websites. By learning from people like Doc, either through blogs, reviews at CNET or ThisNext, pricing at PriceGrabber, Google, or through direct conversations. By trying out products. Even from advertising and retail stores. I happened to learn about Verizon’s data services in the Verizon store. Imagine that.

This is Complex Search. People aren’t going to rely on any one vendor or reference point, unless they have an absolutely trusted guide like a brother or daughter or college roommate to point them in the right direction. They are going to check out different sources, browse multiple websites, collate and corollate a lot of information from a lot of different places. Then, after they have searched and narrowed their needs down to the details, they can put it in the form of a digital RFP and see the power of VRM kick in. Zing! A Market of One.

VRM is still evolving. Questions and answers of many varieties must work their way through the community, from people’s and companies’ needs to draft technological frameworks, APIs, protocols, and working code. Good stuff.

Somewhere in there, I’m confident Complex Search will meet VRM and lots of real value will be created for people, vendors, and innovators alike.

Doc will be at the Identity Workshop in early December to discuss VRM and Identity with all comers. It should be a great opportunity to figure out where VRM is headed and how we can contribute. I hope you can make it.

In a conversation about the potential market lock-in of Google at Abe Burmeister’s blog, Dave Chiu introduced me to a great presentation Seth Godin made to Google early in 2006, explaining that it was marketing, and not technology, that made Google the market leader. I couldn’t agree more, even though most of my technology friends will swear it was all about the quality of PageRank.

Tests show that when Yahoo! & Google results are formatted identically, users can’t tell the difference. And yet, Google matters to people. They matter in a deeply personal way. They have created a powerhouse brand because better technology gave them an opportunity to market to the masses and that marketing worked.

Seth puts this in his framework of remarkable stories. His bestselling books The Purple Cow and All Marketers are Liars discuss this in much greater detail, with lots of anecdotes, examples, and advice on how to improve marketing through remarkable stories. Good stuff.

What he doesn’t talk about is how you create the right stories to tell. He misses that point in his books as well. But that’s ok. Stories are powerful marketing tools. That’s an important enough message by itself. In the presentation he also does a great job pointing out the anticipatory and experiential value of a brand–that value people get just because they buy the brand, independent of the actual value of the product. The driver for this type of value is story, especially when the brand connects with people’s identity in profound ways.

He then goes on to outline his view of Fashion/Permission marketing that is uniquely enabled by the Internet as a one-to-one disintermediated medium. He exhorts Google to create a permission tool that gets users to invite Google into a deeper relationship, one that gives Google more context and more details about what users are really looking for. In other words, leveraging the brand to enhance the technology by meeting users needs in a more meaningful way, which of course will only enhance the brand further. Great stuff. Note to Google: possible areas for development: VRM and Complex Search.

It is worth watching, if only to see the advice one of the hottest minds in marketing gives to the most influention Internet company on the planet.

Curiously, Seth missed the opportunity to explain to Google that their haphazard development strategy is steering them, inexorably, away from the branding that made them the market leader: the promise of making the Internet simple.