The details of what happens with the information we share is often hidden behind long, complicated legal agreements that almost no one reads. If we’re lucky, they are explained in Terms of Service and Privacy Policy documents, sometimes buried out of view, other times forced on us like ransom notes forcing us to state our compliance or leave the site.

It doesn’t have to be that way.

Today, at the Internet Identity Workshop, we officially launch the Standard Information Sharing Label, which makes it easy for websites to say in simple, consistent language what they do with our information, making it easier for individuals to make better decisions about the information we share online.

The Information Sharing Work Group has published a draft specification defining the Standard Label as well as a Kickstarter project to finance its graphic design.

The Kickstarter has a brief video explaining the effort. The official press release is here.

The work is free to use and open to collaborators.

In all my years contributing to the VRM conversation, few projects have made me as proud as I am of the work behind the Standard Label.

Check it out. If you like it, please spread the word and consider chipping into help take this work to the next level.

Normally, I wouldn’t nitpick about this, but there are two key domains where this is vital and I’m knee deep in both: contracts and platforms.

Doc said:

Like, is the customer always the first party and the vendor the second party?

Well, no. So, some clarification.

First and second parties are like the first and second person voices in speech. The person speaking is the first person, and uses the first person voice (I, me, mine, myself). The person being addressed is the second person, and is addressed in the second person voice (you, your, yourself).

And

To sum it up, third parties mostly assist vendors. That is, they show up as helpers to vendors.

The first point is great, and if you continue this further (and make the leap from parties to data providers), you get something like this:

The ownership of “your” and “my” data is usually clear. However, ownership of the different types of “our” data is a challenge at best.  To complicate matters further, every instance of “my data” is somebody else’s “your data”. In every case, there is this mutually reciprocal relationship between us and them. In the VRM case, we usually think of the individual as owning “my data” and the vendor as owning “your data”, but for the vendor, the reverse is true: to them their data is “my data” and the individual’s data is “your data”. Similar dynamics occur when the other party is an individual. I bring my data, you bring your data, and together we’ll engage with “our” data. We need an approach that respects and applies to everyone’s data, you, me, them, everybody..

Which is from my post on data ownership. The trick is that 1st party and 2nd party perspectives are symmetrical.  We are their 2nd party and they are their 1st party. Whatever solution we come up with in the VRM world needs to work for everyone as their own 1st party. Everyone. Including “them”. Including Vendors.

In fact, that’s the only way we can get out of the client-server, subservient mentality of the web. It’s also the only way to make sure that our solutions work even when the “vendor” is our neighbor, our friend, or our family.

This is particularly clear in the work we are doing at the Kantara Initiative’s Information Sharing Work Group. We are creating a legal framework for protecting information individuals share with service providers. As such, it’s vital that the potential ambiguities of language are anchored in rigorous definitions. And what has emerged is that every transaction is covered by a contract between two parties. Not three. Not four. Not one. Two. And to the extent that third (or fourth) parties are mentioned, they are outsiders and not party to the contract. Since we are building a Trust Framework, there is a suite of contracts covering the different relationships in the system, but the legal obligations assumed in each contract have clear and unambiguous commitments between the first and second parties only.

Platforms

But where I think where Doc’s framing most needs a bit of correction is that, in fact, historically, third parties are never presumed to be working for second party. Not in the vernacular and not in any legal context. This presumption only emerges once you add a Fourth Party claiming that it works on behalf of the user. That is, 3rd-party-as-ally-of-the-2nd-Party is a corollary to Fourth Party concept, not a foundation for explaining it.

Take Skype, which I have on my Verizon cell phone. In the contract with Verizon, Skype is a third party application and Skype, Inc. is the third party.  But Skype isn’t working on Verizon’s behalf.

This is not only true in the sense of 3rd party applications whose value proposition is clearly at odds with the 2nd party, it is even more true when it comes to platforms. And especially when you consider the relevance of VRM as a platform for innovation.

In every platform, there are third parties who create apps that run on the platform. Microsoft built Windows, but Adobe built Photoshop. Apple built the iPhone, but Skype built Skype.  For platforms to be successful, they necessarily bring in 3rd party developers to build on top of the platform. These developers aren’t necessarily working on behalf of the platform provider, and it would be a miscarriage of alignment to claim that they are. They are out for themselves, usually by providing unique value to the end user. Some new widget that makes live better.

This becomes even more true when you are dealing with open platforms, or what I called Level 4 Platforms (building on Marc Andreeson’s The 3 Platforms You Meet on the Internet). In open platforms, you actually have 3rd parties helping contribute to the code base of the platform itself.  Netscape adds tables to HTML. Microsoft adds the <marquee> tag.  But here, it is even crazier to imagine that these 3rd parties are acting on behalf of the platform party… because there really isn’t a platform party. Nobody owns the Internet.

I think the right way to think about 4th Parties is that they have a fiduciary responsibility to the 1st party and 3rd parties may or may not.

Fourth Parties answer to the 1st party.

3rd Parties may not answer to anyone.

Personal Data Stores

Personal data stores allow individuals to share online data with service providers. Facebook Connect users can give third-party web sites like Digg, Amazon, and YouTube access to information stored at Facebook, turning Facebook into a personal data store for over 500 million people.

What makes personal data stores special is the seamless sharing with websites for real-time personalization of the web. It’s more than just file back-up or synchronization.  It’s not just publishing “content” to our friends or the public. Personal data stores allow us to bring our information to websites when we want to. It’s a way to treat the user as the point of integration.

Personal data stores can be anywhere, shared with websites whenever we want. Consider giving FedexKinko‘s a link to a Flickr account so they can download photos to print a new calendar. Or giving a new doctor permission to access our personal health history rather than filling out a paper form while we sit in the waiting room. Or giving a website access to our Outlook contact list on our desktop computer so they can give us birthday reminders and gift suggestions. The key is user-managed access, wherever the data lives. Facebook Connect gives this kind of access control over all the data we store at Facebook, enabling web-wide personalization built around the individual.

Mash-ups

In recent years, mash-ups and real-time APIs have made it easier and easier for companies to combine information from different services into a single user experience. Instead of building bigger and more complicated proprietary data silos, companies take advantage of services like Google Maps and IP-address geolocation, using real-time information to enhance their websites.

Some service are even built around other companies’ data: Twitter clients like Seesmic and Tweetdeck, which access our Twitter data on our behalf; Trillian, which works with various instant messaging networks; and Mint, which pulls in our financial data from hundreds of websites. The “real-time web” is constructed on the fly, using linked data and real-time APIs to dynamically customize services for each of us.

Personal data stores let us bring our own data to the mash-up party. Not only do we have better control over who sees what, we can provide more timely, higher quality data than service providers can get from other sources. Effective integration with personal data stores means no more ads for that car we’ve already bought; no more recommendations based on false assumptions. Unfortunately, data in the wild is constantly becoming outdated, miscopied, and misconstrued, because that’s the best companies can do using the billions of dollars worth of proprietary data that’s gathered about us rather than provided by us. Personal data stores easily allow individuals to give the most relevant, most up-to-date information to just those companies we want to do business with. That means not just better data, but more intimate relationships with our favorite companies and organizations.

Perhaps the most liberating aspect of personal data stores is that everyone gets to have as many as we want. We all have our favorite websites for different online activities. As those sites open up their data with a user-driven permissions mechanism, they become personal data stores. So, whether it’s YouTube for videos, Flickr for Photos, Foursquare for location updates, TripIt for travel plans, or RunKeeper for exercise data, we get to bring our best data with us wherever we go. Savvy websites pull in this high quality data to personalize our visits, while those with unique data open it up for use elsewhere to maximize value to their users, which is exactly what Facebook is doing with Facebook Connect.

Facebook Connect

Facebook Connect makes this kind of access simple for everyone, with industry changing adoption rates. Over 66% of the top 100 websites and over 1 million total websites now integrate with Facebook in some way. Nearly 1/3 of Facebook users—over 150 million people—use Facebook Connect every month. Every time we do, we give websites access to information stored in our Facebook accounts, such as our name, gender, names of our friends, and all the posts currently on our wall or posted by us. It’s an archetypal personal data store, with highly credible and timely data in the form of our friend list and our status updates. Sure, Facebook Connect is still far too limited in the amount of information we can store and we lack control over how that information gets used… but architecturally, Facebook has changed the game for a vast portion of the World Wide Web.

To find out what information Facebook is sharing, I built a website called “I Shared What?!?“, an information sharing simulator for Facebook. The site uses javascript and Facebook Connect to display everything it can get from Facebook. Visitors see in specific detail exactly what they share when hitting the “allow” button in the Facebook Connect permissions dialog.

Facebook uses open standard technology to bring mash-ups to a new level, built on information provided directly by the user, in real-time, with minimal fuss or bother. There are shortcomings, of course. A lot of them, but I’ll save those for future posts. For now, think of Facebook as the 800 pound icebreaker of a new way for companies to connect with their customers.

To this veteran VRM evangelist, Facebook has done more in 2010 to usher in the era of the personal data store than anyone, ever. In one fell swoop, Facebook launched a World Wide Web built around the individual instead of websites, introducing the personal data store to 500 million people and over one million websites.

Unexpectedly, Facebook has moved VRM from a conversation about envisioning a future to one about deployed services with real users, being adopted by real companies, today. We still have a lot of work to do to figure out how to make this all work right—legally, financially, technically—but it’s illuminating and inspiring to see the successes and failures of real, widely-deployed services. Seeing what Amazon or Rotten Tomatos or Pandora do with information from a real personal data store moves the conversation forward in ways no theoretical argument can.

There remain significant privacy issues and far too much proprietary lock-in, but for the first time, we can point to a mainstream service and say “Like that!  That’s what we’ve been talking about. But different!”

The term “ownership” simply brings too much baggage from the physical world, suggesting a win-lose, us-verses-them mentality that retards the development of rich, powerful services based on shared information.

Anyone up for sacred cow cheeseburgers?

I’m a member–and a big fan–of Steve Holcombe‘s “Data Ownership in the Cloud” LinkedIn group and I love the efforts of the Dataportability guys and am a big supporter of the Privacy and Public Policy work group at Kantara. There is a lot of good work being done by folks trying to figure out how to give people greater control over the use of data about them (privacy) and gain access to data they use or created (dataportability).

Unfortunately, sometimes the arguments behind these efforts are based on who owns–or who should own–the data. This is not just an intellectual debate or political rallying call, it often undermines our common efforts to build a better system.

Consider this:

1. Privacy as secrecy is dead
2. Data sharing is data copying
3. Transaction data has dual ownership
4. Yours, mine, & ours: Reality is complicated
5. Taking back ownership is confrontational

Privacy as secrecy is dead

First, the data is pretty much already out there. The issue isn’t “How do we keep data from bad people,” it’s “How do we keep people from doing bad things with data?” DRM and crypto and related technology as the sole means to prevent data leakage and data abuse are failures. Sooner or later, the bad guys break the system and get the data.  Sure, there are smart things we can do to protect ourselves. Just like we wear seatbelts and lock our front doors, we should also use SSL and multi-factor authentication, but we can’t count on technology to keep our secrets. We need solutions that work even when the secret is out.

In fact, privacy isn’t about information we keep secret. It is about information we have revealed to someone else with expectation of discretion, e.g., when we tell our doctor about our sexual activities. It’s no longer a secret from the Doctor, but because it is private, we have rules that keep the information from being used inappropriately. Most of the time, with most doctors, it works. Those few who break those rules are dealt with through legal means, both civil and criminal, as well as social approbation. So, because we inherently need to release data to different parties at different times, we can’t control it through secrecy alone. Instead, we need to build a framework for preventing abuse when others do have access to sensitive information. Like in the case with our doctor, we want our service providers to have the data they need to provide the highest quality services.

Data sharing is data copying

Second, in the world of atoms, there can only be one of a thing, which is the reverse of the world of bits. With atoms, even if there are copies, each copy is itself a singular thing. Selling, transferring, or stealing a thing precludes the original owner from continuing to use it.

This isn’t true for information, which can easily be sold, transfered, and stolen without disturbing the original version. In fact, the entire Internet is basically a copy machine, copying IP packets from router to router, as we “send” images, web pages, and emails from user to user and machine to machine–each time a new copy is created whether or not the originating copy is deleted. To think of bits as if they were ownable property leads to attempted solutions like DRM that try to technologically prevent access to the information within the data, which is only good until the first hacker cracks the code and distributes it themselves. Instead, if we build social and legal controls on use, we can give information more freely, but under terms set by each individual when they share that information. Enforced by social and legal rather than purely technological means, this makes the most of the low marginal cost of distributing  online, while retaining control for contributors.

Transaction data has dual ownership

Image via Wikipedia

Third, much interesting data is actually mutually owned… which means the other guy can already do whatever the heck they want with it.  Consider web attention data, the stream of digital crumbs representing the websites we’ve visited and any interactions at each: all our purchases, all our blog posts, all our searches. Everything. Some folks argue that we own that data and therefore have the right to control the use of it. But so too do the owners of the websites we’ve been visiting. We don’t own our http log entries at Amazon. Amazon does. In fact, in every instance where two parties interact, where we engage in some transaction with someone else, both parties are co-creating that information. As such, both parties own it. So, if we tie the issue of control to ownership, then we’ve already lost the battle, because every service provider has solid claims to ownership over the information stored in their log files, just as we, as individuals, own the browsing history stored on our hard drive by Firefox, Internet Explorer and Chrome.

In the movie Fast Times at Ridgemont High, in a confrontation with Mr. Hand, Spicoli argues “If I’m here and you’re here, doesn’t that make it our time?”  Just like the time shared between Spicoli and Mr. Hand, the information created by visiting a website is co-created and co-owned by both the visitor and the website.  Every single interaction between two endpoints on the web generates at least two owners of the underlying data.

This is not a minor issue. The courts have already ruled that if an email is stored for any period of time on a server, the owner of that server has a right to read the email.  So, when “my” email is out there at Gmail or AOL or on our company’s servers, know that it is also, legally, factually, and functionally, already their data.

Yours, mine, & ours: Reality is complicated

Fourth, when two parties come together for any reason, each brings their own data to the exchange. We need a framework that can handle that. Iain Henderson breaks down this complexity in a blog post about your data, my data, and our data, talking about an individual doing business with a vendor, for example, someone buying a car.

“My data” means data that I, as an individual have that is related to the transaction. It could include the kind of car I’m looking for, my budget, and estimates of my spouse’s requirements to approve of a new purchase.

“Your data” means data that the car dealer knows, including the actual cost of the vehicle, the number of units in inventory, the pace of sales, current buzz from other dealers.

“Our Data” means information that both parties have in common. That could be Shared Information, explicitly given by one party to the other in the course of the deal, such as a social security number so the dealer could run a credit check. It could be Mutual Information, generated by the very act of the transaction, such as the final sale price of the vehicle. Or, it could be Overlapping Information, which each party happens to know independently, such as the Manufacturer Suggested Retail Price (MSRP) of a vehicle (which we found online before heading to the dealership).

The ownership of “your” and “my” data is usually clear. However, ownership of the different types of “our” data is a challenge at best.  To complicate matters further, every instance of “my data” is somebody else’s “your data”. In every case, there is this mutually reciprocal relationship between us and them. In the VRM case, we usually think of the individual as owning “my data” and the vendor as owning “your data”, but for the vendor, the reverse is true: to them their data is “my data” and the individual’s data is “your data”. Similar dynamics occur when the other party is an individual. I bring my data, you bring your data, and together we’ll engage with “our” data. We need an approach that respects and applies to everyone’s data, you, me, them, everybody.

In these complex Venn diagrams of ownership, it is more important who controls the data than who owns it.  We’ve already lost the crudest form of control–secrecy–and we are going to continue to lose more as we opt-in to seductive new services based on divulging more and more information: our purchase history, browsing activity, and real-world location data. But we still need to control how all this data is used, to protect our own interests while still enjoying the benefits of the great big copy machine that is the Internet.

Taking back ownership is confrontational

© Regien Paassen | Dreamstime.com

Fifth, we don’t need to pick a fight to change the game. There is a lot of data out there that many of us believe we should have control over. I agree. A lot of people argue that we should have the right to exclude other people’s use because we own the data, because it’s ours in some legal, moral, or ethical framework. The problem is, those other people already have it, and they also believe that they are legitimate owners. In fact, many of them paid for that data, buying it from data aggregators who compile all sorts of things about people, from both public and private sources. This entire ecosystem of customer data is a multi-billion dollar business and every single player “owns” the data they are working with. So if we focus our energy in claiming ownership over that same data in order to take control, we are framing the conversation as a fight, a fight against a powerful, well-healed, well-funded, entrenched bunch of opponents.

Most of these “opponents” are the very people we are trying to win over to our way of thinking. These are the vendors we want to embrace a new way to do business. These are the technologists we want to transform their proven, value-generating CRM systems to work with our data on our terms, instead of their data on their terms. Arguing over ownership puts these potential allies on the defensive, when what we really want is their collaboration.

From Ownership to Authority, Rights, and Responsibilities

Rather than building a regime based on data ownership, I believe we would be better served by building one based on authority, rights, and responsibilities. That is, based on Information Sharing.

• Who has the authority to control access and use of particular information?
• What rights does a party have in using and distributing a piece of information?
• What responsibilities does an information user have to others with respect to that information?

Let’s stop arguing about who owns what and start figuring out how we can share information in ways that allow everyone to win.

When we collect all of our information into a single conceptual repository, and then share access to it with service providers on our own terms, we create a high quality, highly relevant, curated personal data store. This allows us to bootstrap a control regime over all of our data in a way that creates new value for us and for our service providers. Now, instead of iTunes Genius or a Last.FM scrobbler only having access to our media use with their service, they can provide recommendations based on all the information stored in our personal audio data store. We get better recommendations and they get better data to drive their services. This personal data store is entirely under the authority of the user, sharing information with service providers according to specific rights and responsibilities.

The Information Sharing approach neatly sidesteps the complexities involved in privacy and dataportability issues of the information already known by service providers. These remain serious issues, worth addressing. Resolving them will require long term investment in the legal, regulatory, moral, and political systems that govern our society. Fortunately, sharing the information in our personal data store can begin almost immediately once we have working specifications.

This controlled sharing of information will dramatically increase our comfort level when revealing our intentions and interests. We would have control over the use–and would be able to prevent abuse–of that information, while making it easy for service providers to improve our lives in countless ways.

At the Information Sharing Work Group at the Kantara Initiative, Iain Henderson and I are leading a conversation to create a framework for sharing information with service providers, online and off. We are coordinating with folks involved in privacy and dataportability and distinguish our effort by focusing on new information, information created for the purposes of sharing with others to enable a better service experience. Our goal is to create the technical and legal framework for Information Sharing that both protects the individual and enables new services built on previously unshared and unsharable information. In short, we are setting aside the questions of data ownership and focusing on the means for individuals to control that magical, digital pixie dust we sprinkle across every website we visit.

Image by hegarty_david via Flickr

Because the fact is, we want to share information. We want Google to know what we are searching for. We want Orbitz to know where we want to fly. We want Cars.com to know the kind of car we are looking for.

We just don’t want that information to be abused. We don’t want to be spammed, telemarketed, and adverblasted to death. We don’t want companies stockpiling vast data warehouses of personal information outside of our control. We don’t want to be exploited by corporations leveraging asymmetric power to force us to divulge and relinquish control over our addresses, dates of birth, and the names of our friends and family.

What we want is to share our information, on our terms. We want to protect our interests and enable service providers to do truly amazing things for us and on our behalf. This is the promise of the digital age: fabulous new services, under the guidance and control of each of us, individually.

And that is precisely what Information Sharing work group at Kantara is enabling.

The work is a continuation of several years of collaboration with Doc Searls and others at ProjectVRM. We’re building on the principles and conversations of Vendor Relationship Management and User Driven Services to create an industry standard for a legal and technical solution to individually-driven Information Sharing.

Our work group, like all Kantara work groups, is open to all contributors–and non-contributing participants–at no cost.  I invite everyone interested in helping create a user-driven world to join us.

It should be an exciting future.

This material is based upon work supported by the National Science Foundation under Award Number IIP-08488990. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author and do not necessarily reflect the views of the National Science Foundation.

From cable TV to YouTube, from newspapers to blogs, from Wal-Mart to eBay, from Ma Bell to the Internet, the shift from centralized, structured systems of authority to emergent, collaborations between individuals has been reshaping our political, social, and economic world for generations. This is a trend that has driven—and been driven by—the massive success of the Internet, email, the World Wide Web, eBay, Google, RSS, FaceBook, YouTube, and Twitter. Each of these examples took an existing model and made it more user driven: networking, messaging, electronic publishing, buying & selling, content discovery & advertising, news aggregation/syndication, online video, status updates.

The conclusion: companies which find ways to be more user driven are more valuable, more profitable, and more successful.

What does it mean to be “user driven”? At its most basic, it means putting the user in charge, in some way. Fully realized, it means putting the user at the center of the system, as a point of integration, origination, and control. We call these fully realized systems “User Driven Services”.

User Driven Services put users in charge. Users start each interaction, manage the flow of the experience, and control what and how data is captured, used and propagated.  Users are the cause and the controller, working with service providers to co-create collaborations that create value for all parties.

From self-serve gas stations and soda fountains to ATMs and self-checkout grocery stores, companies have been putting users in charge of different aspects of their services for years. With GetSatisfaction—which allows users to self-organize for cooperative customer support—and Facebook—which provides social context for user-generated content—users are not just self-servicing, they provide the core content behind the user experience. Now, through user-centric Identity and API access to most popular online services (Flickr, Twitter, Facebook, etc.), users can direct which parts of their experience are serviced by which providers, allowing unprecedented realtime flexibility in service creation.

User Driven Services are redefining how we interact, how we manage our businesses, and how we engage in both public and personal conversations. Businesses and organizations that want to thrive in this new reality would do well to help co-create a new mutually beneficial marketplace for products, services, and ideas. Individuals, participating in this rising tide of personal power, have an opportunity to coordinate with each other and with service providers to craft a future that meets all of our needs, as individuals, entrepreneurs and business people.

Terminology

A few key terms:

System: a group of independent but interrelated elements engineered to operate as a unified whole.

(The systems to which we refer are not natural or conceptual systems, but rather, operating mechanisms designed and implemented to perform intended functions.)

User: any individual interacting with a system.

Service: a value generating experience available to users through interactions with a system; also the system providing such experiences.

User Driven Services: services that maximize value creation by maximizing user control and authority.

Characteristics

User Driven Services have the following characteristics:

1. Impulse from the User
2. Control
3. Transparency
4. Data Portability
5. Service Endpoint Portability
6. Self Hosting
7. User Generativity
8. Improvability
9. Self-managed Identity
10. Duty of Care

We will explore each of these characteristics in a series of articles over the next few weeks.

I’ve mentioned Google Notebook briefly in the past, as a tool for helping with user-driven searches (more) — or complex searches as I used to call them. Unfortunately, Google never connected the notebook with Search, despite it being a reasonable solution for keeping track of the kind of discoveries you find when doing advanced searches at a lot of different websites.

Instead, Google suggests you try one of their other products:

If you haven’t used Notebook in the past, we invite you to explore the other Google products that offer Notebook-like functionality. Here are a few examples, all of which are being actively improved and should meet your needs:

• SearchWiki – We recently launched a feature on Search that will let you re-rank, comment, and personalize your search results. This is useful when you’ve found some results on Google Search that were really perfect for your query. You can read about how to use SearchWiki in this blog post.

• Google Docs – If you’re trying to jot down some quick notes, or create a document that you can share with others, check out Google Docs.

• Tasks in Gmail – For a lightweight way to generate a todo list or keep track of things, we recently launched Tasks in Gmail Labs.

• Google Bookmarks – For a tool that can help you remember web pages that you liked and access them easily, take a look at Google Bookmarks. You can even add labels to your bookmarks to better organize and revisit them.

Sigh.

Google Notebook fit a unique spot in the Google product portfolio, and as you can see in the comments on the announcement, a lot of people will miss it.

It’s too bad we don’t have our Search organizer product ready, I’d love to swoop in and save the day for all those wayward soles stuck without their Google Notebook. The future holds promise… Still, there something to be gleaned from Google’s recent developments. As I’ve said before, Search is bigger than query/response. And at least some parts of Google know it. But I wonder how much of the rest of the company gets it.

Take SearchWiki for example. Google rolled this out in November of last year. If you use Google through a Google account, SearchWiki gives you three new icons on every result:

• promote
• delete
• comment

Promoting an item moves it to the top of the result list. Delete, predictably, removes it from the results and Comment adds a comment to that result.  The first two are private–only you see the effect of promotions and deletions, while comments are public.

It is an interesting experiment, if only because it shows how seriously Google takes Wikipedia as competition; the functionality is nearly identical to Wikipedia’s search engine, Wikia. (And many of us have noticed how often Wikipedia entries show up early in Google results.)

It also shows a growing belief that users can help improve Search if they are actively involved. We don’t know what Google is doing with all the user data of who deleted or promoted what, but it will be fun to watch and find out. It will certainly present a different reference frame than PageRank–the core algorithm behind Google–which focuses entirely on the authority of HTML authors and the hyperlinks they put in web pages. If they shift the focus of their ranking to the actions of every day users, Google would shift the moral authority behind their results from web page creators to web page visitors–a much more representative population. That’d be pretty cool.

Unfortunately, the problem with SearchWiki is that it pivots on keywords rather than a more durable concept of Search. It turns out that the promotions and deletions apply only to subsequent queries with the exact same keywords.

Seriously.

For example, let’s say you search Google for “travel” and delete Travelocity, Expedia, and CheapTickets, because you’ve already tried those sites and are looking for something new. Then after browsing a bit, you realize you want to see websites for air travel, so you change the query to “air travel”. Suprise!  All those results you deleted are back in the list.

This is more than useless, it makes you feel like all that effort to promote and delete was completely wasted. We know that keywords evolve during advanced Searches. As we explore more of the web, we learn more about what we are looking for and which keywords might work better. And yet, Google’s SearchWiki remains fixated on the keyword query as the central point for tracking user feedback for these kinds of advanced searches–because really, who is going to promote and delete results for one-off, simple searches like finding the phone number for a restaurant? SearchWiki seems like it should be most useful for Searches that take us to dozens and dozens of websites, over days, weeks, even months. And yet, its focus on keywords to track promotions, deletions and comments means SearchWiki is practically useless for anything but the most simplistic queries.

What advanced Searches call for is a tool that helps users track a specific Search across the entire web. One that tracks both explicit and implicit data about the search, lets users organize that data on their own terms, and then lets them share that data with anyone that might be able to help their Search. This combination of keyword queries, clickthroughs, and web captures would be an invaluable representation of their Search Intent. When captured on the user’s behalf, it is a great example of the VRM concept of the user as the point of integration. At SwitchBook, we call the resulting document a Search Map.

Search Maps put the user in charge of all the data related to their Search. Search Maps enable true  user-driven searches (more), where the individual’s Search intent is effortlessly created, easily managed, and expressed to precisely those who can help the most. It is co-created with the user, with full transparency and editability. It allows a complete view of a particular search, organized and confirmed by the user.  It can be sent to any online service that can explicitly acknowledge the user’s own Terms of Use, specifying just exactly how that data can be used. The result is a verified, accurate representation of what the user is looking for right now, ready to be used by any Recommendation Provider capable of respecting the user’s data rights and then responding intelligently to the content of that Search Map.

Search Maps are at the core of SwitchBook’s approach to User-driven Search. We’re working with Doc Searls and the VRM community to explore how Search Maps work, how they can meet the needs of users, and how they can appropriately protect users’ privacy and interests when used to manage and express Search intent.

We hope to discuss this more at the Spring 2009 VRM Workshop, tentatively scheduled for March 2nd, 3rd, and 4th, somewhere on the West Coast. If User-driven Search intrigues you, save the date and look for future announcements on the VRM discussion list. We’d love to see you there.

Whether it’s user-generated content like YouTube, user-written and edited knowledgebases, like Wikipedia and Freebase, or user-centric Identity like OpenID and Information Cards, user-driven thinking is transforming our world. With VRM– Vendor Relationship Management–that revolution reaches the market, creating tools for individuals to get more value out of their relationships with Vendors. The goal is to create a user-driven market, where individuals engage with vendors on their own terms, creating mutually beneficial relationships that generate new value for everyone involved.

So what would it mean to apply user-driven thinking to Search? Traditional search is a mix of user-driven and vendor-centrism. While users can enter any query and be directed to content anywhere on the ‘net, we can’t share our search history with Search Providers of choice, nor do we have control over how our activities are tracked and utilized. There are few, if any, open standards for the searcher side of the experience and few options for moving beyond traditional query-response Search.

At the VRM Workshop 2008, we fleshed out some ideas, building on the thoughts introduced in my previous post, as well as ideas discussed at VRM 2008 in Munich and IIW2008a in Mountain View. What I love about the conversations at these unconferences is that they are so rich, literally creating value on a moment-to-moment basis. And these were no exception.

Here’s what has emerged so far regarding User-driven Search.

1. User Driven Search is bigger than query/response.

User Driven Search is more than what we type into the query box and the results we get back from Search Engines. It covers an entire set of activities that span the Internet, including searches entered at site-specific Search Providers like Expedia, the USPTO, and Circuit City, and all the web pages we visit in-between. It is inherently cross-silo—even non-silo—as it encompasses all of our online efforts around a given Search topic.

A recent Google/Comscore study found that the average Travel searcher takes 29 days from their first query until their first online purchase. These advanced Searches don’t take place all at one Search Provider nor do they usually happen all in one sitting. Users need tools that empower them to manage these advanced, multi-site, multi-session Searches.

2. Users should be able to activate and deactivate Search and tracking easily and at will

With User-driven Search facilitating advanced searches across the entire scope of our online activity, users need to be able to turn it on and off at will. Sometimes we want help and are willing to share to get it. Other times, privacy is preferred. We need to be able to turn off the surveillance and just do our thing. Unfortunately, traditional search and advertising networks don’t let us do that in any reasonable way.

There are ways to disable Doubleclick’s tracking and we can tell Google to stop personalizing our search results—if we also turn off our Search History. Yet most people have no idea this is possible and even more aren’t technically comfortable enough to mess with cookies or custom preferences. We shouldn’t have to jump through hoops to disable tracking, because if that’s the case, the vast majority of users will simply not do it, and even those who do will often opt-out completely, which means there really isn’t any choice at all. The decision shouldn’t be between using advanced search features or being treated like a digital transient. We should be able to get advanced features just when we want them and simply turn them off when we don’t. That choice needs to be transparently obvious and easy and available right in the Search interface.

3. Compartmentalization

When treating Searches that span more than single queries, users need to be able to separate them into their natural topical breakdown, in whatever way makes sense. Collecting our entire search history and/or clickstream into a single attention datastore literally destroys the context that makes the Searches relevant.

Users need a way to collect their Search-related activities into categories that make sense for them. We’d like to keep our summer vacation search activity together, yet separate from our financial planning Search. We’d like to collect our home buying search activity and store that in a different place than the queries and discoveries related to our child’s Search for information about George Washington. User-driven Search must deal with more than query/response and yet not so much that it encompasses our entire attention stream. It must capture the sweet spot of user-defined collections at a scale suitable to each Search individually, as determined by each searcher.

4. Visability and Editability

For users to drive Search, we need to be able to see and edit the all of the information used to provide results. Hidden or unauthorized data or tracking of our clickstream allow current Search Providers and advertising networks to analyze and guesstimate what we are looking for, but they don’t provide any way for us to contribute. Not only are they hiding in my virtual closet surveilling me—often without permission—they are missing a great opportunity to simply ask me what I want. By making all Search activity visible, Providers can say “Here’s the data we are using to try to help you.” By making that editable, they add “Can you help us improve it?” User interface challenges aside, there is no reason Search Providers shouldn’t ask for feedback and input. It is guaranteed to improve the quality of their view and ultimately their Search results.

Currently, Google, and its DoubleClick division, track your entire search history and just about anywhere you might go online, yet you have no idea what information they have on you, except for Google’s Search History—and you certainly can’t edit it. So when you track something down on a lark, or someone else uses your machine, irrelevant data gets bundled into your history, only to clog up the machinery that is actually trying to help you. Buy a book on knots for your young cousin and Amazon will be recommending Boy Scout titles for months. This is sometimes referred to as the “Tivo thinks I’m gay” problem. If users have neither visibility nor control over the data used for recommendations, they can’t correct these types of errors. We must have both visibility into the data driving advertising and search results, and we must be able to edit it as well.

5. Selectable disclosure on users’ terms

Having gone to the trouble to coordinate and maintain a collection of data for their Searches, users should be empowered to share that data with any service capable of responding intelligently. Search is a fundamental part of how we navigate the web; it makes no sense to restrict Search activity to any one provider. Just as your Search might take you to dozens of websites, it is also possible that it will bring you to dozens of Search Providers, from Google and Yahoo! to Amazon and eBay, even to microSearch Providers like Circuit City or Schwab. As users navigate across the web, their Search should go with them, seamlessly disclosed to authorized Search Providers as easily as possible.

Today, Google serves as a locked-in data silo for most people’s search history. There’s no way to send that history to Yahoo! Or MSN Live or Amazon or eBay to see what they might be able to do for you. As technologies for personalized search results improve, the value of that search history will continue to increase. We need to be able to send select parts of our search to providers of choice and we need to be able to do it trivially. As easily as we go from one website to another, we should be able to send our Search to a new Search Provider.

And yet, if we are to facilitate the easy transfer of this data, we also need to protect users’ rights, even as we expose more secrets to more people.

Schwab, for example, could greatly improve the ease of finding appropriate offerings if they could review the relevant parts of the current Search instead of relying on you entering just the right queries and properly navigating their site architecture. But it is unlikely that users are going to want to give Schwab any information unless there’s an understanding about just exactly how that information will be used (and the ability to select just what information is sent). We generally don’t want companies to start sending us junk mail or calling us with sales offers just because our Search shows that we are in the market for one of their products. But, if we could be assured they would use our Search just to provide better results and perhaps to improve their offerings, we are far more likely to share that part of our Search that could help them help us. We want explicit agreement for data rights access and we want it before we give them any data, and when we want to select what we send so they get just the parts that make sense, and not any personal information we don’t want to share. A User-driven Search solution must not only allow users to send select portions of their Search wherever they want, it must allow them to set the terms for exactly what recipients can do once they get it.

6. Impulse from the user as a specific statement of Search Intent

Recommendation systems presume that an analysis of your history is the best way to discover what you might want now. The NetFlix recommendation challenge and Amazon recommendations feature both use this approach. Not only does this place the user in a passive mode, it also has no facility for users to state what they actually want, right now. People have widely varying interests and easily switch between tasks even in the middle of a Search. Our past transactions may paint an interesting picture of who we are, but it rarely describes what we want in any given moment. What we really want from NetFlix isn’t the “perfect” movie for someone with my viewing history, we want the movie that’s perfect for the mood or situation we’re in right now.

Search systems, on the other hand, rely on a specific “objet de Search” as a trigger for directing efforts. The objet de Search is a keyword or other statement that explicitly represents the user’s intent in some way. At traditional search engines, the query serves this purpose, with the user essentially asking “what web pages have these words” in the hope that those words might be on the page that has what they are actually looking for. At structured Search Providers, like Expedia or Orbitz, the entries for departure, destination, date, and number of travelers in the combined form data comprise the objet de Search.

For User Driven Searches, we must move beyond the keywords and limited structured form fields to allow a more complex, more expressive statement of intent. This statement should include the entire range of Search activity for your given Search, including queries, Search Providers, clickthroughs, captures, and annotations. In short, it should bundle up the entire Search and present it to the Search Provider as an explicit statement of intent. This presentation must be independent of any data silo, unlimited by the offerings of any particular vendor. It should be a proactive statement of “Here’s what I’m looking for: here’s what I’ve found so far and where I’ve been. Got anything that might help?”

Most importantly, Search operates in the foreground, with an explicit impulse from the user. User-driven Search isn’t about background profiling and analysis to try to guess user intent. It requires an explicit means for users to state their intent in ways Search Providers can understand. Instead of predatorial “targeting” of users with particular demographic, psychographic, or behavioral profiles, User-driven Search operates exclusively on that objet de search, as the entire representation of user intent. No more guessing. No more secretive or unauthorized tracking. No more stereotypical clustering based on industrial-era models of consumer behavior. Instead, User-driven Search Providers respond directly to clear, unambiguous representations of confirmed user intent.

Towards an Open Standard

This is the kind of solution we are working on at SwitchBook. At the VRM Workshop 2008, I was excited to learn more about MatchMine from J Trent Adams; they are moving in a similar direction for media-based recommendations. There is currently no service we know of that fully delivers on the promise of User Driven Search, but I’m looking forward to working with Trent and others to develop the open standards and protocols to make it possible.

If you are interested in joining the conversation, send me an email. We’ll be setting up a listserv to talk on a more regular basis. All are welcome.

[Update 5/3/2009: "user-driven Search" to "User Driven Search"]

“The Social Graph” is a popular misnomer that has plagued the social networking portability conversation ever since Brad Fitzpatrick catalyzed the blogosphere with a vision about the Global Social Graph.

But in fact, “The Social Graph” has little real value outside of computer science elegance. Nobody but Big Brother, the TSA, the CIA, and [insert surveillance agency of your jurisdiction here], actually want that single, monolithic view of all the relationships in the world. That’s The Social Graph.

In contrast, my social graph is hugely valuable to me. Your social graph matters to you. And it might be interesting to discover where our graph (plural) overlap. But neither of us actually care about The Social Graph.

At the VRM Workshop 2008, here at Harvard’s Berkman Center for Internet and Society, it came out that “social graph” is actually plural.

Like fish.

The Social Graph is a misleading distraction, a handy buzzword we can all slip into our cocktail conversations. But the real value is in the personal, independent social graph we all have. Plural.

If you think about it, that’s the only way you can really make sense of it in our user-centric, user-driven world.

At VRM2008 in Munich and at IIW in Mountain View, I started a conversation about User-Driven Search, the premise: what would it mean for users to truly drive their searches?

User-driven is a new term that came out of the VRM community riffing on the meaning of user-centric development and user-centric identity. User-centric is a nice term, but it could be construed as limiting. For example, user –centric definitely implies that that the user is at the center of attention and the focus of the architecture, but it doesn’t necessarily mean the user is in charge of the experience. That’s a key distinction.

Not just tuna salad

Adriana explains this difference between user-centric and user-driven as metaphorically the difference between buying ready-made tuna salad or picking and choosing your own ingredients and making the tuna salad yourself. When I first talked with Doc about user-driven instead of user-centric, Jim Carrey’s The Truman Show immediately sprang to mind: from birth, Truman is the protagonist in a huge reality show revolving around him… only he doesn’t know it. The climax of the show is Truman discovering the rest of the world and confronting his father/producer. Clearly the Truman Show is Truman-centric… but it is most definitely not Truman-driven.

It’s about impetus and authority

For me, user-driven means that the user provides the impetus and is the controlling authority throughout the transaction. Sure, sometimes there is negotiation or collaboration with others… the user isn’t omnipotent, after all. However, the user is in charge of creating his or her own experience. This fits with user-constructed or customized solutions, like the tuna salad recipe. However, it has implications far beyond the limits user-created or user-customizable architectures.

Is the user initiating the experience? Is the user’s moral authority the primary control throughout the system? Is the system transparent to users, enabling them to make their own informed decisions about what will be presented to them and how it is presented? Is it the user who is shaping the input, intermediary results, and final outcome? If so, then it is user-driven. If not, it isn’t.

When it comes to the tuna salad metaphor, this is the equivalent of the tuna salad being made when I ask for it and on my terms. Not before. And although I could choose to make the salad myself–that is definitely user-driven–it could also be made by someone else to my specifications… extra mayo and black pepper, no onions, thank you.

Search as user-driven

Google’s keyword query-response approach to Search is, of course, user-driven to some extent. Nothing happens until the user enters a query, users are free to enter any query, and the system responds with results tailored for exactly what the user queried. The user does shape the experience to a limited degree. And yet, it still provides only a slim façade of user control. There is no way to modulate the algorithm, no way to let Google know which results are good or bad, and no way to refine the search other than keyword guessing games. And, perhaps most importantly, there is no way to manage the search beyond the immediate query. For that, the user is dependent on other techniques: bookmarking, cut & paste, opening multiple windows or tabs, even printing to paper or PDF to keep track of good finds. Evolution in Search History management is starting in the right direction, but the ideas here have been rather uninspiring so far.

User-driven systems create value inherently

The limits on the user-driven aspects of Google are particularly ironic given that it is precisely the element of user control that creates Google’s greatest asset: focused attention. Google’s money making asset is the collection of user-specified queries, queries that explicitly state words related to the user’s interest and implicitly denote user intent. It is precisely because the individual enters a specific keyword that Google is able to sell targeted ads… at great profit and benefit to advertisers and searchers alike.

The query entered in the Search box gives Google a implicit statement of intent. That intention is the gold Google resells to advertisers. If Google didn’t let users drive that intention, if they looked more like a content site or “Internet portal”, they’d have a lot less intention to monetize.

If we can extend that control, if we can make search even more user-driven, if we can enable richer, more explicit, more user-driven expressions of Search intent, I believe we can create even more value for everyone involved: search companies, advertisers, searchers, even non-paying websites showing up in “organic” results.

What does it mean to have User Driven Search?

At SwitchBook, we’ve been doing a lot of thinking about what User Driven Search might mean. I like starting the conversation with a simple example: what would it mean if I could take my search history from one search provider to another? This “dataportability” example is just an initial notion of how Search might become more user driven.

So, what do you think of when you hear (or read) “User Driven Search”?

I’ll be leading a session on this topic at the VRM Workshop next week. I hope you can join us.

This material is based, in part, upon work supported by the National Science Foundation under Grant No. 0740861. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

[Update 5/3/2009: "user-driven Search" to "User Driven Search"

Is VRM really different from social CRM ?

Yes, although exactly how depends on how you define social CRM. Based on my understanding, I would suggest that VRM is first and foremost about providing value for the user with any vendor, as opposed to using social networking tools with a particular vendor. VRM is vendor agnostic and silo-adverse. The goal is to catalyze the development of tools for individuals through protocols and standards that let them work with any vendor seamlessly, without loss of functionality or services.

Does VRM work with a CRM ?

Sure. A CRM is a company-centric system. Every company should pay attention to its customers and CRM is currently the best-of-class thinking on the enterprise-side for how to do that. Different VRM services act on behalf of the individual, yet still require connecting to enterprise systems. For things to be seamless, VRM services should marry into CRM services for fulfillment.

Can VRM be implemented in all kinds of business?

Yes. Any business can support VRM services and be compliant with general VRM principles. Ultimately, it will be as easy for a small company to be VRM compliant as it is for a small company to run a blog or a wiki today. That takes some level of technical sophistication, but it is within grasp of any company that wants to invest a small amount of effort using freely available open source tools. Eventually, VRM will be available in the same way.

What’s needed for VRM to work ?

We need to work through electronic marketplace issues from customers’ perspectives, with attention to the full power of relationships, finding consistent ways to create new value through the network. For the Standards Committee, that means a public conversation starting with users and requirements. Once that is vetted in an open source manner, we can explore particular implementations. We believe that with a well defined, high quality requirements specification, service providers will emerge to deliver those services.

As customers are looking for lower prices, don’t you think that Personal RFPs are gonna cost more for customers (because they are personalized offers) ?

Two things here. First, I don’t think customers are just looking for lower prices. They are looking for better value.

http://blog.joeandrieu.com/2008/03/07/pricing-markets-and-demand-vrm-style/

One of my favorite examples of this is Shopatron’s business where they sell everything at 100% manufacturer suggested retail price, no discounts, no rebates:

http://blog.joeandrieu.com/2007/01/19/shopatron-redefines-vendor-relationships/

Second, the personal RFP is designed to eliminate transaction costs in the marketplace. Currently, product and vendor discovery is slow, expensive, and uncertain. That means buyers waste time and vendors waste advertising and lead generation dollars seeking the right match between needs and solutions. Any time transaction costs are reduced, you have an opportunity for better prices.

At the same time, Vendors will be discovering ways to provide more value to customers and the net result could easily be that customers will end up paying more for enhanced services or products. Ideally, this will mean that commodity products continue to drop in price while value-added customizations are welcomed by buyers and voluntarily paid for at a premium over the commodities.

What do you expect from VRM?

I expect it will take longer and be more work than any of us would prefer. However, I think that the concepts behind VRM, and hopefully our work developing standards and catalyzing working solutions, will enable a fundamental shift in the marketplace. As Doc Searls has said more than once, the industrial revolution is over: industry won. There is an incredibly powerful legacy of using computers and networks to help companies make more money (and create more value as they do so). Unfortunately, companies tend to think for themselves first, often to the detriment of overall economic benefit.

I see a world where every individual is engaged and empowered to get the most out of their relationships with vendors–vendors of all sizes. In that world, not only are individuals and vendors each getting and creating more value directly, the entire economy is operating at a higher efficiency as less money is spent on wasted advertising and product development and more is spent on fulfilling verified demand. This would supercharge Adam Smith’s invisible hand and provide a significant increase in aggregate global wealth for everyone. It takes the benefits of the zero-distance network and extends it efficiently into the domain of user-driven commerce.