Tuesday, February 01, 2011

Do-Not-Track (How about piggybacking on the User-Agent?)

I think I’ve read just about every white paper, article, blog post, and tweet about Do-Not-Track (DNT), including the FTC’s recent 121 page preliminary staff report that thrust the concept into public consciousness. For those unfamiliar with what DNT is exactly, not to worry, it is really very simple.

The idea behind DNT is providing online consumers, those sitting behind a Web browser, an easy way to indicate to third-parties that they do not want to be "tracked" -- they opt-out. DNT would hopefully replace todays system of having to register with dozens of different provider websites to obtain “opt-out” cookies.

As the FTC pointed out, the out-out cookie approach proved unscalable and could never have been effective with the spirit of its intent, consumer privacy. Adding insult to injuring, anyone seeking to improve their privacy by deleting all their cookies would simultaneous delete their opt-out cookies too. They’d have to perform opt-out registration all over again. No wonder the advertisers and tracking companies support this model.

The FTC report gave no real technical guidance on how DNT should be implemented. Not that they should have. What you must first understand about DNT is that in all models, there is NO real technical privacy enforcement. Basically the consumer is asking (buried somewhere invisible in the HTTP protocol) anyone who is listening, “please do not track me.” It is then on the honor of the tracking companies across the Internet to support the DNT system and comply with the request when they have no legal obligation to do so. Which is not to say DNT is without value. It would be helpful to have a legal remedy available when all technical self protection mechanisms eventually breakdown.

Since DNT started making headlines Google, Microsoft, Mozilla, and various browser plug-in developers have been experimenting with different approaches to DNT in their respective Web browsers. The one seeming to get the most traction at the moment is adding a special 'DNT' header to each HTTP request. For example:

"DNT: 1" - The user opts out of third-party tracking.

"DNT: 0" - The user consents to third-party tracking.

[No Header] - The user has not expressed a preference about third-party tracking.

At first glance this does appear to be the logical and superior model over all others I’ve seen so far. Then I got to talking with Robert “RSnake” Hansen about this and we came to a slightly different conclusion to where DNT would best go. First remember that there are a lot of great big powerful corporate interests that really don’t like DNT and what it represents. If effective and widely adopted, business models are odds with consumer privacy choice would be seriously threatened. Opponents to DNT will seek to confuse, sabotage, derail, downplay, and stall progress at every opportunity. The final accepted protocol must be resilient to a large portion of the Internet hostile to its very existence.

DNT data must be able to traverse the Internet to its destination unaltered and be logged on the other end (the Web Server) for auditing / statistical purposes. If DNT ends up being a new HTTP request header, those headers like most others are rarely logged and never by default. It would be far too easy for a tracking company to ignore DNT headers and claim they never got them. Proving otherwise would be difficult for a plaintiff.

An alternative is piggybacking DNT onto an already well established header. A header one that no one in the connection stream would typically think of touching and that is already widely logged -- by default. The User-Agent header would make sure an ideal candidate. Imagine something like this with the DNT tacked onto the end:

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv: Gecko/20101203 Firefox/3.6.13 DNT: 1

Simple. Easy. Logged.

Now if we can just encourage the browser vendors to enable it by default. :)


Ari E-B said...

This also has the benefit if being very easy to retrofit onto existing browsers. Just make a plugin that modifies the user agent string (and several exist already), and you can easily retrofit your older browsers. I like it.

Ari E-B said...

Also, you don't have to mess with the HTTP spec to implement this idea.

Jeremiah Grossman said...

@Ari E-B: Thanks. @shaver from Mozilla commented about a potential drawback. Essentially that User Agent's are very difficult to parse and that logging a new HTTP header isn't all that difficult. So therein lies our trade-off. Good to view things from the other side.

zonky said...

Why is anyone proposing that we trust servers to behave? This is a classic 'trusting the client' design, flawed from the outset.

(the client is the server, in this case, but the analogy is valid.)

Jeremiah Grossman said...

@zonky: No one is blind to that issue. The idea would be that hopefully one day DNT would be 'legally binding' and that tracking companies would be compelled to honor the preference or face some kind of consequences. To your point, in the meantime we also need better technical privacy control as a complement.

Disenchant said...

And why not standardize a User-Agent notation at the same time? I vote for a JSON conform notation which makes it easy to parse :)

Jeremiah Grossman said...

@Disenchant: Talk about a big can o' worms. :)

Michael. said...

Yay! Let's overload the user-agent string some more! Making it harder to parse, harder to understand, and ultimately, just that little bit more useless.

If it were possible to standardize the user-agent string, I wouldn't mind so much (but then it would be possible to implement a new header without a problem, making the point moot).

In other words, I disagree with your proposal.

Jeremiah Grossman said...

@Michael: Hah! @shaver voiced the same concerns. Ultimately there is a trade-off to be made. Like yourself I'd imagine I want to give DNT the best chance to succeed (adopted). Personally I think stacking onto the User-Agent header is preferable to a completely new DNT header.

zonky said...


Just like how laws against spam have stopped spam?

In any case, I like to use web servers, all over the world- So you proposing solving the browser privacy issue by some sort of multi-state international treaty as well?

This entire idea is totally unworkable worthless. This is because the chosen method of solving this issue relies on good behviour, not a technical solution.

Jim Brock said...

What a great idea and thread!

I wouldn't discard this idea because it relies on voluntary behavior (you can deal with that though add-ons). It's quite possible that privacy audits will be required to confirm compliance with user preferences, and that companies representing 90%+ of tracking activity will have to complete them. In that sense, an approach that is easier to audit could be far superior.

Has any technical problem been identified here to implementing DNT this way?

Anonymous said...

The user agent idea assumes that this header will be sent for each and every request. This will certainly be true for browser implementations and reduces the "implementation" in Firefox to a single line (add a preference). However, Adblock Plus and NoScript for example allow sending this header selectively. In particular, it should be possible to exclude some sites where the DNT flag should not be sent. And changing the user agent dynamically is a lot more complicated than adding a header. Not to mention that you might run into weird issues if you change your DNT settings (some websites tie the session to your user agent for added "security").

Jeremiah Grossman said...

@zonky: Like I said before, DNT is not meant to replace an technically enforced solution, but provides for a legal remedy. And yes, there have been quite a number of spammers successfully prosecuted, civilly and criminally. And of course we don't expect spam laws to stop spam completely, just like we don't expect murder laws to stop murder completely.

@Jim: yes, a few drawbacks have been brought up that are worthy of consideration. First, User-Agent headers are unwieldy beasts and difficult to parse. Overloading the header is unattractive to those that have to do. Secondly, as Wladimir just brought up, sometimes User-Agent headers are an active part of session handling. Any dynamic change may impact the user experience.

@Wladimir: Thanks for the comment. Whether DNT is placed in a new header or in the UA should not impact selectivity of what websites it is sent to. Conceivably the UI implementation could be identical. You bring up a good point about UAs and session handling, Ive seen that behavior before as well. Though this should only be a problem if the UA spontaneously changes during an active session correct?

Aaron Bryson said...

How about this (though this would be HIGHLY objected by the tracking companies...and this assumes that companies would be legally obligated to listen to the user preferences, just like with no-call lists on cellphones):

By default, every browser is opted out, no special header needed. However, if a user specifically set his/her browser to allow tracking, then a DTN HTTP header can be added and sent in their HTTP requests that let the tracking company know it's ok to track them. This way, it makes it hard for a tracking company to lie/deny that they did not know if they user wanted to be tracked or not. There would be no confusion, and it the burden of proving a user wants to be tracked would be on the company (they would have to go through the trouble of modifying their server logs and adding DTN headers to all HTTP requests).

Going on the idea of "default-deny" versus "default-allow".

Anonymous said...

@Aaron: You have to stay pragamtic here. This proposal isn't so much about what is "right" but about what can realistically be done. DNT only has a chance as an opt-out mechanism because the overwhelming majority of users simply don't care, they won't bother changing the prefs. With opt-out you assume that these users are ok with tracking which is actually pretty close to the truth. An opt-in model however declares that these users don't want to be tracked (which misrepresents their opinion in 99% of cases) and is essentially a death sentence for the companies doing the tracking. And I don't think that anybody will be able to force them into signing this death sentence, not even FTC.

Ari E-B said...

Hadn't though about the parsing issue - what if you put the DNT string at the beginning?

Jeremiah Grossman said...

@Ari: I couldn't say for sure if that would help. Maybe it is just people venting about the difficulty of parsing UAs in general, which it is, but this just gives an excuse to vent.