Tracking for an E-Commerce site

It is amazing how many sites do not track well. Nearly ever tracking model I have walked into has problems, but the "right" tracking model is actually pretty simple. Avoid a lot of problems, and build it right from the start.

I know a lot of solutions use page tags (also known as web bugs or 1x1 transparent gif images or sometimes horrendously they are javascript calls), and that is fine and they mostly work (so long as cookies are required), but I actually favor a simple db write by the web server each time it serves a page. Web bugs add weight to a page, unnecessary traffic to the net, and there is overhead associated with the weight as well as the request /response time, and I am a purist. Also, do you really want google(for the free urchin based solution) knowing everything about every web site? Particularly if google is a potential acquirer of your startup you might want to think twice before showing them all your data. Also, it just annoys me to call someone else's site for anything within my domain. Oh I do it all the time in order to make deals happen, but it still annoys me. If you are going to drop a page tag (LOATH), put it at the end of your content so it loads last on the page after the used has their content, and you might even consider putting it in an iFrame if you or someone else is measuring your response time and you can beat the penalty by putting it inside the iFrame.

- Creative, Placement, Offer: An important concept related to advertising tracking. Every promotion that you use should be tagged with three id's, creative, placement, and offer, and these three variables should be recorded for every visit to your site. You will not need more than that, and if you use less, your tracking, segmentation, and optimization will become necessarily convoluted in order to accomplish your goals. Examples: creative = ABCD123 = 160x600 wide sky with the smiley face on it, placement = EFGH567EFGH = yahoo mail message sent page, offer = IJKL987 = buy 1 today and get one at 50% off.

- Visit table: For each visitor to your site, assign a session id that is absolutely unique and persistent through the visit. Attach all the standard http header info that you can stand to swallow with the session id as the primary key. I usually hold server timestamp, ip address, referring url, inbound url, and useragent at an absolute minimum, and it is nice to hold things like does the user accept cookies, what time zone are they in, what is their screen size, do they accept javascript, do they accept flash, quicktime, etc.

- Cookies: If the visitor does accept cookies, and has a cookie from you from something prior, attach the cookie number to the visit. This is where it is really nice to serve your own ads so that you can cookie on adserve and really track the marketing lifetime cost of a user. Some people like to put data in the cookie, some just like to drop a cookie number and do a db lookup for all the profile information, etc. I favor the latter but it doesn't really make too much difference and the latter can be more systems intensive.

- Userid: If the user logs in, grab that userid of course. You can use this to stitch together cookie erasers too by seeing a different cookie upon each visit. Presumably you can hold an obfuscated userid in a cookie. Make sure you differentiate in data between an authenticated userid and a cookied/remembered userid.

- Serial number: If you have something else that is stickier than a cookie on a site like a piece of installed software with a serial number, grab that too of course, and you can tie things together even better than with Cookie or userid potentially. For instance, you can re-populate the same cookie id and/or cookie data on subsequent visits if you have cookie information stored in your database (at least the most important cookie information like testing markers and userids. ou can use this same method to join different cookie ids together later using things like shipping address, payment information, etc.

- Page requests: Each time the user requests a page on the site, write the session id and timestamp of the request. You can also add super important things like log in state, userid, field entry data, as you see fit. Also, it is nice to be able to differentiate which link someone used to get to a page if there are multiple paths (eg, the home page link in page text vs the logo in the upper left corner). Among many other things, you can use this table to ssee which products people viewed but did not put in their shopping cart. This can help organize yourr site to convert best.

- Transactional tables: Almost don't need to mention this, but you will hold things like shopping cart data, shipping address, etc as needed to make your site work. Just make sure that you tie a session id to each of these and you will be able to trace everything that happens on the site back to creative, placement, and offer that brought you the transactions. Normally, hold a cart id for each shopping session that starts to build a cart, all the product in the cart, then if they check out you can join the cart to a payment table, shipping table, etc.

- Test tags: if you are doing random testing, somewhere you need to store which user experience a visitor to your site received and enforce persistence.

Sounds simple, but no one ever does this out of the box, and you need it to Market effectively.

3rd Party Marketing Tracking Systems (Web Bugs, ick)

A smart developer I once worked with, Andres, called this a web bug system. When he built it back in 2000 (I think?), it was the first one that I had ever heard about, though likely others were also built around that time. Since that job, I have either had a similar system built or wished I had a web bug serving system at every company I worked for.

Here's the deal: web bugs, transparent gifs, 1x1 tracking pixels, whatever, are nasty as far as I am concerned. If you haven't worked with them before, here's how it goes. Let's say that ABC advertising is sending traffic to my site, but I only have to pay them when a customer purchases something on the site. I could just look in my database and tell them how many people purchased from the ones they referred to me, I could even ping them in real time or send data to an API they built for that purpose. But laziness is much more prevalent than trust, so web bugs are used most of the time instead. I put an image on my checkout/confirmation page with ABC advertising's domain, and then when my customer's browser calls that image, they can count the purchase. They can also look for their cookie on the customer's machine, and tie each conversion to a specific referral. To make things even juicier, they could see and demand payment for any purchaser that they sent in the last 7 days or the last 30 days because they introduced the customer to your site making it more likely they were going to use your services. They could even see customers that they never referred and infer something about those customers based on the fact that they are using your site (just bought a camera? maybe I should market memory sticks to you).

More recently, some of these tracking pixels were being blocked by browser plugins and privacy schemes, so many of these tags have been getting switched from image tags to java script calls. Since the privacy plugins do not know if the java script is there for tracking or actual page functionality, it would be a little riskier for them to block a java script call automatically. Also java script can suck a lot more information out and do a lot more than a simple image call. For instance, some ad networks get the java script call and drop a whole bunch of image tags on the page automatically. By the time you get dozens of different advertisers and networks and some of them piggy-backing dozens of tags onto one java script call, you can end up with a lot of tags calling all over the place, and you can also end up with more than one advertiser claiming responsibility and demanding payment for your referral if you are not careful.

So build yourself a web bug system like that works like this:

- Query string tagging: Tie inbound query string parameters and their values to firing a web bug. For instance, tell Yahoo to use the link www.mydomain.com/widget.html?tid=12345 when they send traffic to your site. tid is short for tracking id, and it tells the web bug system which web bug to fire. So Yahoo will get a RightMedia web bug for any traffic they refer, but their web bug or javascript will not fire when someone else comes to the site with a tid=54321 query string in there.

- General regular expression matching: Ideally, your system should not only accept tidd, but be able to look for any query string parameter/value pair that you put into a regular expression matcher of sorts. This is because occasionally you will get publishers that are picky about the URLs they are sending traffic to you with. It may also allow you to minimize the number of query strings on the inbound url, which is more consumer friendly.

- Trigger events: Create a set of actions that can trigger firing the web bugs. So in my example above, I talked about the checkout page having a web bug on it. You might also want one for user registration, email newsletter sign up, whatever you are going to pay marketers for. Some marketers and partners may want one on the landing page so they can count total referrals, and some may want a tag on all of your pages so they can monitor their referral's use of your site (yuck!!!! but it does happen). In the end, you may end up with 3 or 4 events, which combined with a query string parameter/value pair on the inbound, cause you to fire a web bug.

- Persistence: If you want to get advanced, use your own cookie to look for and credit prior referrals. This will make publishers that want to get paid for all conversions in the next 30 days happy. So here's how it goes, assign a web bug stickiness when referral shows up the first time. For example, all hotmail referrals get a web bug on checkout page from this time through the next X days. Maybe Yahoo referrals are sticky for 30 days and Google referrals are only sticky for 7 days. Regardless, if the cookie with the sticky visit shows up again, you would still fire the web bug for them, so you have to start looking at the cookie id on the way in for this as well as checking the query string. Not too bad really.

- Precedence: Complication arises if you want to avoid throwing multiple tags for the same person. What if someone came from Yahoo 24 days ago and Google 2 days ago. Do you fire both? Well, it depends. If you really want to get this right, you would fire any tag which is used for tracking but not payment. But if you are using the tag for payment, you would only fire the tag for the one that sent you a referral most recently, regardless of deadline. You can get really convoluted with precedence rules if you want, but what I just listed is quite enough really.

- Cache busting: one last thing, when the web bug fires, many people want a random non-repeating number thrown into the URL somewhere. Sometimes as a query string, sometimes in the directory structure, sometimes more than once in a single URL. They are trying to make sure the browser does not grab a cached image so the browser actually calls the remote server for the image again. So you should have a cache buster variable you can substitute into the web bug call URL or query string at will.

Tags: webbug, tracking, pixel, publisher, referral, gif, transparent

Wednesday June 18, 2008 - 05:25pm (PDT) Permanent Link | 0 Comments