Dayspring Technologies

A no-brainer; the button has to produce more click-throughs, but a fun test. You can see the full page in context at the Wazia web site.

Randomized Trials

The cool part about Google’s Website Optimizer is that you end up getting a classic randomized trial experiment. Subjects are randomly shown different versions of the page, and then responses (conversions) are collected. All the classic statistical techniques are in play: proportions of conversions, confidence intervals, and so on. Google calculates them all for you, and displays nifty visuals to show you the results.

As a statistician at heart (and by education and in my former job), this is like opening toys on Christmas! It’s not often that you can set up a pure randomized trial.

Google uses this type of technique all the time, to analytically determine almost everything in their business, like how much white space to show at the top of your Google search results. That was an actual example they showed at a conference I went to — they had three variations of the search results page that, at first glance, looked completely identical. The difference was only in the white space between the search box and the first result. With their randomized tests, they could conclusively show that visitors clicked on more search results (a conversion for Google) when there was the least amount of white space. They have based their business on data-driven decisions, and this is one part of that.

There are some unspoken assumptions behind making changes based on these experiments that might make one wary, like assuming that future visitors to the page are the same as the test subjects. That’s seems like a reasonable one, but it’s an assumption nonetheless, which needs to be recognized.

Of course, Google gets millions of searches per day. So even a very small difference in conversion rate gets picked up very easily by their experiments. For the rest of us, getting by on maybe dozens of hits per day, life is a little harder. Differences in conversions from various pages will take longer to show up. Still, the Website Optimizer can be a great tool for trying out different page layouts, or visual elements in your page.

Technical Difficulties

Since the Wazia page is a dynamic JSP, it took a bit of work to adapt Google’s instructions, which seem to be primarily written for static HTML pages. First, the original text links get wrapped by script tags and an unbalanced noscript tag, which define a section to be replaced by Website Optimizer.

<script>utmx_section("Amazon links")</script>
   { my JSP code here, to show the text links }
</noscript>

Then I provide Google with the button variation. The catch was that I can’t use any JSP tags that I would normally use, since the Website Optimizer can’t interpret them. So in my original JSP page I set a JavaScript variable that has the info I need to create the Amazon button link:

var keywords = "<c:out value="${myBookBean.title} ${myBookBean.author}"/>";

Then in the Website Optimizer variation, I can use that JavaScript variable to change the link href to what I want:

<a id="amazonLink" href="">
   { image tag for Amazon button }
</a>
<script>
   $('amazonLink').href =
      'http://www.amazon.com/gp/search?ie=UTF8&keywords=' + keywords;
</script>

So far in my experiment, I haven’t received enough conversions to determine a definite winner between the two versions. My free ad money has run out, but we still get a fair amount of organic search visitors, so I’ll keep the experiment running for a while longer.

A funny note — on the day I started writing this, on the Website Optimizer I seem to be getting the German(?) version of the title graphic. Maybe Google is testing whether an unintelligible language is better for conversions?