A/B Split Testing with ASP.Net and C#

Aug 23

Aug 23 A/B Split Testing with ASP.Net and C#

As software developers we often create things others tell us to make. Usually we do so blindly. There is no feedback loop to tell us what features are working well or whether they are being used at all. With no goals, there is way to tell whether what we build is successful or not. It doesn't have to be this way. Books like the "lean startup" discuss "validated learning" which essentially are techniques to prove that the assumptions you have (or the folks giving you requirements have) are actually true.

One of the biggest tools in the tool box for validating these assumptions is called split testing or A/B testing. To achieve this you must figure out what you want to test. Essentially you need to come up with a set of hypotheses of what feature or change to your website will result in a positive way toward a goal you've set. Goals can be purchases, newsletter sign ups or just clicking the next button on a form you have. The features you want to test can be large like the addition of a brand new features, or tiny like changing the title text on your page or anything in between.

Software developers are often good at figuring out what feature like aspects of an application can be tested this way. Do you get more sign ups with a single long form, or a wizard like 3 step set of short forms? Should the button go over here or over there? Then there are the simpler, harder to predict optimizations. Which headline is going to engage visitors more? What images and colors should be used? There is plenty to learn about your visitors, but since this kind of testing is less intuitive, developers probably shouldn't be the ones coming up with the hypotheses. You need marketers to make these decisions and you need to give them the tools to see the results, change the tests and continuously improve.

Client vs Server Side AB Testing

There are a number of frameworks that are implemented using JavaScript and will manipulate the DOM client side and provide their own web analytics. These include free hosted tools like Google Analytics's Content Experiments, commercial tools like Optimizely and visual website optimizer. None of these can be used server side for more advanced AB testing scenarios.

There are a number of tools available to help ASP.Net developers implement split testing in their applications but most are developer focused. They provide a way to articulate split tests in code.

Fairly Certain

FairlyCertain provides ASP.Net web forms tag and a straight forward API that could be use in code behind or directly from Razor views for MVC. Stats are stored to a flat file and a dashboard ASPX page is provided to show statistics.

Fairly Certain Code Examples:
<%@ RegisterTagPrefix="ab"Namespace="ExpatSoftware.Helpers.ABTesting.Controls"Assembly="Web"%>
<ab:Test TestName="sidebar_intro_text" runat=server> <ab:alternative Name="learn_how" runat="server">
It lets you quickly test changes to your website
to learn how real users respond to them.
</ab:alternative>
<ab:alternative Name="sell_more_stuff" runat="server">
It lets you test website changes against each other
so that you'll sell more stuff and make more money.
</ab:alternative>
</ab:Test>
MVC Style
<%if(FairlyCertain.Test("bold_test","bold","standard")=="bold")
{%>
AB Testing is freaking awesome.
<% } else { %>
AB Testing is freaking awesome.
<% } %>
<% if (FairlyCertain.Test("show_disclaimer")){%>
(possible side effects may involve vomiting and hair loss)
<% } %>

There is a fork of this framework that fixes some bugs and adds a few feature called ABTesting.

ABTestMaster

ABTestMaster was built more for MVC and provides some Controller Action attributes to enable more declarative AB Testing. Similarly Goals can be configured on Actions as well via attribute. This framework supports both csv file and sql data stores and actually has a SaaS offerring that provides dashboards to show your metrics.

ABTestMaster Examples:
[SplitView("Version 1", "Payment", .1)]
publicActionResult Payment()
{
return View();
}
[SplitView("Version 2", "Payment", .9)]
publicActionResult Payment2()
{
return View();
}
[SplitGoal("Checkout")]
[HttpPost]
publicActionResult Confirm(PaymentData data)
{
return View("Receipt");
}

mtelligent

Lastly, we have mtelligent, a server side framework for both ASP.Net Webforms and MVC that I built. This frameworks comes with a dashboard application where even non technical users can create and modify experiments, hypothesis, goals and cohorts. It then has a singleton façade API for accessing experiment parameters and registering conversions. All data is stored in a sql database and the metrics captured allow you to segment your users into cohorts who you can then create more specific experiments on or just personalize content to that cohort directly.

mtelligent Code Examples
Variables are associated with an experiment and can be rendered inline:
<h3 style="color:@Html.GetHypothesisVariable("Honey vs Vinegar", "TitleColor")">@Html.GetHypothesisVariable("Honey vs Vinegar", "Title")</h3>
@Html.GetHypothesisVariable("Honey vs Vinegar", "Copy")

<img src="@Html.GetHypothesisVariable("Honey vs Vinegar", "Image Source")" />

The ExperimentManager is a singleton instance that can be used to get details about experiments and register conversions.
ExperimentManager.Current.AddConversion("Honey vs Vinegar");
if (ExperimentManager.Current.GetHypothesis("Honey vs Vinegar").SystemName == "Honey")
{
ViewBag.Message = "Thanks you so much you wonderful user.";
}
else
{
ViewBag.Message = "We knew you would do as you were told. Carry on worm.";
}
Html Helpers are provided to make it easier to use from Razor MVC views
@if (@Html.IsVisitorInCohort("Authenticated Users"))
{
You are in the Authenticated User Cohort.
}
else
{
You are not in the Authenticated User Cohort.
}

I created this framework, before I started getting into Sitecore, which also has it's own built in AB Testing framework. It is fully functional, however it's by no means complete.

Conclusion

AB Testing is a vital tool in measuring and improving your web applications. You have a number of choices both client and server side for implementing AB Tests. Do you know of any other frameworks or approaches? What do you think of the ones I mentioned? Let me know in the comments.