Friday, May 22, 2015

StartupVille: Testing vs Optimizing

TL;DR - Optimizing, is figuring out how to do make a result better.  Testing is checking if something is so good it doesn’t need to be optimized, or so bad you're better off doing something different.

Founders spend a significant portion of their time comparing alternative to see which is better and why. For example how to make a landing page more appealing, or what price they should set for a product. This process is incredibly important and called optimizing.

Optimizing is often expensive so before you start optimizing, test to see if something has so little traction it's not worth starting to optimize, or has so much traction there's no need to optimize.

For example, imagine you have a customer acquisition cost (CAC) budget of 5$ and you run some search ads and get a 0% click through rate.  In that case, instead of trying to optimize search ads, you are probably better off trying a different channel like social network  ads.  Now, imagine when you try social network ads you find your CAC is only 2$, in that case, you can skip optimizing completely as you’ve already achieved your goals.

A few final thoughts on the topic:
• Testing is essentially optimizing relative to the null hypothesis
• A side effect of optimizing rigorously is you describe why a change is important.
• You can always optimize after testing, in fact if it's important you certainly will.
• If it's trivial to do some optimizing while testing, go for it.

No comments: