Quickly and Cheaply Validating Your Ideas

So you have a great product idea, but how do you prove that the world wants it? Well, one way to do this is to spend 3-6 months building that product (for your sake, we assume that's all it will take). You hack away, building the features that you want. When it is finished, you attempt to get some people to come to your site and try it out. If all goes well, you get a bunch of people to try it, and they give you feedback on whether they like it. Hopefully they do, and if not, well, it's back to the drawing table. Maybe they even tell you what they want to build next time. However, you have just burnt 3-6 months of runway (which could be coming out of your pocket). It's also pretty dejecting to build a product that no one wants.

There has to be an easier way to do this. Well, there is. Until recently, it was pretty expensive and difficult to validate your ideas (pay someone to run a focus group), but recently this has become easy, thanks to two inventions: Google Adwords and rapid development frameworks (Ruby on Rails and Django are two that come to mind).

So there are two things that every web startup needs to do: user behavior studies and experiments. These can save you a lot of time and money, and can help put your product on the right track in minimal time. If you haven't already started. you should be doing them now. So what are these things, and why are they important?

User Behavior Studies
I'll start with these first, because they are simple. This could more simply be called "talking to users," but they are actually better than talking to users, because you get to see how a user truly feels about your application (and this is pretty obvious when you do a user behavior study). To your face, a lot of users will tell you that your product idea sounds great, but that doesn't mean that they truly mean it (or that they will use it).

If you were a big company like Google, you would have a fancy usability lab. You would have people who were hired just to do usability testing. They would run users through a set of actions, and the product managers and engineers would watch behind one-way glass (the product managers would furiously scribble notes about every move that the user made). There would even be fancy computerized systems tracking where the user is looking at all time. In the end, you would be able to see real users use the product. It would be a big production. No one would get any work done for days, and then you would have a big meeting to discuss the "learnings."

How do startups hack this process to their advantage? Well, you can put together a quick demo of your product, and load it on your laptop. Build the simplest possible demo that exposes the functionality you want to test. Then head out to the nearest busy coffee shop. Talk to some random people, and ask them if they will take part in a quick test (most people are pretty nice about this). I know it's scary to talk to random people, but much less scary than spending 6 months on a product that flops. Start by asking general questions about how they already accomplish what your product wants to help them do. If, for example, your product helps them to shop, show them a blank browser, and ask them to shop for a product that they want to buy. Observe what they do. Encourage them to speak their thought process aloud. Observe what pains they encounter.

Next, whip out your demo. Ask users to attempt to "use" it. Give them as little instruction as possible. Mostly, just watch. Encourage them to narrate. If they get stuck, observe that before you tell them what to do.

Watching a few users can give you invaluable feedback. My current team had some conflicting hypotheses about how users would use our product, so we resolved this by showing the product to five random people at a coffee shop. In two and a half hours, and pretty much for free (I got a parking ticket for being over time in a two hour zone), we learned key facts about how people shop for products on the Internet. And we probably saved ourselves one to two months of wasted effort.

If you haven't done this yet, do it tomorrow.

Experiments
So while user behavior studies are descriptive, these give you hard data. To do this, you hack together several possibilities, and you put it out to the users to vote. But you don't tell them what you are doing - you can randomize the behavior of your site. First of all, build the minimum possible experience required to expose a feature. Then build a few alternatives, and make your application randomly serve one of the possibilities (be sure to keep track of which possibility you are testing). Then, use analytics software such as Mixpanel to record what the user does in each situation.

Finally, drive some Google or Facebook Ads to your page. Don't pay a lot of money - you often don't need all that many impressions to figure out what is going on (I think it cost us $6 to run one test that gave us valuable information). You may need to do some work figuring out which creatives lead to the highest click-through rates. Click-through rates don't matter terribly much, although Google will stop serving your ads if your CTR is too low. Plus, you will probably need to learn how to do this for when your product launches.

In the end, you can analyze the results, and draw conclusions based on what happens. It can be surprisingly quick - each time we have run an experiment, we have had fairly convincing results within a few hours. Of course, it is important to control the variables. If you change too many things between conditions, it may be hard to figure out what the true differences are.

In our case, we looked to see what would lead users to give us their email address (a big portion of our functionality involves emailing updates to users). We tried three conditions:

1) We immediately prompt the user for their email address.
2) We wait until the user does something on our site, and then we prompt them for their email address.
3) We display some site functionality related to the query term that the user clicked through on, and then we ask for the email.

Interestingly, the third condition had a massively higher conversion rate (actually pretty respectable), followed by the second condition, and no users gave us their email in the firstcondition (big shocker).

Ego Check
If you love your product idea more than your first-born child, you probably don't want to do this. There is nothing more crushing than having your product idea completely devalidated by an experiment or user behavior study. But it is even more crushing to spend six months to a year working on something that you end up killing because no one wants it. I've done both, and I would drastically prefer the former to the latter.

So get out there and start already.