Entrepreneur & Innovation Exchange
https://eiexchange.com
Five Very Fast Ways to Better Understand a New Product or Market
Ethan Mollick
Wharton School, University of Pennsylvania
These techniques are all designed to provide very quick feedback. However, with the exception of the
fast consumer surveying, it is extremely unlikely that the feedback will be statistically valid. Instead, the
goal is to provide rapid feedback in order to stimulate your own thinking. Don’t take the data from
these approaches too seriously and please use them cautiously!
Good for
Not good for
Customer Interviews Getting rich information about a
problem and how people are
solving it.
Social media/advertising Seeing overall market trends,
analysis estimating market size
Survey of people you know Get fairly detailed feedback on
an idea, product, or problem
area from many people
Fast consumer survey Getting statistically valid
($) information about preferences
Smoke and A/B Test Testing a variety of potential
($) markets, judging consumer
Predicting customer actions.
Making assumptions about
demand, features, or pricing.
Getting feedback on a particular
product or service.
Drawing conclusions about
overall demand for a product,
getting statistically valid results,
figuring out pricing.
Asking complex questions,
getting rich information, getting
feedback on pricing.
Getting deep insight into
behavior
action
Items marked with a $ cost money, they can be done for $25-$100, more if you want to run a more
elaborate campaign.
Please note that the tools listed here can change at any time, so consider the specific websites to be
examples of how to conduct a particular type of test.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
1
Entrepreneur & Innovation Exchange
https://eiexchange.com
Interviews
The goal of these quick interviews is to rapidly get a feel for a market or problem area. Since you are
doing these quickly, you will be skipping a lot of the steps that make interviews, and especially customer
interviews, useful tools: creating personas, validating question, finding appropriate people to interview,
etc. You can look at www.startupinnnovation.org for more information on these steps. Still, interviews,
even fast interviews, can be useful and important in the earliest stages of testing an idea. Here is how to
do them.
1. Decide who to interview. You can choose a number of different approaches:
a. Potential end customers/users. These are people who have the need you are trying to
solve. You can learn about their needs, and about how they try to address them on your
own. Make sure you understand the basic customers you are trying to address first.
b. Potential buyers. If your product is sold through middlemen, retailers, or other buyers,
you may want to understand how they view the customers and your potential product.
c. Industry experts. In a complex field there are often many complicated factors that drive
the market. Government regulation, alliances between companies, competition, and
other factors may influence how your product succeeds or fails. You can interview
industry experts, taste makers, or journalists to get a sense for the market.
2. Find interviewees. Here are 95 ways from Jason Evanish (http://is.gd/95ways)
3. Prepare your questions in advance. Make sure everyone asks roughly the same questions.
Good areas to ask about:
a. Ask them about themselves first, this sets them at ease and gives you context
b. Ask them about the problem you are trying to solve. You want this to be open-ended,
and you want rich details about how they describe the problem in their own words.
There are lots of ways to do this, including (pick one or two of these, don’t ask them
all!): What is the biggest issue associated with [your problem]? Describe your problem in
your own words… If you had a magic wand that could take away one part of the
problem, what could it be? Tell me about the last time you encountered this problem…
What’s the most annoying or frustrating thing about the problem?
c. Ask them about how they solve their problem now: How are you dealing with this now?
What are your workarounds? Describe to me how you deal with this? How have you
attempted to solve this issue? What do you like about this solution? What do you dislike?
Have you tried other approaches? How have you searched for solutions?
d. Only then should you talk about your solution. You should introduce the idea or
solution and let the interviewee ask questions about it. It is generally not worth asking
directly if they like it or not, and you should not be seeming to sell your product
4. Interview carefully. Make sure to listen and be quiet, resist the urge to talk too much. Also, ask
open ended followup questions: “what happened next?” “and then what?” etc.
5. Write up your results quickly. If you wait too long, you will forget the important details!
6. Bring all of your notes together and look for patterns or ideas that matter.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
2
Entrepreneur & Innovation Exchange
https://eiexchange.com
Do a trend analysis
The goal of a trend analysis is to get a quick picture of potential market demand and a sanity check on
your ideas. There are three quick tools you may want to use.
Google Trends
Google Trends (http://www.google.com/trends) gives you a quick snapshot of how often various terms
are searched of, as well as regional and time-based patterns. In this case, we can observe that the
market for sweaters is highly seasonal.
Use a Twitter Trend Analyzer
Though there are a number of options, two possibilities are Keyhole.co and Socialmention. Both provide
key word analysis using Twitter, letting you get a sense of how much “buzz” there is around a topic.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
3
Entrepreneur & Innovation Exchange
https://eiexchange.com
Google Keywords
Using Google Keywords, you can see what kinds of ideas are being searched for, and how competitive
the advertising space for particular markets are. This can give you a rough sense of both demand and
competition.
1. Log into Adwords Keyword Planner (https://adwords.google.com/KeywordPlanner)
2. Search for a new keyword (depending on your setup, you may have to launch a campaign for
$1/day to get access to this tool)
3. Use the keyword planner to examine your concepts.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
4
Entrepreneur & Innovation Exchange
https://eiexchange.com
4. Use the data from the planner to show you what related concepts are being searched for, and
how competitive the markets are
5. The Keyword ideas tab can show you related search concepts that might help you improve your
idea
6. You can also filter by location, language, and so on.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
5
Entrepreneur & Innovation Exchange
https://eiexchange.com
Do a Survey
In this case, our goal is to poll a group of people that we already have access to for details about their
response to a product or service idea. Because we are not doing careful sampling, you can’t draw any
large-scale conclusions based on this information. It can, however, be useful for getting quick feedback
from many people to help improve your idea and get a gut check on whether it is good.
1. Go to SurveyMonkey and log in.
2. Go to Create a Survey
3. You can either make your own survey or you can start with a template. Generally, making your
own survey is much more useful, but it requires effort to think of the right questions and test
them properly. It also depends on what you want to accomplish.
a. If you are early in the idea generation process, and you want to better understand a
problem people have, you will need to write your own questions.
b. If you already have an idea for a product or service, you can start with the Market
Research Template as a guide.
4. For the rest of this tutorial, I will assume that you are testing an idea, so select the Market
Research Template to start.
5. You are going to need to add a description of the proposed product or service at the top. Do
that by going to the Builder and adding either a text question or an intro page describing your
product and service. Include pictures or additional data if you can.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
6
Entrepreneur & Innovation Exchange
https://eiexchange.com
6. Make sure to delete or edit the questions that are not relevant to the survey. Twenty questions
is a rather long survey, so deleting the less relevant half is likely to increase your response rates.
7. Now go the Collect Responses Tab. The easiest way to administer the survey is just to send the
web link to your potential responders. You can also post it on Facebook using the button on the
page. You can also get a more statistically valid sample using the email feature that sends each
recipient their own link, or by buying responses.
8. If you are sending the emails to people you already know, make sure you understand how
representative they are of the market as a whole. Aim for at least 20-30 replies
9. The Analyze Results tab will give you data so far.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
7
Entrepreneur & Innovation Exchange
https://eiexchange.com
Do a quick preference test
The goal of a fast preference test is to get a statistically sound answer to a concrete question. This
works well for things like deciding between two different product designs to pursue, or getting
information about a single product feature.
1. Go to Google Consumer Surveys and log in
2. Create a new survey and pick an audience. You can pick from a variety of countries, regions, and
demographics.
3. Pick a question type, in this case, we are asking people a side-by-side question to decide which
of two product ideas sounds most appealing.
4. Launch the survey. You will have to wait about an hour for Google to approve the survey
question.
5. The results analysis tab has large amounts of detail about the results, including statistical tests
of whether your results were significant.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
8
Entrepreneur & Innovation Exchange
https://eiexchange.com
Do a smoke and A/B test
In this case, our goal is test a variety of pitches to see which one is most appealing.
In this case, we are interested in trying to find out how to pitch a fashionable waterproof tote bag
decorated with stylized pictures of common childhood germs. Proceeds of the sale of the bag go to a
children’s hospital.
We are going to do this in two parts. First, we are going to create a landing page, the “smoke test.” The
landing page is a single page that lets people sign up for more information about a product. By seeing
how many people sign up versus how many come to the page, you can get a sense of demand.
1) We are going to use LaunchRock, though there are other ways to do this as well. Go to
LaunchRock and choose Landing Page
2) Set up a landing page and choose a style and fill in the required forms. You will need a reason
for people to sign up for your site, so you may want to offer a discount, notification when a
product is available, etc. Please make sure you are being ethical, and not promising something
that will lead to disappointment or anger.
3) To create the smoke test, we will want some sort of background image that makes the landing
page look good and makes the product clear. I did this in a few minutes, and it is pretty bad, so
please forgive my terrible Photoshop skills.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
9
Entrepreneur & Innovation Exchange
https://eiexchange.com
4) Launch the page! You can monitor sign ups using the insights tab. You can also set up sharing
over Facebook, Twitter, etc.
The A/B test involves running an online ad campaign to drive traffic to your landing page. This is only the
basics using Google adwords. Importantly, there are some good reasons why you might have trouble
with Google Adwords due to their policies and costs – Google has cracked down on ads that lead to
landing pages. You may want to consider using Facebook, Bing, or Reddit for advertising instead. The
principles are the same.
1) Set up a Google Adwords account, and identify the keywords you will want to use (see keywords
in the advertising analysis above). More popular keywords are more expensive. Generally you
want to find narrow keywords that appeal to your audience.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
10
Entrepreneur & Innovation Exchange
https://eiexchange.com
2) Write your ads. Remember that you want to test a variety of possible pitches or ideas to see
which gets traction. Here we are trying to understand whether the social good pitch is actually
appealing to consumers. This is an example of some of the ads:
3) Choose how much you are going to pay per day to have your ads displayed. Usually $25/day will
give you results for less common keywords.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
11
Entrepreneur & Innovation Exchange
https://eiexchange.com
4) Track the results. You want to look at both which keywords are working and which ads are
working. The CTR shows you what percent of people seeing that ad click it. In this case, “Carry a
Bag, Help a Kid” is over three times more successful than its competitors.
You can also configure your landing page so it has multiple variations, and conduct an A/B test that
way as well.
This is just the basics! Be creative and consider other ways to get traffic to your landing page (blog
posts, social media, etc.). You can also conduct a survey on the landing page as well.
FIVE WAYS TO TEST IDEAS - ETHAN MOLLICK
www.eiexchange.com | Entrepreneur & Innovation Exchange
©Copyright Schulze Family Foundation
12