Paying for Beta Testers: A Case Study

Following the advice of Eric Ries in his just released book The Lean Startup (which is excellent and I highly recommend), I wanted to get inside the heads of my users to see where I was going wrong and why my engine of growth wasn’t turning. 

My real-time news aggregator, Newsfeedy, can’t seem to retain users, which is essential to growing my user base (the “sticky” engine of growth in the jargon of Eric Ries). Not only that, my web presence is so small that unlike Joel Spolsky, I can’t just launch a product and have an instant influx of willing early adopters to split-test and otherwise identify why my users won’t come back.

So I did what any normal person in America does these days: I outsourced it. Specifically I used Amazon’s fantastic Mechanical Turk platform to get people to use the site and give me feedback on it. I probably spent too much (this was my first time as a Turk user), but I got some good information which I can now share with you.

My setup was simple: I embedded Newsfeedy as an iframe in the Turk page, and had 6 questions for the people looking at it:

1. Do you have a Twitter account? (I was interested in the correlation between those that liked it and had Twitter)

2. Is the content interesting?

3. Is this website useful?

4. What is the best part?

5. What is the worst part?

6. Will you return to this website?

I’ve only run the experiment twice, with 10 respondents each time. I probably should do more, but I’m in no rush. The results were surprisingly positive. 14 out of 20 (70%) respondents said they liked the content and thought that it was a useful website. Others said they didn’t find the content interesting because the topics covered weren’t very interesting (there is a lot of celebrity “news” on Twitter and Google, so I can understand their sentiment).

So far so good, most people like it, some people aren’t particularly enthralled by its subject matter, which is to be expected.

The primary complaint about Newsfeedy was its design, which, although I’m not sure how to fix, is far from perfect I’m sure. So I should have a good starting point to increase my retention.

Now to the real question, which gives me a baseline metric from which I can solve my problem of retention: “Will you return to this website?” 16 out of 20 respondents said that they would! An unbelievable 80% of visitors to Newsfeedy (confidence interval of +/- 18% with a confidence level of 95%) said that they would return. 

My retention problem is solved right? All I have to do is dump money into Adwords or Facebook Ads or something and I’ll be all set!

Wait, hang on a sec, this is weird. 5 days later, and my traffic is still flat (and by flat I mean I’m the only one visiting. literally). Now, I don’t expect everyone who said they’d come back to return within 5 days, but the content of Newsfeedy is real-time and ephemeral. Yet no one has returned? But they liked it! And they said they would come back! I should have made them pinky-promise.

I feel betrayed by the Mechanical Turk users. But what should I expect? At the end of the day, I paid these people. So they aren’t really my target audience. Not only that, but what people say and what they do is very different. Apparently, 80+/-18% different.

What can you learn from all this? Paying for beta testers might be useful, but take the results with a heaping spoonful of salt. It’s much better to have real users as guinea pigs on which to test your engine of growth.

Maybe they’ll come back to me. I can only hope. But until then I’ll keep working on my retention problem. And probably throw some more money at Mechanical Turk, warts and all.

PS - see if you can solve my retention problem. Go to Newsfeedy and tell me in the comments why you wouldn’t come back. Or prove me wrong and come back several days in a row.

Introducing Newsfeedy: Real News in Real-Time

Newsfeedy is a website that pulls trending topics from different sources on the web and displays news headlines for them, all in one place.

Right now, the service includes trending topics from Twitter trends for the United States and Google Hot Searches for the US. The news headlines are pulled from Google News with supplementary information pulled from Twitter and Reddit.

I created Newsfeedy for myself as a way to quickly check on breaking news, since Twitter trending topics seemed to be an increasingly real-time source for news. But most of the Twitter topics didn’t mean anything to me without looking up the specific story on Google News. Newsfeedy solved that problem for me. When a friend of mine asked me where she could find it, I thought it might be a useful tool for others too.

It’s by no means perfect (load times are a problem that I’m still working on) and the trending topics don’t always refer to news headlines, but overall it provides a quick and easy way to catch up on breaking news in pretty close to real-time.

I’d love to hear any questions, comments, or suggestions for it.