Following the advice of Eric Ries in his just released book The Lean Startup (which is excellent and I highly recommend), I wanted to get inside the heads of my users to see where I was going wrong and why my engine of growth wasn’t turning. 

My real-time news aggregator, Newsfeedy, can’t seem to retain users, which is essential to growing my user base (the “sticky” engine of growth in the jargon of Eric Ries). Not only that, my web presence is so small that unlike Joel Spolsky, I can’t just launch a product and have an instant influx of willing early adopters to split-test and otherwise identify why my users won’t come back.

So I did what any normal person in America does these days: I outsourced it. Specifically I used Amazon’s fantastic Mechanical Turk platform to get people to use the site and give me feedback on it. I probably spent too much (this was my first time as a Turk user), but I got some good information which I can now share with you.

My setup was simple: I embedded Newsfeedy as an iframe in the Turk page, and had 6 questions for the people looking at it:

1. Do you have a Twitter account? (I was interested in the correlation between those that liked it and had Twitter)

2. Is the content interesting?

3. Is this website useful?

4. What is the best part?

5. What is the worst part?

6. Will you return to this website?

I’ve only run the experiment twice, with 10 respondents each time. I probably should do more, but I’m in no rush. The results were surprisingly positive. 14 out of 20 (70%) respondents said they liked the content and thought that it was a useful website. Others said they didn’t find the content interesting because the topics covered weren’t very interesting (there is a lot of celebrity “news” on Twitter and Google, so I can understand their sentiment).

So far so good, most people like it, some people aren’t particularly enthralled by its subject matter, which is to be expected.

The primary complaint about Newsfeedy was its design, which, although I’m not sure how to fix, is far from perfect I’m sure. So I should have a good starting point to increase my retention.

Now to the real question, which gives me a baseline metric from which I can solve my problem of retention: “Will you return to this website?” 16 out of 20 respondents said that they would! An unbelievable 80% of visitors to Newsfeedy (confidence interval of +/- 18% with a confidence level of 95%) said that they would return. 

My retention problem is solved right? All I have to do is dump money into Adwords or Facebook Ads or something and I’ll be all set!

Wait, hang on a sec, this is weird. 5 days later, and my traffic is still flat (and by flat I mean I’m the only one visiting. literally). Now, I don’t expect everyone who said they’d come back to return within 5 days, but the content of Newsfeedy is real-time and ephemeral. Yet no one has returned? But they liked it! And they said they would come back! I should have made them pinky-promise.

I feel betrayed by the Mechanical Turk users. But what should I expect? At the end of the day, I paid these people. So they aren’t really my target audience. Not only that, but what people say and what they do is very different. Apparently, 80+/-18% different.

What can you learn from all this? Paying for beta testers might be useful, but take the results with a heaping spoonful of salt. It’s much better to have real users as guinea pigs on which to test your engine of growth.

Maybe they’ll come back to me. I can only hope. But until then I’ll keep working on my retention problem. And probably throw some more money at Mechanical Turk, warts and all.

PS - see if you can solve my retention problem. Go to Newsfeedy and tell me in the comments why you wouldn’t come back. Or prove me wrong and come back several days in a row.