Eric Ries

Empirically Validating The Lean Startup: Making A Startup Great By Choice

Every few years a new book is published extolling a new way of thinking about business management. Some of these books find their way into the canon of management thought, if perhaps only for a specific industry. Most management theories, however, are based primarily on anecdotal evidence from the author’s experience. Other success stories are sometimes cited from outside sources, but as any statistician would tell you, this is a case of choosing data which fits the model. As a result, we’ve built up a body of management knowledge which may or may not be empirically true, and is therefore susceptible to being swept away by the next fad.

In the world of Startups, I would say that Clayton Christensen’s The Innovator’s Dilemma, Geoffrey Moore’s Crossing the Chasm, and Steve Blank’s The Four Steps to Epiphany have all made it into the canon of management thought (even if most entrepreneurs refuse to call themselves managers). More recently, Eric Ries of The Lean Startup Movement, which itself is an offshoot of Steve Blank’s theories on Customer Development, published The Lean Startup, codifying the central tenets of the movement and highlighting specific examples where they have been applied (especially IMVU, where Ries and company first developed their theory). Time will tell if The Lean Startup will last as a management theory, but all signs so far seem to indicate that it will.

But is The Lean Startup empirically true, or only anecdotally so?

The answer lies in Jim Collins’ Great By Choice, also released in 2011, which examines the question “Why do some companies thrive in uncertainty, even chaos, while others do not?”. If you’re not familiar with Jim Collins’ previous works, they all follow a similar formula: comb through historical business data in search of companies which meet certain criteria (in this case, achieving 10X performance over a 22 year period of relatively tumultuous conditions) and compare them to their peers, teasing out what characteristics differentiates them in an attempt to explain what caused their relative success. While this method certainly has its shortcomings, it is heavily data-driven, in contrast to most other works on management theory.

The question answered by Great By Choice is eerily similar to the stated goal of The Lean Startup, which is to increase entrepreneurs’ odds of building a successful startup, defined by Ries as “a human institution designed to create a new product or service under conditions of extreme uncertainty.” (pg 27) Jim Collins has inadvertently created the perfect empirical dataset by which to test the principles of The Lean Startup by finding the differentiating traits of those human institutions which thrive in conditions of uncertainty.

So let’s examine what Ries sets out as the five principles of the Lean Startup (pgs 8-9) in light of the evidence presented by Collins:

1. Entrepreneurs are Everywhere

If a startup is any human institution that meets Ries’ definition, this principle validates this comparison of Ries’ theory to Collins’ findings.

2. Entrepreneurship is Management

Again, this principle simply validates the comparison; entrepreneurs are not just creating a product, they are building an institution that thrives in uncertainty.

3. Validated Learning

Ries explains that startups exist to learn, and they way to do that is through validated learning: small scale experiments to test different aspects of the product vision. This process is exhibited in Jim Collins’ “Shoot Bullets” approach in which a low-risk, low-reource method is used to calibrate a new product or service.

4. Build-Measure-Learn

This is the process that embodies Validated Learning above. The cycle for a startup is to build a product and measure it’s success, constantly iterating to improve it. This concept is validated in Collins’ “Shoot bullets, then cannonballs” approach in which successful institutions were able to experiment with new products and continually tweak them until they got them to a point of dead-on accuracy, which Ries might call Product-Market Fit. Collins’ firms, however, were shown to “Shoot cannonballs” after achieving Product-Market Fit, putting all of their resources behind the successful concept. The difference between the two works is likely due to the resources available to each typical firm, in Ries’ case, a resource-starved startup.

5. Innovation Accounting

Innovation Accounting is how startups can measure their progress. Ries describes most startups as needing to measure their performance against metrics which make up the startups engine of growth. Taken at a more abstract level, this focus on consistent, measurable performance is echoed by Collins’ 20 Mile March concept, in which firms adhered to certain rigorous standards of performance, which may or may not be financial, year in and year out. Stryker’s was 20% sales growth. Progressive’s a combined ratio of 96%. Likewise, Ries advises startups to determine the critical metrics in their businesses and focus on them.

Beyond the Lean Startup

Great By Choice contains a few other topics that are not addressed in The Lean Startup, or at least not in the same way, the most notable of which is Collins’ concept of leading above the Death Line, stockpiling resources for times of distress that are almost certainly around the corner in order to survive bad outcomes. This is perhaps what differentiates a startup from another institution in uncertain times: the inability to mitigate Death Line risk by stockpiling resources. This is probably also the source of a common piece of advice given to entrepreneurs: raise money before you need it. 

Collins’ book also extends beyond simply bringing a new product to market and describes how these firms continue to succeed in uncertain times by applying the same recipe over and over.


From an empirical perspective, it seems that the principles that Ries advises startups to employ while bringing a product to market in conditions of uncertainty are spot-on. This isn’t exactly a hard-hitting analysis, but I think it’s clear that Ries’ principles can result in success in uncertain times, at least according to Jim Collins’ analysis.

These two books in tandem form an essential guidebook to any entrepreneur and they validate each other’s findings even while looking through totally different lenses. Empirical validation of management theory is rare, and the existence of it here should be a call to action to implement these principles in your startup.

Have you read these works? Have you found contradictory evidence? Let’s discuss in the comments.

Paying for Beta Testers: A Case Study

Following the advice of Eric Ries in his just released book The Lean Startup (which is excellent and I highly recommend), I wanted to get inside the heads of my users to see where I was going wrong and why my engine of growth wasn’t turning. 

My real-time news aggregator, Newsfeedy, can’t seem to retain users, which is essential to growing my user base (the “sticky” engine of growth in the jargon of Eric Ries). Not only that, my web presence is so small that unlike Joel Spolsky, I can’t just launch a product and have an instant influx of willing early adopters to split-test and otherwise identify why my users won’t come back.

So I did what any normal person in America does these days: I outsourced it. Specifically I used Amazon’s fantastic Mechanical Turk platform to get people to use the site and give me feedback on it. I probably spent too much (this was my first time as a Turk user), but I got some good information which I can now share with you.

My setup was simple: I embedded Newsfeedy as an iframe in the Turk page, and had 6 questions for the people looking at it:

1. Do you have a Twitter account? (I was interested in the correlation between those that liked it and had Twitter)

2. Is the content interesting?

3. Is this website useful?

4. What is the best part?

5. What is the worst part?

6. Will you return to this website?

I’ve only run the experiment twice, with 10 respondents each time. I probably should do more, but I’m in no rush. The results were surprisingly positive. 14 out of 20 (70%) respondents said they liked the content and thought that it was a useful website. Others said they didn’t find the content interesting because the topics covered weren’t very interesting (there is a lot of celebrity “news” on Twitter and Google, so I can understand their sentiment).

So far so good, most people like it, some people aren’t particularly enthralled by its subject matter, which is to be expected.

The primary complaint about Newsfeedy was its design, which, although I’m not sure how to fix, is far from perfect I’m sure. So I should have a good starting point to increase my retention.

Now to the real question, which gives me a baseline metric from which I can solve my problem of retention: “Will you return to this website?” 16 out of 20 respondents said that they would! An unbelievable 80% of visitors to Newsfeedy (confidence interval of +/- 18% with a confidence level of 95%) said that they would return. 

My retention problem is solved right? All I have to do is dump money into Adwords or Facebook Ads or something and I’ll be all set!

Wait, hang on a sec, this is weird. 5 days later, and my traffic is still flat (and by flat I mean I’m the only one visiting. literally). Now, I don’t expect everyone who said they’d come back to return within 5 days, but the content of Newsfeedy is real-time and ephemeral. Yet no one has returned? But they liked it! And they said they would come back! I should have made them pinky-promise.

I feel betrayed by the Mechanical Turk users. But what should I expect? At the end of the day, I paid these people. So they aren’t really my target audience. Not only that, but what people say and what they do is very different. Apparently, 80+/-18% different.

What can you learn from all this? Paying for beta testers might be useful, but take the results with a heaping spoonful of salt. It’s much better to have real users as guinea pigs on which to test your engine of growth.

Maybe they’ll come back to me. I can only hope. But until then I’ll keep working on my retention problem. And probably throw some more money at Mechanical Turk, warts and all.

PS - see if you can solve my retention problem. Go to Newsfeedy and tell me in the comments why you wouldn’t come back. Or prove me wrong and come back several days in a row.