The Rise of the Machines

Unbundling the Big Ten: John McCain vs. Jim Delany

Thu, 09 May 2013 16:13:52 -0400

John McCain is trying to force cable tv providers to unbundle their channels, either into much smaller packages or even individual channels. A lot of powerful people make a lot of money on cable bundling, and McCain has tried a similar bill in the past, so it’s unlikely that this piece of legislation will the cause unbundling of TV. However, it seems only a matter of time until unbundling happens, either due to market pressure from consumers who would rather cut the cord than pay for channels they don’t watch, or from legislation like the bill McCain proposed.

As the linked AllThingsD article mentions, such unbundling will have wide reaching effects on the economics of content providers. The Big Ten is one such content provider [1], and to understand just how much unbundling could hurt the Big Ten, we need to take a look at the most recent conference realignment.

We just finished the largest college sports conference realignment in history. The reasons for it are unsurprising: money and football money. But for conference expansion to work (i.e. be profitable for the existing member schools) the addition of an institution has to result in more revenue than the average existing member produced prior to the expansion. This can happen by bringing a larger, higher revenue-generating program into the fold, or by taking advantage of the “whole is greater than the sum of its parts” activities. In this light, the Big Ten’s addition of Nebraska in 2010 makes perfect sense - Nebraska’s football product is historically better than the average Big Ten team [2], their fan base (and as a result the number of television viewers they bring to any given Big Ten matchup) is larger than the average Big Ten school, and with the expansion from 11 to 12 schools, the Big Ten was able to sponsor a football championship game [3], which is extraordinarily lucrative for conferences [4].

But what about the Big Ten’s more recent additions of Maryland and Rutgers? On the surface, they don’t provide any of the benefits that a school like Nebraska does - relatively small fan bases, historically less than stellar on-field performance [5], and no new events to air. The real key to understanding the addition of Maryland and Rutgers lies in understanding TV bundles.

The Big Ten created the Big Ten Network (BTN) in 2006 to air Big Ten events that didn’t make it to air on traditional broadcast or cable networks. Market economics would suggest that if these events were popular enough to be on the air they already would be - but market economics hardly touches an oligopoly like cable TV. Instead, the Big Ten demonstrates to cable TV distributors that in a particular market, like Chicago/Cleveland/Columbus, a certain percentage of their clientele are Big Ten fans and would like access to BTN. The cable network then puts BTN into their basic or expanded basic package, and BTN takes a cut of every subscriber (about $0.70 - $1.00) [6]. For subscribers outside of areas where the Big Ten has a large presence, BTN takes a much smaller cut (about $0.05 - $0.10) [6], and only of subscribers to a premium sports tier.

BTN has shown itself to be a huge moneymaker for the conference [7]. But based on the math of their subscriber fees, they stand to make a lot more money if they can get television markets outside of the 8 traditional Big Ten states to include BTN in the basic or expanded basic package - a huge boost in the number of subscribers, as well as the per-subscriber fee. The only way to convince cable providers to include BTN is to expand the Big Ten’s footprint into new, populous television markets.

That’s where the Rutgers and Marlyand expansion comes in: they give the Big Ten access to the lucrative New York and DC television markets [8]. Neither Rutgers nor Maryland are the dominant college sports team in those cities (if there even is one), but they allow the Big Ten to turn a small number of premium sports subscribers, at $0.10 a pop, into a huge number of basic cable subscribers at $1.00 each. And so the Big Ten’s pie is expanded before being sliced for the new members.

Problems for the Big Ten, however, lurk just around the corner; if cable unbundling happens, either due to legislation or market pressures, and BTN is sold a la carte, they lose the vast majority of the premium sports subscribers in markets outside the 8 Big Ten states, and in the newly expanded markets like DC and New York, they only have access to subscribers who are huge Marlyand or Rutgers fans - a tiny fraction of the number of basic cable subscribers. It’s unclear whether the BTN would even be profitable under an a la carte system, let alone the cash cow it’s proven to be so far.

And so Jim Delany finds himself with an unlikely foe; if John McCain’s bill passes, the economics of the BTN, which were the underpinnings of conference expansion, stop working, and the newly added members of the Big Ten only serve to dilute the quality of the football product of the nation’s oldest conference.

Even if content providers like the Big Ten get their way and defeat this bill, which is likely, it seems to be only a matter of time until cable unbundling happens. When it does, we’ll likely be in for another round of conference shuffling as schools try to figure out how to finance costly cross-country sports programs that rely on huge payouts from conference TV networks.

[1] The Big Ten is not alone here, they were just the primary instigator of the conference expansion. Both the Pac-10/12 and SEC are trying a similar play, with the Pac 12 creating their own television networks while cutting out a large equity partner like Fox.

[2] 5 national championships, 10 undefeated seasons, 43 conference championships, 53 consensus All-Americans.

[3] NCAA rules stipulate that a conference has to have at least 12 members to have a championship game

[4] According to ESPN, Fox is paying $145m over six years for the Big Ten football championship game alone

[5] Rutgers is 5-8 in bowl games,and has never gone to a major bowl. It won a Big East Championship in 2012. Maryland is 1-4 in major bowls, and 11-11-2 in all bowls, with 2 national championships.

[6] Reported figures vary, and probably fluctuate by television market and provider, but the figures I’ve seen are $0.88 and $0.05, $0.70 and $0.10, $1.10 and $0.10

[7] It has contributed $42.5m to each member school since its inception in 2006

[8] #1 and #8, respectively. Baltimore (#27) and New Jersey are also contributors.

To the Cloud!

Wed, 08 May 2013 12:06:00 -0400

SAP is committing to the cloud. For those who don’t know, SAP has a huge business of on-premises ERP software (something like 80% of the Fortune 500 run SAP). This software is customized and implemented by huge tech consulting firms, the largest of which are Deloitte, Accenture, and IBM. (I used to work for Deloitte doing SAP implementations.)

The move of ERP software to the cloud was inevitable. Over the next decade we’ll see more and more software, both enterprise and consumer, move to the cloud. The only question will be who is willing and able to disrupt their own business, as SAP is attempting to do here, and who gets replaced by newer products and companies that are entirely focused on cloud solutions, like Salesforce.com and Workday.

Part of the promise of cloud-based software is the cost savings that result from elimination of customization. Enterprise customers all believe that their businesses are unique and beautiful snowflakes, and they pay out the nose to make sure that their software is too. That kind of customization is extremely lucrative for implementers and consultants, but extremely costly for the customer, both in the implementation and in ongoing maintenance.

One of the most frustrating things about my time doing SAP implementations for Deloitte was that start-from-scratch approach to every project and client. The firm had implemented for nearly every kind of company in every industry, and yet every project was treated as a complete customization. Is Kraft’s business so different from Nestle’s that they require totally different software? Is John Deere so different from Caterpillar? [1]

That’s part of the reason that I saw disruption from the cloud-based solutions coming from a mile away - there was just so much room for improvement, especially in areas that the cloud is so well-suited to handle: scalability [2], maintenance [3], and most of all reduced customization (also known as software that “just works” out of the box, er, browser).

SAP has recognized the potential for cost-savings that comes with reduced customization. Vishal Sikka is quoted in the article as saying:

At some point in the future, complex implementations should go away.

With their purchase of SuccessFactors and this most recent announcement, SAP has stated their intention to cannibalize their own on-premises software business in order to save themselves in the long term. But with this move to the cloud, and the focus on more vanilla distributions of software, what happens to the integrators for whom these complex SAP implementations are a cash cow [4]?

When I posed this question to the head of Deloitte’s SAP practice at a Q&A session for new analysts almost two years ago, he told me (paraphrased):

It doesn’t matter. The economics for Deloitte don’t change whether the software is on-premises or in the cloud.

Only time will tell.

[1] I didn’t perform implementations for these companies, and their businesses might indeed be very different from each other, but not so different that they can’t have the same or substantially similar software.

[2] Anecdotally, I experienced scaling issues with SAP at my client that resulted in downtime on the project while they put more machines on it. This wouldn’t be a big driver for most large enterprises, but it is an issue.

[3] Aside from the initial implementation, the big money for the tech consulting firms (and big expense for clients) is in upgrades to newer versions of SAP

[4] SAP alone made up around a quarter of Deloitte’s technology consulting business when I was there. Oracle was roughly another 20-25%

Impress: Express in the browser

Sat, 04 May 2013 12:37:51 -0400

A couple of weeks ago a friend of mine and I were discussing the potential for sharing code between the front- and back-end when using Node.js. With the explosion in popularity both of Node and front-end Javascript frameworks, code-sharing across environments seems like a natural course of action to maximize developer happiness.

Since we both use Express frequently, we started kicking around the possibility of using it in the browser - capturing all the actions that would normally lead to an HTTP request (clicks on links, form submissions) capturing them, and processing them the same way that Express does.

Last weekend, I threw together Impress, a front-end Javascript framework that has (very) stripped down Express functionality [1]. With the help of my Node.js-style module-loading library Joules, you can build what looks like an Express application entirely on the front-end.

In building that, I learned a lot about why this was a bad idea:

Rendering an entire response (the way that res.render or res.send does) doesn’t make sense in the browser - you rarely want to refresh the entire screen if you don’t have to.
URL’s are only useful as a way to tie together the front-end and back-end. If you’re dealing with just the front-end, URL’s are a very indirect way to define events (e.g. clicks, form submissions) and event handlers (e.g. app.get).
In order to really share code between the front- and back-end, you would have to maintain a different set of API endpoints for database access that both your front- and back-end talk to. CouchDB would be a good fit for this, but if you’re using something like MongoDB/Mongoose, you’re looking at defining an entirely separate set of routes to handle - not exactly minimizing work.

In discovering why Express was a bad fit for the front-end, I began to wonder if it’s really the best fit for the server. It feels right, but I suspect that’s mostly because it borrows paradigms from popular Ruby web frameworks (Sinatra and Rails).

Javascript and Node.js are heavily event-driven, so I’m beginning to think that it might be better to design the entire web server around event handling. In that case, a URL would just be one particular type of event. That mindset makes sense - when a user clicks a button it’s because they want to do something. Whether we listen for that event on the front-end (button.on("click", handleButtonClick)) or the back-end (app.get('/button/link/location', handleButtonClick)) we’re really just interpreting and acting on user intent.

If you know of any frameworks that work that way, let me know on Twitter.

1 - The biggest missing piece is the lack of support for middleware, which is probably the biggest benefit of using something like Express/Connect

Why isn't all email encrypted?

Fri, 12 Apr 2013 13:12:32 -0400

More and more of our lives are online now, and more of our communication is happening online. The most ubiquitous communication mechanism is email, and, for better or worse, more people are putting more sensitive information into their emails every day.

With HTTPS becoming a near standard for accessing popular websites (and for good reason) I got to thinking about email security. STARTTLS is used by most major email providers (like Gmail), but that only serves to encrypt messages from relay to relay - it doesn’t provide true end-to-end encryption, from sender to recipient.

With Google Apps taking over the nearly the entire University email client ecosystem, I think it’s safe to say that in the near future, if not already, almost all email will be conducted through a cloud provider like Gmail. When it comes to sending unencrypted messages, you can choose to trust your cloud provider (and all the relays between them and your intended recipient), but I think that’s a mistake because:

Providers have a profit motive that isn’t always in your favor (e.g. contextual ads based on the contents of your messages)
Cloud providers have a less than stellar security record
Good security exists in layers
Trust is not security

Those wearing tinfoil hats right now won’t see the problem because they already use PGP to encrypt all their emails. However PGP, with its near military-grade encryption available to the public, is not a mainstream solution in its current state. We need regular people - the same people who don’t know what HTTPS is beyond a green indicator in the address bar - to be able to communicate securely.

SafeGmail is an interesting take on this problem, and requires fairly seamless, strong security, but it requires sharing of a password of passphrase ahead of time, not exactly something that can happen every day.

I think the reason our email remains unencrypted is that we don’t have a scalable solution for person-to-person encryption. I think PGP is the answer, but we need to find a way to give everyone, including my grandmother, a PGP key that can encrypt and decrypt her communications effortlessly.

Login-less Apps

Fri, 29 Mar 2013 01:35:59 -0400

Identity on the web is a big problem. We have logins for every website we visit, and only the most savvy among us use a password manager to make sure that the inevitable database leak doesn’t burn us everywhere.

There have been attempts to solve this problem - OpenID for one, Facebook Connect (as well as the similar services from Twitter and LinkedIn), and Mozilla’s new Persona, which is the first real alternative to a traditional login system.

But I saw today a really interesting paradigm - the login-less app. Emmanuel Bégué created Urgeous, a Posterous clone, which is primarily focused on post-by-email, one of Posterous’s most interesting features.

Since he was implementing post-by-email, and email is the default definition of identity, he got rid of a login system altogether. Instead, everything sent from your email address is a post on your behalf. To edit posts you use a link with a unique code that is generated and sent to you when a post is published.

This is a very specific scenario, and one in which you don’t need to rely on a persistent state, but it brings up an interesting point - do we really need all these login systems? Does Quora really need me to login to read beyond the first answer?

It’s almost reflex to include a login system when building a new web app, but we as developers would be wise to really consider the needs of our application before adding yet another layer of broken login systems onto our users.

My Problem With Javascript-only Apps

Wed, 20 Mar 2013 15:10:00 -0400

I love Javascript. I code almost exclusively using Javascript. My startup’s entire stack is Javascript (MongoDB + Node + jQuery + Hand-rolled front-end libs).

But I hate apps that require Javascript to load content. I’m looking at you Backbone.js/AngularJS/Ember.js.

I understand that the increasing speeds of SpiderMonkey and V8 (and even Chakra) have turned the web into a place where you can build client-side web apps with much richer functionality. And I understand why that has led us to more and more “single-page” apps that load additional data and render it using on-page templates.

What I don’t understand is a reliance on Javascript to load content. Not rich functionality, not drag-n-drop WYSIWYG editing, not even AJAX-y autocompletions, but plain old content. And not just as the primary choice to provide snappy responses once the page has been loaded once, but the only way to load content.

Earlier today, I was checking out the popular article on Hacker News about Ember.js’s Quick Start shortcomings. It was a forum post on the new Discourse platform from the Stack Exchange folks, which was itself built on Ember.js.

Here’s what I saw:

A blank page.

I checked the errors, and it looks like something failed to load.

When I checked the source, I see that Discourse has some nice backwards-compatibility for people that don’t have Javascript enabled — they include the entire rendered HTML page in the <noscript> tags. But for those of us that have javascript enabled, but for whatever reason it doesn’t load properly? No forum for you.

The irony to me, is when looking at Discourse’s about page:

The state of forums has been unchanged for so long that forums are considered unworkable and undesirable; few sites want forums any more because the software is so poor.

In my book, those old forums worked, as ugly as they were, and working and ugly is better than pretty and not working. All your fancy live-updates, inline editing, etc, simply don’t matter when the content doesn’t load. As has been said a million times before, Content Is King.

I don’t mean to pick on Discourse here. They built what is I’m sure a great product, which is itself built on a great framework by very smart people.

The problem is in the assumption that rich functionality and traditional HTML rendering have to be divorced. That if we want all the fancy rich-client stuff, we can’t deliver rendered content to the client, we must render content client-side.

I don’t think that’s true. I think you can build apps that deliver rendered HTML content to the browser, and add rich functionality on top of that, hopefully with appropriate fallbacks for the most important actions.

Doing so provides a number of benefits:

Faster startup time - “Time to first tweet” is Twitter’s benchmark, and the best way to ace such a benchmark is by delivering rendered HTML to the browser on first page load
Graceful failure - No blank screens like that I one I got today at discuss.emberjs.com
Search engine friendliness - The big crawlers are supposedly going to get better at rendering Javascript to index pages, but why would you make it harder to index your site?
Smaller payloads - With Discourse’s noscript tags, they are delivering both the data and the templates to the user’s browser twice

When building a true app, one in which the functionality instead of the content is most important, the exclusive reliance on client-side Javascript makes sense. But any web app for which content plays an important role, loading that content with Javascript is the wrong choice.

Fixing "Error: EMFILE, too many open files" in Node.js

Fri, 08 Mar 2013 11:10:00 -0500

At Endorse.me, we use Amazon S3 as a CDN for all our static assets - images, scripts, and css. To make sure that we’re only uploading new assets to Amazon, and to make sure that we’re always using the latest versions on the production site, we fingerprint all of our static assets.

The result is that, using Node.js’s asynchronous filesystem functions, we’re opening nearly all of our static files at once to create hashes for the fingerprints. I recently ran up against OSX’s default maxfiles setting of 256, which resulted in the descriptive Error: EMFILE, too many open files error, crashing the Node process.

I didn’t find a good solution, either from the trusty StackOverflow, the Node.js docs, Google, or the #nodejs channel on IRC. It seems this isn’t a common enough use case to warrant a best practice.

Based on this StackOverflow answer, I created my own replacement for some of the most common fs methods that avoids the EMFILE error. It does this by keeping track of the total number of files opened at once by the module, and queuing up those that will take us over a predefined limit (200 by default).

A solution like this (as opposed to say, large batches) allows us to keep the maximum concurrency that the filesystem allows without changing the underlying pattern of opening, reading, and writing files using Node’s built-in fs module.

Using my module, reading files from a directory looks the same, just without the crash:

var Filequeue = require('filequeue');
var fq = new Filequeue(200); // max number of files to open at once

fq.readdir('/path/to/files/', function(err, files) {
    if(err) {
        throw err;
    }
    files.forEach(function(file) {
        fq.readFile('/path/to/files/' + file, function(err, data) {
            // do something besides crash
        }
    });
});

The module, Filequeue, is available on NPM and Github, and is MIT Licensed. Let me know if you have any questions or suggestions for improvement.

Versioning in Mongoose

Thu, 28 Feb 2013 13:58:00 -0500

Today I encountered a interesting piece of the Mongoose internals added in v3. In a nutshell, Mongoose uses versioning to prevent you from accidentally updating the wrong element when editing a member of an Array.

As Aaron Heckmann explains:

To see how this could be problematic we need to take a closer look at the underlying operation used to update the comment. When post.save() is executed, an update is issued to MongoDB that looks like the following:

posts.update({ _id: postId } , { $set: { 'comments.3.body': updatedText }})

Notice comments.3.body, this is called positional notation. This tells MongoDB to set the body of the comment in the comments array at index position 3 to the updated text.

If, for instance, while you were editing comments.3 in the example above, a new comment was added, then comments.3 would not be the correct comment, and your update would likely be successful, but not in the way you intended.

Mongoose’s solution to this problem is to add a version number the document that is incremented every time the array changes. If you try to update an array element using positional notation with the wrong version key, you’ll get a nice VersionError. The text of the error is “No Matching Document Found”, which isn’t very helpful, but the ability to test for instanceof mongooose.Error.VersionError is quite useful.

In my case, I just test for the VersionError, and re-fetch the document and re-update the array if I encounter one. I put a limit on the number of times this loop can continue so as to not accidentally melt my database.

I can see how, at a huge scale, something like this might become an issue if you’re adding and and updating a lot of subdocuments concurrently, but for my case, it works like a dream.

Email management treats the symptoms, not the disease

Wed, 13 Feb 2013 11:14:13 -0500

I signed up for Mailbox on the first day of its release and found myself sitting in line near position 280,000. As the number ahead of me has slowly wound down, and the number behind me has slowly grown, I’ve had a quite a bit of time to think about Mailbox, and what it really means.

There are more than three-quarters of a million people waiting for access to Mailbox. And since only a few people have actually used the app and can recommend it on the basis of its functionality, it seems that all those people are waiting in line for an idea, a cure for their email problem.

I haven’t used the app myself, but from what I’ve heard it is a tool to formalize the usage of Inbox Zero. Inbox Zero is a management philosophy for incoming email. As Merlin Mann, creator of Inbox Zero, explains:

Clearly, the problem of email overload is taking a toll on all our time, productivity, and sanity, mainly because most of us lack a cohesive system for processing our messages and converting them into appropriate actions as quickly as possible.

Inbox Zero proposes itself to be just that, a “cohesive system for processing our messages”, and Mailbox is the tool by which that system is accomplished. If Inbox Zero describes how to put together a house, Mailbox is the hammer and nails.

But not everyone agrees that the problem with email is that we lack a cohesive system. Paul Graham, in his essay on Frighteningly Ambitious Startup Ideas, says that the way to replace email, is to replace the inbox with a better todo list.

Email was not designed to be used the way we use it now. Email is not a messaging protocol. It’s a todo list. Or rather, my inbox is a todo list, and email is the way things get onto it.

One of his suggestions for email’s replacement is that it should give more power to the recipient.

I want there to be more restrictions on what someone can put on my todo list. And when someone can put something on my todo list, I want them to tell me more about what they want from me. Do they want me to do something beyond just reading some text? How important is it?

We already have some of the infrastructure in place in email itself to support what Graham is asking for. Specifically the “Importance” and “Reply-by” headers exist as defined in RFC 1327. As far as I can tell, those flags are largely unused. As Cory Doctorow notes (albeit on a different topic) in Epoch:

if you let a coder (or, ::shudder::, a user) specify the importance of her alert, give her a little pull-down menu that has choices ranging from “nice to know” to “white-hot urgent,” and nine times out of ten, she’ll choose “NOW NOW NOW URGENT ZOMGWEREALLGONNADIE!”

So, Boris from The Next Web took Graham’s suggestion one step further, and he requested additional metadata from his senders via his own personal InboxPro. This data allows him to easily address certain types of emails, like Yes/No questions, without asking questions about the priority of the message.

But do all these management systems and apps really solve the problem?

Put another way, what exactly is the problem?

Merlin Mann believes it is the lack of a cohesive system to manage incoming messages.
Paul Graham believes we’re using a messaging protocol where a todo list protocol would be better.
Boris believes we don’t have enough information about incoming messages before we manually parse them.

But could the problem be the sheer volume of email itself? The Slow Web Movement, a reaction against the increasingly real-time nature of the Web, seems to indicate that there are people who value not being beholden to the real-time paradigm. More and more, it seems that email is turning into a constant stream of incoming messages, and while it’s not real-time, it shares many of the same problems, and is antithetical to the Slow Web’s philosophy: “users should have a life.”

Every new email management app, and every new email management system seems to be treating a symptom of the disease rather than the disease itself. I’m not sure if the disease is the increasing amount of email, but I can be fairly certain that the disease is not the lack of a management system, nor a todo list communicated through messages.

Before we try to “fix email”, let’s figure out what’s wrong with it. Let’s diagnose the disease.

MongoDB: find a document that matches any element in an array

Tue, 22 Jan 2013 17:15:06 -0500

I recently ran into a problem while developing some of our advanced search capabilities on Endorse.me: I wanted to find all the documents in our MongoDB database that had at least one of a number of elements as a property. This is fairly easy to do using MongoDB’s ‘$in’ operator when the property is a primitive, like a string:

https://gist.github.com/4599048

But what if one of our toys has multiple colors, listed in an array? I searched around for a some sort of $any operator before looking again at the MongoDB documentation for $in. Turns out, if the field has an array, it behaves exactly how I was hoping it would!

https://gist.github.com/4599072

"The managerial attendees were, in every case, decked out with multiple high tech gadgets while the..."

Fri, 04 Jan 2013 10:47:19 -0500

“The managerial attendees were, in every case, decked out with multiple high tech gadgets while the technical types all used pen and paper for note taking.”

- Fogus, on a Hacker News discussion on Pen & Paper

"Your house is not an asset. It is a hedge."

Tue, 11 Dec 2012 12:42:00 -0500

"Your house is not an asset. It is a hedge.":

I don’t completely agree with the analogy that you are naturally “short” housing, it’s probably more accurate to say that you have a natural exposure to housing prices since they are an essential input for your life (although I would have to revisit my college Investment Theory class to say that with real conviction).

This article does rightly point out though that your house does not act as an asset for you, it simply reduces your exposure to housing prices by locking in a fixed price of housing.

I don’t agree that you shouldn’t consider it a part of your total assets (it most certainly is, as are futures contracts purchased as hedges) but you should also consider your yearly need for housing as a definite liability. Better yet, you should create a retirement plan with an investment professional.

Two Things About Conditionals In Javascript

Mon, 10 Dec 2012 23:12:28 -0500

Two Things About Conditionals In Javascript:

The first “thing” in this post is the most interesting: in Javascript, there is no “else if”. After working in Ruby for awhile, where they use the almost-english symbol “elsif”, I was wondering why “else if” in Javascript was two symbols instead of one. Turns out, it’s not an “else if” symbol, it’s merely an “else”, with no curly brace, followed by an “if” block.

You learn something new every day. (As is to be expected, this is well-documented on MDN. But when something works the way you expect it to, you don’t always look into why.)

Photo

Sun, 02 Dec 2012 21:11:49 -0500

Accidental Denial-of-Service

Wed, 14 Nov 2012 16:58:43 -0500

When pushing a bug fix to the Endorse.me servers this morning, I noticed that the site was responding extraordinarily slowly. A quick check in the logs showed that we were being flooded with requests from a single IP address for a single resource (a Flash/SWF file that we use to allow users to copy text to the clipboard) that was hogging almost an entire web process on Heroku.

I quickly spun up another dyno on Heroku (cloud computing FTW) and looked a little bit deeper into the issue. I emailed the user who’s account seemed to be the source, but received no response. Some light googling didn’t reveal any easy ways to block specific IP addresses using Heroku/Node.js (I found some ways using Rack), and I started considering just how bad it would be to continue to throw new dynos at the problem.

I moved the SWF file to our CDN, where it should have been all along. For convenience in development, I had kept it local, thinking “what’s the worst that could happen with ONE static asset?” Sigh.

Eventually I signed us up for CloudFlare which routes all of our traffic through their network, allowing me to block specific IP’s. Their sign-up process was completely painless aside from the DNS propagation which is unavoidable.

As soon as we were set up, I blocked the IP, and everything went back to normal. I was able to scale our Heroku setup back down again, and actually read my logs.

Now, I’m inclined to think that this whole thing was an accident for a few reasons:

The User Agent string indicated it was from Chrome
The requests were for a SWF file (which seems like it could be a bug, it wasn’t even a big file)
While there were a lot of requests, it wasn’t even close to actually crippling us or taking us down.

If anyone else has run into this issue before and knows a better long term fix without blocking an IP, please let me know.

A Founder’s Constant State of Rejection

Tue, 13 Nov 2012 12:41:18 -0500

A Founder’s Constant State of Rejection:

Articles like this help you cope with the almost bipolar state of mind of being a founder. Just knowing that there are other people out there going through the exact same violent swings of fate that you are helps keep you sane.

As I’ve come to saying lately, there are great days, and there are terrible days. In a start up, instead of “two steps forward, one step back,” it’s often times “two jumps forward, one haymaker to the face.” But you have to stay in the ring to win the fight.

minming: A billion dollar software tech company is founded every 3 months *in U.S.

Mon, 12 Nov 2012 12:12:37 -0500

minming: A billion dollar software tech company is founded every 3 months *in U.S.:

lominming:

Ron Conway once said: “There’s a billion company created every 3 months. Now, it is every 2 months”. I finally got the chance compile “The Billion Dollar” list. I tried to limit this list to only software tech companies. I also included a couple of near billion dollar companies (~0.7 B and above).

Photo

Tue, 23 Oct 2012 14:48:56 -0400

Survivorship Bias in Startups

Tue, 23 Oct 2012 11:55:45 -0400

Confluence, from Hacker News, makes a great argument for why all advice given by successful startup founders should be taken with a grain of salt - statistical variability. Humans tend to create narratives around events, thereby turning random outcomes into events with perceived predictability.

I myself have been guilty of reading and listening to startup founders tell their stories to try to find a pattern that I can emulate for success (Jim Collins is a guilty pleasure). But the truth is probably closer to what confluence is preaching - there is a very limited set of commonalities in startup founders, and a lot of what we interpret as skill or foresight is more likely luck.

It is a must-read comment for anyone involved in startups. A choice quote:

[Taking lessons from successful startup founders is] like taking lessons from survivors of the Titanic on how to survive the sinking of a ship. It’s quite simple - be a young female child with a life vest and rich parents (or in startup land - a young upper-middle class male living in California during a venture bubble, a cyclical investment in the Valley with a convergence of secondary technologies, above average intelligence and a college degree from a reputable university).

http://news.ycombinator.org/item?id=4685928

Bukowski: People Simply Empty Out

Fri, 19 Oct 2012 16:33:55 -0400

Bukowski: People Simply Empty Out:

An excerpt from Bukowski’s letter to his publisher 15 years after leaving the workforce at age 49:

And what hurts is the steadily diminishing humanity of those fighting to hold jobs they don’t want but fear the alternative worse. People simply empty out. They are bodies with fearful and obedient minds. The color leaves the eye. The voice becomes ugly. And the body. The hair. The fingernails. The shoes. Everything does.

As a young man I could not believe that people could give their lives over to those conditions. As an old man, I still can’t believe it. What do they do it for? Sex? TV? An automobile on monthly payments? Or children? Children who are just going to do the same things that they did?

Full Letter