The Rise of the Machines

With the PRISM scandal in the news, my attention was brought back to an issue that was nagging me a few months ago: even with fast computers in the palm of everyone’s hand and near ubiquitous networking, we don’t use strong encryption in our everyday lives.

For me personally, and I suspect many other Americans, the things I care most about are in my Gmail and Dropbox accounts. For the most part, we trust Google and Dropbox (and others) to keep that data safe and private. PRISM invalidates that trust. It’s unclear the extent to which these organizations participated in PRISM, or even what “participation” means, but it is clear that with FISA, even if they were handing over your data, they wouldn’t be able to tell you about it.

Even without Google’s compliance, if the NSA (or any other party) controls one the email relays between the sender and the recipient, it can read the contents of an email message, as SMTP lacks end-to-end encryption.

That revelation turned my beef with the lack of everyday encryption from a minor annoyance into a serious pain point.

We have an open source, near-military-grade encryption standard in OpenPGP available, but it seems that only the most paranoid and tech-savvy take advantage of it. Why isn’t PGP more widespread?

The Web of Trust can’t grow effectively. In order for Public Key Cryptography to work, you have to trust that a user’s public key is truly the public key of that user, and not someone impersonating that user (or, less maliciously an old public key). PGP solves this problem in a decentralized manner, having users sign each other’s identity certificates to certify them as trusted. If several people you trust have signed another user’s certificate, you can be pretty confident that the identity is accurate. This works great if everyone uses PGP, but at anything but near complete adoption  it runs into issues. New users (with no one to sign their certificates) can’t be trusted at all, and two users without a common connection have no way to secure transmissions.

Too much human intervention. There are some good plugins to make email encryption fairly painless, but it requires that the end user find and maintain a set of public keys for their contacts. There are public key servers to help with the distribution of keys, but they are intended for human users to find the proper key, and encrypt their message with it. This is far too much human intervention, especially for a cryptography solution, to ever become a mainstream activity.

With these issues in mind, I came up with the concept of Authoritative Key Servers: PGP Key Servers designated as the holder of up-to-date and accurate public keys for the users of a domain.

I describe HTTP Authoritative Keyserver Protocol in more detail here, but it essentially allows you to designate a keyserver as authoritative for a particular domain in the DNS records. The designated server will then respond to requests for a particular user’s public key block, and the response can be trusted as up-to-date and accurate.

As a result, if a sender and a recipient have no prior communication and no common links, the sender can trust the DNS records of the recipient’s domain in order to send secure communications, and with a simple plugin, can do so automatically, without user intervention.

This allows adoption of a strong encryption standard to happen at a natural pace - I can use this scheme today, and anyone can send me secure communications [1], but I can still communicate with those that have yet to adopt it.

I created an open source implementation of my Authoritative Keyserver, and anyone can set up their own key server. Alternatively, if you don’t want to host your own keyserver, you can designate my key server (keys.tgriff3.com) as authoritative and send me your public key block.

For email encryption, the creation of a Gmail plugin that automatically looks for an Authoritative Keyserver and encrypts the message contents is the next step.

I’m by no means an crypto expert, so this protocol might be vulnerable to serious security issues that I’m not aware of — please let me know if that’s the case. I know that DNS isn’t the most trustworthy system, which is the achilles heel of this protocol. But it has to be better than the status quo, communication with no encryption.

[1] try it: my email is trey @ this domain, and the authoritative keyserver for this domain is keys.tgriff3.com, meaning you can access my public key block at http://keys.tgriff3.com/users/tgriff3.com/trey . But don’t take my word for it: check my DNS records!

I recently set up a dedicated domain for Joules, the library I created for Node.js-style module loading in the browser. The idea behind the library was to create something as flexible as RequireJS - which allows for very rapid development by skipping a build step, and still allows you to compile scripts into a compressed version for production - that supports Node.js modules as well as Browserify.

Joules accomplishes both of these, resulting in a Node.js-like environment for the front-end that still lets you just refresh the page to see your changes, making development faster.

Take a look at the demo to see what I mean.

Making front-end Javascript more modular is a real problem. There have been some good attempts to make it much more modular; notably Component has taken a holistic approach to modularizing the front-end, including other assets in “components”.

Joules doesn’t attempt to do anything that ambitious - it’s a module loader, and just a module loader. It is intended to perform exactly like Node’s module loader, even where it diverges from the CommonJS spec. It’s unopinionated about how you build your project, or what libraries you use.

My hope is that by having a dependable front-end module loader that takes advantage of the existing eco-system for javascript modules (NPM, and the many thousands of packages on it) developers will be able to create clean structures whatever way is best for their project.

It’s still a very young project: I haven’t written test scripts that ensure it’s adherence to Node’s module loading style, so there are almost definitely bugs that will continue to come up. That’s the biggest hurdle to making it a stable, usable library. It also lacks the core modules that are in most every Node project. While some of these are unique to a server environment, Browserify has gone a long way toward making many of these modules available in the browser. I’d like to take advantage of that work to bring core modules to the browser using Joules.

I’ll continue to develop Joules into a more mature, stable project that can be used in production. I hope that it helps some developers who were facing similar issues that I was when looking at the options available for writing modular, reusable, front-end javascript.

John McCain is trying to force cable tv providers to unbundle their channels, either into much smaller packages or even individual channels. A lot of powerful people make a lot of money on cable bundling, and McCain has tried a similar bill in the past, so it’s unlikely that this piece of legislation will the cause unbundling of TV. However, it seems only a matter of time until unbundling happens, either due to market pressure from consumers who would rather cut the cord than pay for channels they don’t watch, or from legislation like the bill McCain proposed.

As the linked AllThingsD article mentions, such unbundling will have wide reaching effects on the economics of content providers. The Big Ten is one such content provider [1], and to understand just how much unbundling could hurt the Big Ten, we need to take a look at the most recent conference realignment.

We just finished the largest college sports conference realignment in history. The reasons for it are unsurprising: money and football money. But for conference expansion to work (i.e. be profitable for the existing member schools) the addition of an institution has to result in more revenue than the average existing member produced prior to the expansion. This can happen by bringing a larger, higher revenue-generating program into the fold, or by taking advantage of the “whole is greater than the sum of its parts” activities. In this light, the Big Ten’s addition of Nebraska in 2010 makes perfect sense - Nebraska’s football product is historically better than the average Big Ten team [2], their fan base (and as a result the number of television viewers they bring to any given Big Ten matchup) is larger than the average Big Ten school, and with the expansion from 11 to 12 schools, the Big Ten was able to sponsor a football championship game [3], which is extraordinarily lucrative for conferences [4].

But what about the Big Ten’s more recent additions of Maryland and Rutgers? On the surface, they don’t provide any of the benefits that a school like Nebraska does - relatively small fan bases, historically less than stellar on-field performance [5], and no new events to air. The real key to understanding the addition of Maryland and Rutgers lies in understanding TV bundles.

The Big Ten created the Big Ten Network (BTN) in 2006 to air Big Ten events that didn’t make it to air on traditional broadcast or cable networks. Market economics would suggest that if these events were popular enough to be on the air they already would be - but market economics hardly touches an oligopoly like cable TV. Instead, the Big Ten demonstrates to cable TV distributors that in a particular market, like Chicago/Cleveland/Columbus, a certain percentage of their clientele are Big Ten fans and would like access to BTN. The cable network then puts BTN into their basic or expanded basic package, and BTN takes a cut of every subscriber (about $0.70 - $1.00) [6]. For subscribers outside of areas where the Big Ten has a large presence, BTN takes a much smaller cut (about $0.05 - $0.10) [6], and only of subscribers to a premium sports tier.

BTN has shown itself to be a huge moneymaker for the conference [7]. But based on the math of their subscriber fees, they stand to make a lot more money if they can get television markets outside of the 8 traditional Big Ten states to include BTN in the basic or expanded basic package - a huge boost in the number of subscribers, as well as the per-subscriber fee. The only way to convince cable providers to include BTN is to expand the Big Ten’s footprint into new, populous television markets.

That’s where the Rutgers and Marlyand expansion comes in: they give the Big Ten access to the lucrative New York and DC television markets [8]. Neither Rutgers nor Maryland are the dominant college sports team in those cities (if there even is one), but they allow the Big Ten to turn a small number of premium sports subscribers, at $0.10 a pop, into a huge number of basic cable subscribers at $1.00 each. And so the Big Ten’s pie is expanded before being sliced for the new members.

Problems for the Big Ten, however, lurk just around the corner; if cable unbundling happens, either due to legislation or market pressures, and BTN is sold a la carte, they lose the vast majority of the premium sports subscribers in markets outside the 8 Big Ten states, and in the newly expanded markets like DC and New York, they only have access to subscribers who are huge Marlyand or Rutgers fans - a tiny fraction of the number of basic cable subscribers. It’s unclear whether the BTN would even be profitable under an a la carte system, let alone the cash cow it’s proven to be so far.

And so Jim Delany finds himself with an unlikely foe; if John McCain’s bill passes, the economics of the BTN, which were the underpinnings of conference expansion, stop working, and the newly added members of the Big Ten only serve to dilute the quality of the football product of the nation’s oldest conference.

Even if content providers like the Big Ten get their way and defeat this bill, which is likely, it seems to be only a matter of time until cable unbundling happens. When it does, we’ll likely be in for another round of conference shuffling as schools try to figure out how to finance costly cross-country sports programs that rely on huge payouts from conference TV networks.

[1] The Big Ten is not alone here, they were just the primary instigator of the conference expansion. Both the Pac-10/12 and SEC are trying a similar play, with the Pac 12 creating their own television networks while cutting out a large equity partner like Fox.

[2] 5 national championships, 10 undefeated seasons, 43 conference championships, 53 consensus All-Americans.

[3] NCAA rules stipulate that a conference has to have at least 12 members to have a championship game

[4] According to ESPN, Fox is paying $145m over six years for the Big Ten football championship game alone

[5] Rutgers is 5-8 in bowl games,and has never gone to a major bowl. It won a Big East Championship in 2012. Maryland is 1-4 in major bowls, and 11-11-2 in all bowls, with 2 national championships.

[6] Reported figures vary, and probably fluctuate by television market and provider, but the figures I’ve seen are $0.88 and $0.05, $0.70 and $0.10, $1.10 and $0.10

[7] It has contributed $42.5m to each member school since its inception in 2006

[8] #1 and #8, respectively. Baltimore (#27) and New Jersey are also contributors.

SAP is committing to the cloud. For those who don’t know, SAP has a huge business of on-premises ERP software (something like 80% of the Fortune 500 run SAP). This software is customized and implemented by huge tech consulting firms, the largest of which are Deloitte, Accenture, and IBM. (I used to work for Deloitte doing SAP implementations.)

The move of ERP software to the cloud was inevitable. Over the next decade we’ll see more and more software, both enterprise and consumer, move to the cloud. The only question will be who is willing and able to disrupt their own business, as SAP is attempting to do here, and who gets replaced by newer products and companies that are entirely focused on cloud solutions, like Salesforce.com and Workday.

Part of the promise of cloud-based software is the cost savings that result from elimination of customization. Enterprise customers all believe that their businesses are unique and beautiful snowflakes, and they pay out the nose to make sure that their software is too. That kind of customization is extremely lucrative for implementers and consultants, but extremely costly for the customer, both in the implementation and in ongoing maintenance.

One of the most frustrating things about my time doing SAP implementations for Deloitte was that start-from-scratch approach to every project and client. The firm had implemented for nearly every kind of company in every industry, and yet every project was treated as a complete customization. Is Kraft’s business so different from Nestle’s that they require totally different software? Is John Deere so different from Caterpillar? [1]

That’s part of the reason that I saw disruption from the cloud-based solutions coming from a mile away - there was just so much room for improvement, especially in areas that the cloud is so well-suited to handle: scalability [2], maintenance [3], and most of all reduced customization (also known as software that “just works” out of the box, er, browser).

SAP has recognized the potential for cost-savings that comes with reduced customization. Vishal Sikka is quoted in the article as saying:

At some point in the future, complex implementations should go away.

With their purchase of SuccessFactors and this most recent announcement, SAP has stated their intention to cannibalize their own on-premises software business in order to save themselves in the long term. But with this move to the cloud, and the focus on more vanilla distributions of software, what happens to the integrators for whom these complex SAP implementations are a cash cow [4]?

When I posed this question to the head of Deloitte’s SAP practice at a Q&A session for new analysts almost two years ago, he told me (paraphrased):

It doesn’t matter. The economics for Deloitte don’t change whether the software is on-premises or in the cloud.

Only time will tell.

[1] I didn’t perform implementations for these companies, and their businesses might indeed be very different from each other, but not so different that they can’t have the same or substantially similar software.

[2] Anecdotally, I experienced scaling issues with SAP at my client that resulted in downtime on the project while they put more machines on it. This wouldn’t be a big driver for most large enterprises, but it is an issue.

[3] Aside from the initial implementation, the big money for the tech consulting firms (and big expense for clients) is in upgrades to newer versions of SAP

[4] SAP alone made up around a quarter of Deloitte’s technology consulting business when I was there. Oracle was roughly another 20-25%

A couple of weeks ago a friend of mine and I were discussing the potential for sharing code between the front- and back-end when using Node.js. With the explosion in popularity both of Node and front-end Javascript frameworks, code-sharing across environments seems like a natural course of action to maximize developer happiness.

Since we both use Express frequently, we started kicking around the possibility of using it in the browser - capturing all the actions that would normally lead to an HTTP request (clicks on links, form submissions) capturing them, and processing them the same way that Express does.

Last weekend, I threw together Impress, a front-end Javascript framework that has (very) stripped down Express functionality [1]. With the help of my Node.js-style module-loading library Joules, you can build what looks like an Express application entirely on the front-end.

In building that, I learned a lot about why this was a bad idea:

• Rendering an entire response (the way that res.render or res.send does) doesn’t make sense in the browser - you rarely want to refresh the entire screen if you don’t have to.
• URL’s are only useful as a way to tie together the front-end and back-end. If you’re dealing with just the front-end, URL’s are a very indirect way to define events (e.g. clicks, form submissions) and event handlers (e.g. app.get).
• In order to really share code between the front- and back-end, you would have to maintain a different set of API endpoints for database access that both your front- and back-end talk to. CouchDB would be a good fit for this, but if you’re using something like MongoDB/Mongoose, you’re looking at defining an entirely separate set of routes to handle - not exactly minimizing work.

In discovering why Express was a bad fit for the front-end, I began to wonder if it’s really the best fit for the server. It feels right, but I suspect that’s mostly because it borrows paradigms from popular Ruby web frameworks (Sinatra and Rails).

Javascript and Node.js are heavily event-driven, so I’m beginning to think that it might be better to design the entire web server around event handling. In that case, a URL would just be one particular type of event. That mindset makes sense - when a user clicks a button it’s because they want to do something. Whether we listen for that event on the front-end (button.on("click", handleButtonClick)) or the back-end (app.get('/button/link/location', handleButtonClick)) we’re really just interpreting and acting on user intent.

If you know of any frameworks that work that way, let me know on Twitter.

1 - The biggest missing piece is the lack of support for middleware, which is probably the biggest benefit of using something like Express/Connect

More and more of our lives are online now, and more of our communication is happening online. The most ubiquitous communication mechanism is email, and, for better or worse, more people are putting more sensitive information into their emails every day.

With HTTPS becoming a near standard for accessing popular websites (and for good reason) I got to thinking about email security. STARTTLS is used by most major email providers (like Gmail), but that only serves to encrypt messages from relay to relay - it doesn’t provide true end-to-end encryption, from sender to recipient.

With Google Apps taking over the nearly the entire University email client ecosystem, I think it’s safe to say that in the near future, if not already, almost all email will be conducted through a cloud provider like Gmail. When it comes to sending unencrypted messages, you can choose to trust your cloud provider (and all the relays between them and your intended recipient), but I think that’s a mistake because:

1. Providers have a profit motive that isn’t always in your favor (e.g. contextual ads based on the contents of your messages)
2. Cloud providers have a less than stellar security record
3. Good security exists in layers
4. Trust is not security

Those wearing tinfoil hats right now won’t see the problem because they already use PGP to encrypt all their emails. However PGP, with its near military-grade encryption available to the public, is not a mainstream solution in its current state. We need regular people - the same people who don’t know what HTTPS is beyond a green indicator in the address bar - to be able to communicate securely.

SafeGmail is an interesting take on this problem, and requires fairly seamless, strong security, but it requires sharing of a password of passphrase ahead of time, not exactly something that can happen every day.

I think the reason our email remains unencrypted is that we don’t have a scalable solution for person-to-person encryption. I think PGP is the answer, but we need to find a way to give everyone, including my grandmother, a PGP key that can encrypt and decrypt her communications effortlessly.

Identity on the web is a big problem. We have logins for every website we visit, and only the most savvy among us use a password manager to make sure that the inevitable database leak doesn’t burn us everywhere.

There have been attempts to solve this problem - OpenID for one, Facebook Connect (as well as the similar services from Twitter and LinkedIn), and Mozilla’s new Persona, which is the first real alternative to a traditional login system.

But I saw today a really interesting paradigm - the login-less app. Emmanuel Bégué created Urgeous, a Posterous clone, which is primarily focused on post-by-email, one of Posterous’s most interesting features.

Since he was implementing post-by-email, and email is the default definition of identity, he got rid of a login system altogether. Instead, everything sent from your email address is a post on your behalf. To edit posts you use a link with a unique code that is generated and sent to you when a post is published.

This is a very specific scenario, and one in which you don’t need to rely on a persistent state, but it brings up an interesting point - do we really need all these login systems? Does Quora really need me to login to read beyond the first answer?

It’s almost reflex to include a login system when building a new web app, but we as developers would be wise to really consider the needs of our application before adding yet another layer of broken login systems onto our users.

I love Javascript. I code almost exclusively using Javascript. My startup’s entire stack is Javascript (MongoDB + Node + jQuery + Hand-rolled front-end libs).

But I hate apps that require Javascript to load content. I’m looking at you Backbone.js/AngularJS/Ember.js.

I understand that the increasing speeds of SpiderMonkey and V8 (and even Chakra) have turned the web into a place where you can build client-side web apps with much richer functionality. And I understand why that has led us to more and more “single-page” apps that load additional data and render it using on-page templates.

What I don’t understand is a reliance on Javascript to load content. Not rich functionality, not drag-n-drop WYSIWYG editing, not even AJAX-y autocompletions, but plain old content. And not just as the primary choice to provide snappy responses once the page has been loaded once, but the only way to load content.

Earlier today, I was checking out the popular article on Hacker News about Ember.js’s Quick Start shortcomings. It was a forum post on the new Discourse platform from the Stack Exchange folks, which was itself built on Ember.js.

Here’s what I saw:

A blank page.

I checked the errors, and it looks like something failed to load.

When I checked the source, I see that Discourse has some nice backwards-compatibility for people that don’t have Javascript enabled — they include the entire rendered HTML page in the <noscript> tags. But for those of us that have javascript enabled, but for whatever reason it doesn’t load properly? No forum for you.

The irony to me, is when looking at Discourse’s about page:

The state of forums has been unchanged for so long that forums are considered unworkable and undesirable; few sites want forums any more because the software is so poor.

In my book, those old forums worked, as ugly as they were, and working and ugly is better than pretty and not working. All your fancy live-updates, inline editing, etc, simply don’t matter when the content doesn’t load. As has been said a million times before, Content Is King.

I don’t mean to pick on Discourse here. They built what is I’m sure a great product, which is itself built on a great framework by very smart people.

The problem is in the assumption that rich functionality and traditional HTML rendering have to be divorced. That if we want all the fancy rich-client stuff, we can’t deliver rendered content to the client, we must render content client-side.

I don’t think that’s true. I think you can build apps that deliver rendered HTML content to the browser, and add rich functionality on top of that, hopefully with appropriate fallbacks for the most important actions.

Doing so provides a number of benefits:

• Faster startup time - “Time to first tweet” is Twitter’s benchmark, and the best way to ace such a benchmark is by delivering rendered HTML to the browser on first page load
• Graceful failure - No blank screens like that I one I got today at discuss.emberjs.com
• Search engine friendliness - The big crawlers are supposedly going to get better at rendering Javascript to index pages, but why would you make it harder to index your site?
• Smaller payloads - With Discourse’s noscript tags, they are delivering both the data and the templates to the user’s browser twice

When building a true app, one in which the functionality instead of the content is most important, the exclusive reliance on client-side Javascript makes sense. But any web app for which content plays an important role, loading that content with Javascript is the wrong choice.

At Endorse.me, we use Amazon S3 as a CDN for all our static assets - images, scripts, and css. To make sure that we’re only uploading new assets to Amazon, and to make sure that we’re always using the latest versions on the production site, we fingerprint all of our static assets.

The result is that, using Node.js’s asynchronous filesystem functions, we’re opening nearly all of our static files at once to create hashes for the fingerprints. I recently ran up against OSX’s default maxfiles setting of 256, which resulted in the descriptive Error: EMFILE, too many open files error, crashing the Node process.

I didn’t find a good solution, either from the trusty StackOverflow, the Node.js docs, Google, or the #nodejs channel on IRC. It seems this isn’t a common enough use case to warrant a best practice.

Based on this StackOverflow answer, I created my own replacement for some of the most common fs methods that avoids the EMFILE error. It does this by keeping track of the total number of files opened at once by the module, and queuing up those that will take us over a predefined limit (200 by default).

A solution like this (as opposed to say, large batches) allows us to keep the maximum concurrency that the filesystem allows without changing the underlying pattern of opening, reading, and writing files using Node’s built-in fs module.

Using my module, reading files from a directory looks the same, just without the crash:

var Filequeue = require('filequeue');
var fq = new Filequeue(200); // max number of files to open at once

fq.readdir('/path/to/files/', function(err, files) {
    if(err) {
        throw err;
    }
    files.forEach(function(file) {
        fq.readFile('/path/to/files/' + file, function(err, data) {
            // do something besides crash
        }
    });
});

The module, Filequeue, is available on NPM and Github, and is MIT Licensed. Let me know if you have any questions or suggestions for improvement.

Today I encountered a interesting piece of the Mongoose internals added in v3. In a nutshell, Mongoose uses versioning to prevent you from accidentally updating the wrong element when editing a member of an Array.

As Aaron Heckmann explains:

To see how this could be problematic we need to take a closer look at the underlying operation used to update the comment. When post.save() is executed, an update is issued to MongoDB that looks like the following:

posts.update({ _id: postId } , { $set: { 'comments.3.body': updatedText }})

Notice comments.3.body, this is called positional notation. This tells MongoDB to set the body of the comment in the comments array at index position 3 to the updated text.

If, for instance, while you were editing comments.3 in the example above, a new comment was added, then comments.3 would not be the correct comment, and your update would likely be successful, but not in the way you intended.

Mongoose’s solution to this problem is to add a version number the document that is incremented every time the array changes. If you try to update an array element using positional notation with the wrong version key, you’ll get a nice VersionError. The text of the error is “No Matching Document Found”, which isn’t very helpful, but the ability to test for instanceof mongooose.Error.VersionError is quite useful.

In my case, I just test for the VersionError, and re-fetch the document and re-update the array if I encounter one. I put a limit on the number of times this loop can continue so as to not accidentally melt my database.

I can see how, at a huge scale, something like this might become an issue if you’re adding and and updating a lot of subdocuments concurrently, but for my case, it works like a dream.

I signed up for Mailbox on the first day of its release and found myself sitting in line near position 280,000. As the number ahead of me has slowly wound down, and the number behind me has slowly grown, I’ve had a quite a bit of time to think about Mailbox, and what it really means.

There are more than three-quarters of a million people waiting for access to Mailbox. And since only a few people have actually used the app and can recommend it on the basis of its functionality, it seems that all those people are waiting in line for an idea, a cure for their email problem.

I haven’t used the app myself, but from what I’ve heard it is a tool to formalize the usage of Inbox Zero. Inbox Zero is a management philosophy for incoming email. As Merlin Mann, creator of Inbox Zero, explains:

Clearly, the problem of email overload is taking a toll on all our time, productivity, and sanity, mainly because most of us lack a cohesive system for processing our messages and converting them into appropriate actions as quickly as possible.

Inbox Zero proposes itself to be just that, a “cohesive system for processing our messages”, and Mailbox is the tool by which that system is accomplished. If Inbox Zero describes how to put together a house, Mailbox is the hammer and nails.

But not everyone agrees that the problem with email is that we lack a cohesive system. Paul Graham, in his essay on Frighteningly Ambitious Startup Ideas, says that the way to replace email, is to replace the inbox with a better todo list.

Email was not designed to be used the way we use it now. Email is not a messaging protocol. It’s a todo list. Or rather, my inbox is a todo list, and email is the way things get onto it.

One of his suggestions for email’s replacement is that it should give more power to the recipient.

I want there to be more restrictions on what someone can put on my todo list. And when someone can put something on my todo list, I want them to tell me more about what they want from me. Do they want me to do something beyond just reading some text? How important is it?

We already have some of the infrastructure in place in email itself to support what Graham is asking for. Specifically the “Importance” and “Reply-by” headers exist as defined in RFC 1327. As far as I can tell, those flags are largely unused. As Cory Doctorow notes (albeit on a different topic) in Epoch:

if you let a coder (or, ::shudder::, a user) specify the importance of her alert, give her a little pull-down menu that has choices ranging from “nice to know” to “white-hot urgent,” and nine times out of ten, she’ll choose “NOW NOW NOW URGENT ZOMGWEREALLGONNADIE!”

So, Boris from The Next Web took Graham’s suggestion one step further, and he requested additional metadata from his senders via his own personal InboxPro. This data allows him to easily address certain types of emails, like Yes/No questions, without asking questions about the priority of the message.

But do all these management systems and apps really solve the problem?

Put another way, what exactly is the problem?

• Merlin Mann believes it is the lack of a cohesive system to manage incoming messages.
• Paul Graham believes we’re using a messaging protocol where a todo list protocol would be better.
• Boris believes we don’t have enough information about incoming messages before we manually parse them.

But could the problem be the sheer volume of email itself? The Slow Web Movement, a reaction against the increasingly real-time nature of the Web, seems to indicate that there are people who value not being beholden to the real-time paradigm. More and more, it seems that email is turning into a constant stream of incoming messages, and while it’s not real-time, it shares many of the same problems, and is antithetical to the Slow Web’s philosophy: “users should have a life.”

Every new email management app, and every new email management system seems to be treating a symptom of the disease rather than the disease itself. I’m not sure if the disease is the increasing amount of email, but I can be fairly certain that the disease is not the lack of a management system, nor a todo list communicated through messages.

Before we try to “fix email”, let’s figure out what’s wrong with it. Let’s diagnose the disease.

I recently ran into a problem while developing some of our advanced search capabilities on Endorse.me: I wanted to find all the documents in our MongoDB database that had at least one of a number of elements as a property. This is fairly easy to do using MongoDB’s ‘$in’ operator when the property is a primitive, like a string:

https://gist.github.com/4599048

But what if one of our toys has multiple colors, listed in an array? I searched around for a some sort of $any operator before looking again at the MongoDB documentation for $in. Turns out, if the field has an array, it behaves exactly how I was hoping it would!

https://gist.github.com/4599072

When pushing a bug fix to the Endorse.me servers this morning, I noticed that the site was responding extraordinarily slowly. A quick check in the logs showed that we were being flooded with requests from a single IP address for a single resource (a Flash/SWF file that we use to allow users to copy text to the clipboard) that was hogging almost an entire web process on Heroku.

I quickly spun up another dyno on Heroku (cloud computing FTW) and looked a little bit deeper into the issue. I emailed the user who’s account seemed to be the source, but received no response. Some light googling didn’t reveal any easy ways to block specific IP addresses using Heroku/Node.js (I found some ways using Rack), and I started considering just how bad it would be to continue to throw new dynos at the problem.

I moved the SWF file to our CDN, where it should have been all along. For convenience in development, I had kept it local, thinking “what’s the worst that could happen with ONE static asset?” Sigh.

Eventually I signed us up for CloudFlare which routes all of our traffic through their network, allowing me to block specific IP’s. Their sign-up process was completely painless aside from the DNS propagation which is unavoidable.

As soon as we were set up, I blocked the IP, and everything went back to normal. I was able to scale our Heroku setup back down again, and actually read my logs.

Now, I’m inclined to think that this whole thing was an accident for a few reasons:

1. The User Agent string indicated it was from Chrome
2. The requests were for a SWF file (which seems like it could be a bug, it wasn’t even a big file)
3. While there were a lot of requests, it wasn’t even close to actually crippling us or taking us down. 

If anyone else has run into this issue before and knows a better long term fix without blocking an IP, please let me know.