Node.Js

Filequeue supports Streams


Node.Js · Javascript · Filequeue

Filequeue, the module I wrote to fix an issue I was having while trying to open too many files at once in Node.js, now supports Node 0.10.x, and has support for the new ReadStream and WriteStream methods that were included with the 0.10.x release.

Adding streams support required that I rewrite most of the module to be more flexible in terms of method signatures allowed (previously, a callback as the final argument was required for the module to function properly).

While doing the rewrite, I also changed the way new methods were added to pass the Grep Test. The actual logic of what’s happening when the methods are called is much clearer now, which should make maintenance and extension much easier, both for myself and other contributors.

Filequeue is available on NPM.

MongoDB: find a document that matches any element in an array


Node.Js · Mongoose · Mongodb

I recently ran into a problem while developing some of our advanced search capabilities on Endorse.me: I wanted to find all the documents in our MongoDB database that had at least one of a number of elements as a property. This is fairly easy to do using MongoDB’s ’$in’ operator when the property is a primitive, like a string:

https://gist.github.com/4599048

But what if one of our toys has multiple colors, listed in an array? I searched around for a some sort of $any operator before looking again at the MongoDB documentation for $in. Turns out, if the field has an array, it behaves exactly how I was hoping it would!

https://gist.github.com/4599072

Accidental Denial-of-Service


Node.Js · Dos · Cloudflare · Heroku

When pushing a bug fix to the Endorse.me servers this morning, I noticed that the site was responding extraordinarily slowly. A quick check in the logs showed that we were being flooded with requests from a single IP address for a single resource (a Flash/SWF file that we use to allow users to copy text to the clipboard) that was hogging almost an entire web process on Heroku.

I quickly spun up another dyno on Heroku (cloud computing FTW) and looked a little bit deeper into the issue. I emailed the user who’s account seemed to be the source, but received no response. Some light googling didn’t reveal any easy ways to block specific IP addresses using Heroku/Node.js (I found some ways using Rack), and I started considering just how bad it would be to continue to throw new dynos at the problem.

I moved the SWF file to our CDN, where it should have been all along. For convenience in development, I had kept it local, thinking “what’s the worst that could happen with ONE static asset?” Sigh.

Eventually I signed us up for CloudFlare which routes all of our traffic through their network, allowing me to block specific IP’s. Their sign-up process was completely painless aside from the DNS propagation which is unavoidable.

As soon as we were set up, I blocked the IP, and everything went back to normal. I was able to scale our Heroku setup back down again, and actually read my logs.

Now, I’m inclined to think that this whole thing was an accident for a few reasons:

  1. The User Agent string indicated it was from Chrome

  2. The requests were for a SWF file (which seems like it could be a bug, it wasn’t even a big file)

  3. While there were a lot of requests, it wasn’t even close to actually crippling us or taking us down. 

If anyone else has run into this issue before and knows a better long term fix without blocking an IP, please let me know.

PayPal could explore Node.js (and TechCrunch needs to change its name)


Node.Js · Javascript · Yahoo · Crockford · Techcrunch · Paypal

In Ingrid Lunden’s post about Douglas Crockford departing Yahoo for PayPal, she made several serious errors, most of which have been corrected by now. In an earlier version of the article, she called Crockford a Java icon, and after being called out in the comments, she changed it to (correctly) read JavaScript.

As one commenter pointed out, “Java is to JavaScript as Ham is to Hamster.”

The blunder remains obvious despite the correction in other parts of the article, such as where Ms. Lunden mentions “Java (not JavaScript) is used in featurephones, Android and web interfaces,” and “PayPal is ‘moving rapidly’ from C++ to Java as a backend technology.” Both of which are totally irrelevant to Crockford’s hiring.

Silly mistakes (and the fact that half of the article is regurgitating Crockford’s Wikipedia page) aside, what irks me the most is the total lack of both technical understanding and journalism. Lunden has a gem in the article: Edwin Aoki specifically mentions to Lunden that “[Crockford] will be partnering with [Aoki] on more ways to utilize JS both in the server and on the browser for PayPal.” (emphasis mine).

PayPal could very well be exploring/expanding their use of Node.js, a significant move in the technology community, and Lunden totally missed it!

Regardless of the poor reporting, this is an exciting time for Node developers if another big player like PayPal is putting some resources into the area.

And TechCrunch, if you’re not even going to try to report on the technology side of things,  please change your name to something more suitable, like SiliconValleyGossipCrunch.