Cellar is Open Source
Endorse.me relies heavily on Open Source software, especially the many npm modules maintained by companies like LearnBoost (Mongoose and Express).
In that spirit, we’ve started open-sourcing certain modules of our Endorse.me codebase. The first release is Cellar, a simple short-term single-document storage engine for MongoDB. Built on top of Mongoose, it is intended for caching of simple schemas.
At Endorse.me, we use it in most of our external API’s to cache OAuth tokens or query results. There are more packages coming soon (including a simpler Key-Value implementation of Cellar), and pull requests are welcome!
Child Processes in Node 0.8 ‘exit’ prematurely
Due to a recent change in the API, a lot of existing code based around spawning child processes will break. The change is explained on the Node Github Wiki:
'exit'event is now emitted right after the child process exits. It no longer waits for all stdio pipes to be closed.
'close'event was added that has is emitted after the child has exitedandall the stdio pipes are closed.
Most spawned processes wait for the ‘exit’ event, and should now listen for the ‘close’ event. This is a gotcha that burned me with the node.gzip library.
Airbrake and Forever to keep you happy
When developing with Node.js, you’ll notice that, unlike in PHP, if you encounter a really nasty unhandled exception or error, the entire application crashes, instead of just that particular request. While that can be helpful during development, it isn’t ideal to show the Heroku error page to your users in production.
Airbrake (formerly HopToad) is available as a Heroku add-on, and along with the node-airbrake module (thanks felixge!), sending yourself emails with a stack trace when the server crashes is stupid easy.
While that makes your response time to crashes much better, it doesn’t do anything to keep your server available until you manually restart the Heroku process. That’s where Forever comes in, an excellent module from Nodejitsu that will automatically restart your server when it goes down.
Obviously restarting your borked server over and over isn’t a permanent solution, but it allows your server to stay available for the rest of your users even if one of them finds an obscure bug, and gives you a bit more time to find the source of the issue.
Just add a file (mine’s called forever.js) that starts your server with forever, and notifies Airbrake when the server is actually down.
That should let you sleep a little bit easier!
On the Bleeding Edge of Node.js
Developing with new tools is a lot of fun. They’re usually new because they allow you to do something that you couldn’t before, or in an environment that you couldn’t before.
However, Node is very young, and very immature. There are tons of modules out there, and more coming every day, but the fact is that you will encounter situations that no one else has encountered before (or at least documented). Coming from Rails and PHP, that’s quite the change of pace.
For starters, how do you store passwords securely in a database? Well, you use bcrypt of course! And there is, in fact, a bcrypt module for Node that is excellent…if you’re not using Windows. One of our developers was using a Windows box, and despite trying to figure out how to get it to build for about a day and half, it just wasn’t meant to be. At some point, it’s faster to boot up a Linux VM than to embody the definition of insanity.
Last I checked, the issue was still being actively worked, with an update from the bcrypt author as of about two weeks ago. The community is there, the ecosystem is just young.
But it’s not just Windows. We work with MongoDB for storage, and on Node, you can’t do better than Mongoose. But as I said, it’s young. So the findAndModify behavior wasn’t available in Mongoose. Taking a closer look at the source on GitHub, I found that the edge version of Mongoose did support findAndModify (in the form of findOneAndUpdate for all you Mongoose users). So I did what any overconfident dev might do: I switched to edge in order to take advantage of the new feature.
Unfortunately, there is a reason that it is not a complete release. While findOneAndUpdate works, there was at least one very nasty bug that I encountered in the Mongoose source. That, and the changes to functionality make certain design patterns much more attractive (namely, passing a Mongoose document as the update argument) that aren’t currently supported in Mongoose.
And these are just the issues I’ve found. And I’ve barely scratched the surface of what Node can do.
But that’s a good thing. That’s what makes it fun. Everything has pro’s and con’s, and to me, the pro’s of what Node can do, and what I see for the future of Node are well worth some time spent here and there digging into the source code of a newer module.
After all, some people find debugging fun.
 I’m still working adding tests to my pull requests to get both of these complaints addressed upstream.
My first real product launch: Sweat, tears, and viral growth
3 weeks ago yesterday, I did my first real product launch. I’ve shipped apps before, but I’ve never done developed with a team, and I’ve never tried so hard to get support drummed up beforehand.
This time I did. I spent the 6ish weeks prior to launch developing a social web application in NodeJS with 2 other developers, a designer, and a business guy. We set a launch date ahead of time, and had a promotional team on the ground in our target market pushing people to our early signup page. We got great press, driving even more traffic. We had more people sign up early than I have ever had use one of my applications before, and this was the first time I’ve ever developed in NodeJS. It was exciting.
We planned a soft-launch the day before we were to send out emails to all of the early signups, to be able to start testing the product at some scale before it got out of control. Sunday at 4pm was our soft-launch, and Monday morning was the “real” launch.
One of the developers, the stronger NodeJS and mongoDB guy, had his sister’s wedding the weekend of launch, leaving me and our HTML dev to handle things during launch. Needless to say, we left way too much until the last minute, and the night before our soft-launch I was up until 5am trying to make all of our last-minute changes. The day of the soft-launch was dedicated to moving everything over to the production database and fixing the bugs that popped up during all those last minute changes I had made.
Of course, we ran into difficulties moving the database over, which consumed the better part of our morning prep (who knew that forgetting to drop old indices would cause the entire site to crash, but only on the second access?). Heading into the afternoon, everything on the site seemed to be crashing constantly, and I was living only on several cups of coffee.
Then at 3:30, I had fixed it. Nothing was breaking. Just in time for our 4pm launch. Our business guy was introducing the product to the 4pm soft-launch crowd of around 40 when I saw a horrible message scroll over the logs H10 (App Crashed). I gave him the kill sign, which he apparently didn’t understand, and he tried to log in to the app on the big screen. Everyone in the room started to chuckle when they saw Heroku’s “Application Crashed” screen.
I raced to fix the issue, which I don’t even remember now, and we finally got things underway. The next 3 hours I hardly remember, mostly because I was staring at the logs fly by, waiting for the app to crash so that the other dev could restart the server while I tried to quickly trace and patch the issue. I was so heads-down that a friend of mine didn’t even see me at the launch at all (see below).
The people at our soft-launch were very kind for their patience with our shaky product, and told us immediately of any issues they saw. That also meant that the moment an “App Crashed” message scrolled over the logs, the entire room filled with a chorus of users saying “crash!”. Most of it was property checking null or undefined variables, stuff that should have been caught much earlier.
Some of them were a little more complex, revolving around the data looking differently than we expected. For those of you who don’t know, NodeJS doesn’t play nice when you send two responses to the client. Turns out, looping through an array or object and triggering a res.render or res.send when certain conditions are met is a great way to crash the server if the data isn’t exactly what you expect.
The viral growth was much stronger than we anticipated, and our userbase grew from 40 to 500 in the next 6 hours. Our business guy was ecstatic. My heart pounded seeing the logs scroll by faster and faster, but that made it even harder to watch for crashes and other obscure errors as they happened. I spent the 8 hours after launch fixing things, some of them much more obscure than others (Logging in didn’t work when you didn’t have a gender listed on Facebook, for one).
Launching was one of the most exciting things I’ve done, and also one of the most stressful. I didn’t eat until 9pm that night (Chipotle FTW), but hardly felt hungry all day. I must have sweated out a gallon of water, and I probably took a year off of my life. But it was awesome. To see people use something you’ve worked hard on feels great.
I only got a few hours of sleep that weekend, and spent the entire time working, and it was still one of the most rewarding and refreshing weekends I’ve had.
Lessons learned (most of I already knew but that didn’t stop me from making the mistakes anyway) :
- Don’t make changes the day before you launch.
- Test, test, test.
- The data will not be what you expect it to be.
- Calls to the database and external API’s WILL fail. Prepare for it.
- Install airbrake BEFORE launching
- Set your expectations low (in user numbers), prepare yourself for the worst (out of control growth).
How can client-side databases be secure?
Seeing Meteor and Firebase all over HN has been exciting - I’m currently building a Node.js application but not using as much real-time updating as I could, largely because of the effort involved all over the application.
One of the benefits of developing with Node is that you can use the same code on the front and back-end, which is especially helpful for things like data validation, which some sites are inconsistent about. Meteor and Firebase are the answer to the implied question: if the code is the same in both places, do we really need it in both?
In the aftermath of building my own Firebase app in 5 minutes (!), I saw the concerns sprout up. No, I’m not talking licensing, I’m talking security. Timo Zimmerman’s post outlines how the lack of security and maturity in Meteor makes it just a toy:
There is currently no authorization or authentification system. It is planed[sic] but not implemented. So you basically expose your whole database to every client without any way to add some security.
I had seen the point about the lack of authorization in another comment on HN about Firebase (can’t find it now, oops), both of which got me thinking: how could you do authorization for a database that’s fully exposed on the client side?
When the database is completely accessible from the client, and the client can freely modify the code used to access it, isn’t it impossible to have real security?
You could limit what they have access to through some sort of ACL, but that would introduce a lot of backend complexity to your app, and involve Firebase/Meteor dictating part of your database schema. And in fact, you would have to have even more granular limits than just “Can Only Modify Record 1”, as certain parts of records have to be protected. So much for “all in the client” or “No servers.”
It seems like a great idea, it really does. And maybe I’m missing something obvious here (I hope), but it just seems downright impossible to secure a system like this. Isn’t one of the guiding principles of security “Don’t trust the client”?
I guess I’ll wait until they reveal what they’re planning.
Dropbox + Git = Develop Anywhere
For a new nodeJS project I’m working on in collaboration with a few other developers, we’re using Git in combination with Github, and I’m finally learning how to use Git the right way, with branches, several remote repositories, and all the other good stuff that makes everyone love Git.
But this weekend I ran into a serious problem: my computer died, and I found that I left my charger at work. We have a major milestone we need to hit on Monday, and I was up a creek.
Thankfully, a friend of mine said I could use her Mac to work on it. I had saved my development work in a Dropbox folder, so I went to Dropbox and shared it with her. Then I downloaded Git, nodeJS, and Sublime Text 2 to get my dev environment ready.
I generated a couple of new SSH keys for Github, and checked
git config --list
Lo and behold, everything was there. My remote repo’s, my commit history, everything. Part of the genius of Git is that everything is stored in files in your working directory.
I can’t stress this enough: put your development directory in your Dropbox. You never know when you’ll need it, and with Git, you’ll be set to go in a matter of minutes even in the worst case scenario. I’m considering adding the installers I use to develop to the directory just to make the process even more painless.
Let me know if you have any other tips about using Git, I’m still learning all the good stuff it can do.