Heroku

Accidental Denial-of-Service


Node.Js · Dos · Cloudflare · Heroku

When pushing a bug fix to the Endorse.me servers this morning, I noticed that the site was responding extraordinarily slowly. A quick check in the logs showed that we were being flooded with requests from a single IP address for a single resource (a Flash/SWF file that we use to allow users to copy text to the clipboard) that was hogging almost an entire web process on Heroku.

I quickly spun up another dyno on Heroku (cloud computing FTW) and looked a little bit deeper into the issue. I emailed the user who’s account seemed to be the source, but received no response. Some light googling didn’t reveal any easy ways to block specific IP addresses using Heroku/Node.js (I found some ways using Rack), and I started considering just how bad it would be to continue to throw new dynos at the problem.

I moved the SWF file to our CDN, where it should have been all along. For convenience in development, I had kept it local, thinking “what’s the worst that could happen with ONE static asset?” Sigh.

Eventually I signed us up for CloudFlare which routes all of our traffic through their network, allowing me to block specific IP’s. Their sign-up process was completely painless aside from the DNS propagation which is unavoidable.

As soon as we were set up, I blocked the IP, and everything went back to normal. I was able to scale our Heroku setup back down again, and actually read my logs.

Now, I’m inclined to think that this whole thing was an accident for a few reasons:

  1. The User Agent string indicated it was from Chrome

  2. The requests were for a SWF file (which seems like it could be a bug, it wasn’t even a big file)

  3. While there were a lot of requests, it wasn’t even close to actually crippling us or taking us down. 

If anyone else has run into this issue before and knows a better long term fix without blocking an IP, please let me know.

Learning MVC, Ruby, Rails, Heroku, Git, and PostgreSQL at the same time


Heroku · Git · Postgresql · Mvc · Rails

So I finally took the plunge. I started developing with Ruby on Rails.

Little did I know that learning Ruby on Rails consists of learning a host of other things that I should probably have learned by now, but for various reasons, haven’t.

MVC is complicated. Not complicated like difficult to grasp, complicated like “wait, where do I put this function?”. I read somewhere it should go in your controller. Then I read the skinny controllers, fat models post by JamisBuck. Oops. Time to “refactor” I guess. (is it refactoring if you’re only playing with the default scaffold?)

Ruby is amazing. Thank goodness DHH chose this when developing Basecamp. A language designed to behave as you expect it to? How could it get any better?

ActiveRecord is amazing. If I never see another SQL query in my life, I’ll be happy.

Rails is incredible. DHH is the man. I’ve never used a framework before, so maybe I’m giving him too much credit, but the amount of productivity you can get when you have Rails in place is just mind-blowing.

I’ve never installed a server locally before, and I’ve never done my development on a local machine. I know that sounds REALLY crazy, but I have always done my hacking in PHP on a shared box somewhere that I FTP'ed to. And with the Notepad++ FTP extension, it was super easy. But developing on your local machine is…easier. Except for the setup part – which was made worlds easier by EngineYard’s Rails Installer for Windows. (Yes, I’m developing in Windows. Yes Rails is much slower on Windows.)

I’m using Git wrong. I know that I am. And I’m so used to developing without version control that I don’t really git it yet. (That’s not a funny pun, but I couldn’t help myself). But it is nice to put little messages next to your saves I guess, even if they are just a string of obscenities.

Heroku is cool. But not as cool to me as to long-time Rails developers because it basically does what you’ve been able to do with PHP forever: upload it and have it work the right way. I know that for Rails with all the dependencies, etc, this is a big deal, but from a pure user experience perspective, it’s pretty much the same to me.

Migrating to PostgreSQL is a big pain. I did all my developing in SQLite3 (against the wishes of Heroku) and when I deployed to Heroku I had to fix one query in 3 different places. And of course I had to re “git push” every time. And I haven’t figured out how to make Heroku not re-install all my gems every time I do that.

All in all, even with the huge learning curve of what feels like 10 new systems and a totally different model of development and deployment, I’ve accomplished in a few days what would have taken me probably a week in PHP (which I’m very comfortable in). So yes, it was a good decision. It hurts right now, but I can already feel the productivity gains.

Also, Sublime Text 2 is really nice.