3 weeks ago yesterday, I did my first real product launch. I’ve shipped apps before, but I’ve never done developed with a team, and I’ve never tried so hard to get support drummed up beforehand.
This time I did. I spent the 6ish weeks prior to launch developing a social web application in NodeJS with 2 other developers, a designer, and a business guy. We set a launch date ahead of time, and had a promotional team on the ground in our target market pushing people to our early signup page. We got great press, driving even more traffic. We had more people sign up early than I have ever had use one of my applications before, and this was the first time I’ve ever developed in NodeJS. It was exciting.
We planned a soft-launch the day before we were to send out emails to all of the early signups, to be able to start testing the product at some scale before it got out of control. Sunday at 4pm was our soft-launch, and Monday morning was the “real” launch.
One of the developers, the stronger NodeJS and mongoDB guy, had his sister’s wedding the weekend of launch, leaving me and our HTML dev to handle things during launch. Needless to say, we left way too much until the last minute, and the night before our soft-launch I was up until 5am trying to make all of our last-minute changes. The day of the soft-launch was dedicated to moving everything over to the production database and fixing the bugs that popped up during all those last minute changes I had made.
Of course, we ran into difficulties moving the database over, which consumed the better part of our morning prep (who knew that forgetting to drop old indices would cause the entire site to crash, but only on the second access?). Heading into the afternoon, everything on the site seemed to be crashing constantly, and I was living only on several cups of coffee.
Then at 3:30, I had fixed it. Nothing was breaking. Just in time for our 4pm launch. Our business guy was introducing the product to the 4pm soft-launch crowd of around 40 when I saw a horrible message scroll over the logs H10 (App Crashed). I gave him the kill sign, which he apparently didn’t understand, and he tried to log in to the app on the big screen. Everyone in the room started to chuckle when they saw Heroku’s “Application Crashed” screen.
I raced to fix the issue, which I don’t even remember now, and we finally got things underway. The next 3 hours I hardly remember, mostly because I was staring at the logs fly by, waiting for the app to crash so that the other dev could restart the server while I tried to quickly trace and patch the issue. I was so heads-down that a friend of mine didn’t even see me at the launch at all (see below).
The people at our soft-launch were very kind for their patience with our shaky product, and told us immediately of any issues they saw. That also meant that the moment an “App Crashed” message scrolled over the logs, the entire room filled with a chorus of users saying “crash!”. Most of it was property checking null or undefined variables, stuff that should have been caught much earlier.
Some of them were a little more complex, revolving around the data looking differently than we expected. For those of you who don’t know, NodeJS doesn’t play nice when you send two responses to the client. Turns out, looping through an array or object and triggering a res.render or res.send when certain conditions are met is a great way to crash the server if the data isn’t exactly what you expect.
The viral growth was much stronger than we anticipated, and our userbase grew from 40 to 500 in the next 6 hours[1]. Our business guy was ecstatic. My heart pounded seeing the logs scroll by faster and faster, but that made it even harder to watch for crashes and other obscure errors as they happened. I spent the 8 hours after launch fixing things, some of them much more obscure than others (Logging in didn’t work when you didn’t have a gender listed on Facebook, for one).
Launching was one of the most exciting things I’ve done, and also one of the most stressful. I didn’t eat until 9pm that night (Chipotle FTW), but hardly felt hungry all day. I must have sweated out a gallon of water, and I probably took a year off of my life. But it was awesome. To see people use something you’ve worked hard on feels great.
I only got a few hours of sleep that weekend, and spent the entire time working, and it was still one of the most rewarding and refreshing weekends I’ve had.
Lessons learned (most of I already knew but that didn’t stop me from making the mistakes anyway) :
- Don’t make changes the day before you launch.
- Test, test, test.
- The data will not be what you expect it to be.
- Calls to the database and external API’s WILL fail. Prepare for it.
- Install airbrake BEFORE launching
- Set your expectations low (in user numbers), prepare yourself for the worst (out of control growth).
