This was a post I wrote in the middle of 2013 but never published. I wanted to share this since it’s a common story across all technologies and developers of all skill levels. Sometimes things really just don’t work. As a post-script, I did come back to this project and had a lot of success. When in doubt, let time figure it out :)

For the last couple of weeks, I’ve been trying my hand at the node.js ecosystem. I had an app idea but I wanted to make sure I chose tech stacks wisely. There’s no better way to get familiar with different stacks than to get your hands dirty and try them all, so that’s what I did.

Sometimes when you start with a new language or platform things come easy, you can blaze a burning trail writing great software. You’re like an extension of the computer, everything just works and works great. Sometimes, though, like this time for me, you putter and stall and hit roadblocks at every corner.

At times it feels like a waste of time. And at times I’m frustrated. I keep saying to myself “I’m better than this! Why am I stuck??”. But, even though I haven’t made any real progress in my app idea, I have learned tons about node.js and its accompanying workflow. Frequently, failing is where you really learn the most. It’s easy to forget that, even when you are ground to a halt, you are still learning. As long as you are learning then it’s not time wasted.

Just to prove the point, let me recant some of my recent failings.

Sequelize

I started my exploration with just regular node.js. I set up some routes and everything was cool. Then it was time to add in a backing store. At first I wanted to try using a MySql ORM (because in the past I’ve always done SQL by hand and I wanted to do something different). I tried out sequelize but found that not only did it not support transactions (and apparently transaction support in node.js is a pain since you need to write a connection pool to manage concurrent MySql connections), but it also never set up any actual foreign keys in the schema. This means you can easily corrupt your data even if the database would have prevented you from doing that. While, I did have a copy working, I didn’t particularly like the workflow so I started over. To be fair, the sequelize developer was extremely helpful and responsive on twitter, and maybe I’ll try this library again in the future (when transaction support is added).

Mongoose

After sequelize, I switched over to using mongoose with mongoDb. My only history with using document based stores is with Lucene, but in that situation I was storing actual documents and using full text searching for it. I spent a couple days reading up on mongo and mongoose and I had a quick document example up. I was able to insert and query users and related data pretty easily. Then I started to think about how to properly structure my schema for the app I wanted to write in such a way that working with the data was a pleasure, and at the same time maximized performance and throughput. This stumped me. Researching NoSQL schema design patterns led me down to understanding about linked vs embedded documents, map/reduce with mongo, populating embedded documents with mongoose, different query types and syntax, etc. Embedding too much meant that I had to search through my document to find an inner document. Linking too much meant that I had to make lots of extra data calls. Duplicating too much meant I ran the risk of out of sync data. I still haven’t quite settled on a good schema, so I took another step back and tried a different approach.

Typescript

Then I decided to give typescript a try. Since I wasn’t making good progress maybe doing things with typescript would help the ideas gel. At first, again, this was great. I got strong typing, succinct lambdas syntax, cleaner classes and functions, etc. Since I was doing so well, I thought that maybe I’d try and strongly type the parts of mongoose that I had working for my app. Here I hit another roadblock. I wanted to map a function proxy that mongoose gives you to a strongly typed class declaration. This lead me down to reading about ambient declarations in typescript, poring over the typescript spec, furiously searching every typescript stackoverflow post and blog out there. I also had to learn how modules are loaded with CommonJS. On top of all of that, I ran into a problem with running unit tests using nodeunit with typescript (though I finally did figure this out, the trick is to export a variable argument that has references to your testing class functions). At one point I even managed to crash the typescript compiler!

What’s next?

I’m at the point now where I have learned a lot, some things still don’t work, but I need to sit back and take a break. It’s disheartening failing to make progress at every turn you make, but that’s the way you learn. Without major failures you don’t come to appreciate the nuances of how things work and how things are pieced together. I’ll come back to this project in a few weeks and probably feel a whole lot better about it. In the end, what I’ve been reminded of, is that there’s no shame in sometimes failing hard. Really really hard.