My experimental project to start Node was a Twitter clone in Redis. I was experimenting with Redis, and reading antirez' article gave me the idea to the same but using Node.
[A note for the interested: Not long after I started this, I realized I had made two mistakes: Coffeescript and AngularJS. Not that they weren't any good. But I should have resisted my curiosity and keep new concepts to a minimum. I've struggled with unnecessary setup and bug hunting steps for Coffeescript, which I've blogged here. For Angular, I was quite enthusiastic about trying it but it wasn't necessary for this project and I ended doing some weird things with it. It is a big and vast framework and better left alone if you're in the phase of learning other things.]
Table of Contents
- Moving parts
- Folder structure
- Architecture & Dependencies
- Mocha, the test runner
- Change a file
- Deployment (and source control)
- Conclusion and further exploration
Forget Node for a second. Just think of the parts you need to consider while building a dynamic website. The first logical separation is probably the server side and the frontend. Then you will have tests, for both sides, which will preferably run on every time you make a change to a file. You will have a development environment (local servers, etc) and from time to time you would deploy to production. Your production and development environment will have different settings. So you will want your code to adapt to these environments without intervention. And you would obviously automate these things as much as possible.
In the following sections, I will go on these parts and try to build a stable development environment where it's possible to focus only to the problems in hand instead of struggling (like I did) with infrastructure machinery.
A high level view of my project folders is as follows:
[retwisn] ├── [node_modules] ├── [src] │ ├── [app] │ │ ├── [common] │ │ ├── [home] │ │ ├── [post] │ │ ├── [user] │ │ ├── [views] │ │ └── app.coffee │ └── [web] │ ├── [css] │ └── [scripts] ├── [test] ├── [vendor] ├── [target] ├── [dist] ├── Gruntfile.coffee └── package.json
Coming from Java world, I started this project with the more common structure of model, view, controller and service folders. In this approach you divide your files per type. Your controllers go to the controller folder, your services go to the service folder, etc.
But what does this separation convey to the reader of your code? Maybe she would understand that I'm using a MVC pattern but so what? Instead of this, I've chosen a separation by feature. If we look to the
app folder, we see components separated by what they are responsible from, not by how. The only exception is
views folder where I had to put all my views, forming an organization per type, not feature. This was dictated by the view engine mechanism of the Express framework. There might be ways to overcome this but I didn't look further.
(If you would like to listen more about per type or per feature organization of modules, I recommend a very good talk by Uncle Bob: Architecture, The Lost Years)
Since I'm developing a Twitter clone, two most important entities of the domain is users and posts. Let's look inside the
post folder to further elaborate:
├[post] ├── post-api.coffee ├── post-controller.coffee └── post-service.coffee
We have all the components for posting grouped here. Filenames tell clearly what they are. I decided not to use model entities for
Post, since they were very simple, but if I would, this folder would have also contain a
common folder contains other configuration files and common modules. One important member of this folder is
routes.coffee which we will explore in the next section.
web folder is the container for static files. All your scripts, images, stylesheets and templates go here. It's where your frontend lives.
We'll see more about the
dist on following sections but for now,
target is automatically kept in sync with your
src folder. It's where you run your local server.
dist is the folder you build your project for deployment. In my case, I push to a remote repo for Heroku. Scripts in this folder should be compiled, minified, concatenated and whatever else you need to do for the latest version of your application to live in production.
A very simple (http) request flow of this project is like this:
----- ------------------------------------------------ ------- | Web | --> | App --> Middlewares --> Controller --> Service | --> | Redis | ----- ------------------------------------------------ -------
A request coming from web is accepted by the http server of Express and passes through different middlewares. One of them is the Router middleware. Router, by examining the request decides which controller should handle it and passes the request to the controller.
Controller, being in general a thin layer, asks from Service layer to do its business related jobs. In our sample application, this means reading from and writing to Redis. Once Redis answers, this goes all the way back. Everything in this scenario is in fact asynchronous. I won't go into the details about the asynchronous nature of Node. Every article you'll find on Node will probably start by explaining asynchronous callbacks and the Event Loop in Node.
On the other hand, you can see the diagram as a dependency hierarchy. Every component in the middle is dependent on the right one. Most importantly, Controller needs a Service to do its job, and Service needs Redis.
For testing purposes, it's extremely important that those dependencies shouldn't be hard coded. The basic rule is you are never to see
new in your code. There are many techniques to break these dependency chains. I employed constructor injection. I have one single factory called
provider.coffee. It's responsible of creating a Redis client, and services. The single other place where object creation happens is in the
routes.coffee. It's where the request path / controller mapping happens.
provider is environment aware, so while running on localhost it looks for the Redis server with default settings, on the other hand, if in production, it gets the connection information from environment settings.
(For further reading on Dependency Injection, you can start with Martin Fowler's article)
Let's see a small part of
# USER userController = new UserController provider.getUserService() app.post '/signup', userController.createUser app.post '/api/:followingUsername/follows/:followedUsername', userController.follow userAPI = new UserAPI provider.getUserService() app.post '/api/isFollowing', userAPI.isFollowing app.get '/api/:username/followedBy', userAPI.followers
In any but extremely small projects, you'll have many routes to handle. Putting them in the main
app.coffee becomes quickly a bit cumbersome. This is a handy way to separate route definitions to its own file, by passing the
app reference to the exported function. This is also where you create your controllers, by passing services (asked from provider) to their constructors.
Mocha is the test runner I've used for my tests. I'll only give some tips that I've found useful:
mocha.optsfile, and especially
--requireoption. It saves you from passing arguments and requiring frequently used frameworks, like Sinon.js. Here is mine:
--compilers coffee:coffee-script --recursive --reporter spec --require sinon --require should
done, a callback for your asynchronous tests, accepts also an error. You can pass it directly to your async method, instead of calling it and asserting separately.
Well, if I had to chose the single most important part of any Node project (or maybe any project at all), I would probably vote for the build system. This is more important than you might think in the first place, because:
- If you use Coffeescript or other compilation steps with your styling or templates, it's better to leave your source tree clean, and use a separate folder (
targetin my case) for running your local server.
- Having an easy way of running the same steps every time you release helps more than you think, minimizing human errors.
I'll explain briefly my development workflow. While in development, I run grunt with default settings and let it run. Here is what it does for me:
- For the first run:
- It cleans the target folder.
- For every
.coffeefile in my source folder it runs
coffeelintto check for errors and consistency problems.
- It runs all unit tests, under
- It compiles all
srcfolder into the
- It copies every file from
target\weband also every file in
.coffee, more on this later) to
- It runs
nodemontasks in parallel. Again, I'll cover the detail later on.
- Then, when I make an individual changes to a file:
watchkicks-in and executes the above steps again, but only for the changed file.
nodemonreflect this change to your running server and to the browser.
I have another
dist task which prepares the deployment package. No source maps, no coffee files, concatenated and minified files, etc. (Btw, I don't concatenate and minify my files in the example project, but I would, if this was a real project, and this would be the task to do it)
In conclusion, for any project which takes more then a day, I recommend a build system. And as far as I know, Grunt is the standard.
What happens when you change a file depends on where and what type of file you change. Let's see for different types:
- This files are your application source files and if you change one, you would normally restart Node to see its effects. This is covered by both
nodemontasks in my workflow. When
watchdetects a change in a
appfolder, it executes the necessary steps and copy the
mapand the resulting
nodemonwatching this folder, restarts node.
- This files do not require a restart of the node server. But they require all the steps executed by
watchtask, and they end up copied in
target/web. While optional,
liveReloadtakes care of reflecting the changes to the browser without a reload of the page.
- Being template files, they do not require restarting. We only copy them to target folder. It's not something that can be sent to the browser, thus
liveReloaddo a refresh on the page.
- - other files in
watchcopies them to the
liveReloadsends them to the browser causing an immediate change. (No refresh needed)
I chose Heroku for hosting, so part of the deployment steps are somewhat specific to it.
I have a branch, a folder and a task, all called
dist. This might be confusing explaining but comes handy when using them.
dist folder is generated by my grunt's
dist task. This folder is ignored by git in the master branch. I also ignore the root
/node_modules folder. When I want to deploy to Heroku, I checkout the dist branch, and run
grunt dist. This task creates the
dist folder, and copies the
/node_modules folder inside. Note that the
dist folder is not ignored. Then, I push the folder to the remote Heroku repo using
git subtree push --prefix dist heroku master
Subtree allows you to push only a part of your repository. There are two tricks to keep in mind, in dist branch, only the root
/node_modules is ignored, the one inside
dist folder is pushed along with the folder. This prevents a long
npm install executed by Heroku every time you deploy your app. The second is the
dist folder should be ignored in master branch. It shouldn't be part of your source code.
As I tried to stress in the beginning, this guide represents my opinions on how to structure a Node application. But since I just began the adventure, those opinions will probably change along the road. The project accompanying the post is by no means a finished product. While doing it, I stumbled on many other interesting stuff. I will most probably use them in my next project. So I suggest you to take a look to:
- Yeoman: Its motto is "Modern Workflows for Modern Webapps". By directly copying from their site: "Our workflow is comprised of three tools for improving your productivity and satisfaction when building a web app: yo (the scaffolding tool), grunt (the build tool) and bower (for package management)."
- Q: Introduces
promises, among others, a way to flatten the pyramid of async callbacks.
- Winston: A multi-transport async logging library for node.js
Your feedback and suggestions is more than welcomed, so don't hesitate to drop a line.