Creating a Software Ecosphere: MyBooks Tutorial: Step 3 – Node RESTful Webservices – Overview and Setup

Welcome Back!

What’s what in this step

If you’re new to the series, you can see all the current steps here.  As always, the code is available on github, the main branch will contain all the steps to date, and I’ve branched the steps starting at step003 (This one).  I’m not going to be documenting every step in detail, so I’d strongly suggest you either use the github branch code directly, or have it open as you follow along. If you’re all caught up, then let’s be about it…

Today we’ll take the first steps in creating our RESTful webservices framework on Node.   Originally I’d planned on doing this using Ruby, but I’ve been looking at some performance numbers and decided to use Node.  After I complete this series, I hope to loop back and do a comparison of RESTful webservices on/in Node, Ruby, PHP, Java and possibly C#.

Arguably, there are an infinite number of ways of setting up a project, some or more effective than others depending on your environment, application space, and organization.  First off, this tutorial is going to include the RESTful API, a web app, and a iOS app.   But because I’m doing it as a tutorial, I want to include all the source in one repository.   Normally, that’s not at all how I’d approach it.   I’d create a different repository for each element.   So, my underlying directory structure may seem a bit odd at first.    If you’re using an IDE, you’ll want to do the git work at the command line level from the root directory (mybooks for me) instead of using the IDE to do your git management.  To give you an idea of what we’ll be doing in creating the API, I’d like to go over some of the Givens and Druthers that I’ll be using as guidelines for this phase:

Givens and Druthers for the RESTful API

  • We’ll be using ES6 on the server.
  • I reserve the right to NOT use ES6 features.   I will try to explain why, but if I don’t you’re welcome to ask.
  • We will NOT be using a transpiler.
  • We will continue to use require.  Import/export isn’t supported by node (v8.1.2).  See James Snell’s [IBM Technical Lead for Node.js] article on the subject.  Transpiling isn’t worth the overhead just for import/export.
  • We won’t be using a task runner, but I reserve the right to change my mind (KISS).
  • Here are most of the packages/tools we’ll be using for the API:
    • node: version 8.1.2, but if it’s later you should be fine.
    • npm: yarn is fast, and I like the syntax better, but I’ve had issues with bad optional dependency packages (package fault, not yarn’s, and I’d rather not spend the time digging into it for a tutorial.
    • webstorm:   Good tools save you time and money.   JetBrains makes good ones, and I believe in buying good tools (Here that JetBrains?   I wouldn’t mind some free swag!). You don’t want a Harbor Freight IDE that breaks when you need it most or worse, gets in the way of writing code by making spend all your time configuring it and figuring out why it’s not working.   Hell, it costs probably a fraction of what you spend on coffee.   You can use whatever floats your boat, but if you use emacs I reserve the right to make fun of you.   I spend a lot of time working from the command line, so I won’t be going over how I use it too much (unless you ask).   P.S. If the swag comment was confusing, I pay for my own licenses.
    • restify: Later on I hope to do some benchmarks of restify vs express for RESTful API, but for now I’ll rely on my experience.   In general a multi function library/framework will be slower than a targeted library/framework.   I did sufficient (IMNSHO) investigation without overanalyzing (ok, maybe I overanalyzed a bit).   Good documentation, good support, good looking code, just a tinch of dependency bloat, but pretty slim.
    • restify-router: This will help us keep the app.js clean and allow us to better organize and maintain our routes.
    • sequelize: I debated on using raw sql and writing my own DAOs (small tight fast code) as what we’re doing here isn’t rocket surgery.   But sequelize seems like it’s a fairly tight ORM.   We’ll see.   Some of their docs seem to have a weird view on persistence development lifecycle (IMNSHO). But I can see where it might be more effective in a persistence prototyping environment.   I tend to think in terms of data laying the groundwork for the code, and I do a fair amount of analysis to that end.  Anyhoo, I have an upcoming project I think will lend itself to putting sequelize through its prototyping paces, so here it is.
    • mysql2: Why are you using mysql still?
    • ESLint: Not much to say here.   It does what I want, and does it well.
    • Airbnb Style Guide:  You need a style guide.   A lot of people have written a lot about this subject.   I like what Alec Moldovan wrote., and mostly agree with it.  For example, I too am a two spaces indentation guy, but not for bracketing and parentheses.   I’ll be using my own overrides which can be found in the .eslintrc in the repository.
    • mocha & chai: We’re doing TDD here.  Well… mostly 😉  I’m going to try and get away with not using a task runner (KISS), but I like the mocha & chai combo for testing.
    • dotenv: Allows us to load .env files into process.env.   This is a great way to store your username/passwords on your dev machine.

Setting up Node

So, we’re working in node.  In the past I’ve used brew to install node, but just recently, I’ve switched to using the node native installer.   Not for any specific issues, but I’d rather do the updates via nvm after install than rely on brew and its linkages.   (NOTE: The node install will also install npm).   Make sure you use the current version, not the LTS version!   Node should be v8.1.2 or greater, and npm should be v5.0.3 or greater.

Create the mybooks-api directory (sibling to the sql directory) move into it and create the package.json with “npm init -y“.  Edit it with the following changes:

  • “main”: change this to app.js
  • “description”: you’ll get a warning if you don’t fill this in.
  • “author”: your name here 🙂
  • “license”:  I released my source for this under the MIT license, change it to MIT

And we need to add in two entries.    Repository, because I hate seeing warning messages, and the node version because it’s a good habit to get into:

  • repository“: “rpcarver/mybooks” – this is typical github repository notation.   If you want to use your own (not github) repository server see the package.json documentation on how to set it.
  • engines: { “node”: “8.1.2” } – this lets anyone downstream know that I developed and tested this api on node 8.1.2.   Without extra flags, its advisory only.

Here’s my package.json at this stage:

{
  "name": "mybook-api",
  "version": "1.0.0",
  "description": "MyBooks Software Ecosphere Tutorial - RESTful API implemented with node",
  "main": "app.js",
  "author": "Randy Carver",
  "license": "MIT",
  "repository": "rpcarver/mybooks",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "engines": {
    "node": "8.1.2"
  }
}

Adding packages dependencies

Now that the package.json is initialized, let’s add in the packages we’ll be using.   I’ve put what I’d recommend as globals first.   I’ve also listed the binaries we’ll be using for each package so you can decide to install it globally or .   Read this if you’re not familiar with the difference between global and local.

  • Normal dependencies:
    • dotenv
    • mysql2
    • restify
    • restify-router
    • sequelize

Install copy/paste:

npm install --save --save-exact dotenv mysql2 restify restify-router sequelize
  • Development Dependencies:
    • chai
    • eslint – I recommend you install this globally as well!
    • eslint-config-airbnb
    • eslint-config-airbnb-base – dependency of eslint-config-airbnb
    • eslint-plugin-import- dependency of eslint-config-airbnb
    • eslint-plugin-jsx-a11y- dependency of eslint-config-airbnb
    • eslint-plugin-react- dependency of eslint-config-airbnb
    • sequelize-auto
npm install --save-dev --save-exact chai eslint eslint-config-airbn eslint-config-airbnb-base \
  eslint-plugin-import eslint-plugin-jsx-a11y eslint-plugin-react sequelize-auto

If you’re using webstorm, now is a good time to configure eslint and the airbnb style guide settings.  If you’re working off of the git files, I included my .eslintrc.

Create sequelize models from mysql using sequelize-auto

Assuming you’re all set to write some code, we’re going to use sequelize-auto to import the models from the database.  This is only a one time run, and we could run sequelize-auto from the command line, but let’s toss the code into the app.js and run it from there. So in the mybook-api directory (where the package.json file is) create app.js:

const SequelizeAuto = require('sequelize-auto');

const auto = new SequelizeAuto('mybooks', 'your-user-name', 'your-password', {
  host: 'localhost',
  dialect: 'mysql',
  port: '3306',
  additional: {
    timestamps: false,
  },
});

auto.run(function (err) {
  if (err) throw err;
});

And then we add a script to the package.json.  NOTE: we’ll learn about the NODE_ENV being set in just a little.   It doesn’t change how the import works and saves us the hassle of writing it again when I talk about NODE_ENV:

{
  "name": "mybook-api",
  "version": "1.0.0",
  "description": "MyBooks Software Ecosphere Tutorial - RESTful API implemented with node",
  "main": "app.js",
  "author": "Randy Carver",
  "license": "MIT",
  "repository": "rpcarver/mybooks",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "NODE_ENV=development node app.js"
  },
  "engines": {
    "node": "8.1.2"
  },
  "dependencies": {
    "dotenv": "4.0.0",
    "mysql2": "1.3.5",
    "restify": "4.3.0",
    "restify-router": "0.4.2",
    "sequelize": "4.2.0"
  },
  "devDependencies": {
    "eslint": "4.1.1",
    "eslint-config-airbnb-base": "11.2.0",
    "eslint-plugin-import": "2.6.0",
    "sequelize-auto": "0.4.28"
  }
}

Don’t forget to add the comma after the test placeholder.   Now, we can’t run it yet.  While sequelize has been updated to use mysql2, sequelize-auto has not.   So we need to install mysql temporarily, then run app.js, then uninstall it:

npm install mysql 
npm start
npm uninstall mysql

We should now have a models directory with authors.js, books.js, books_authors.js, formats.js, locations.js, and publishers.js.   These models will be imported into a sequelize instance before we can use them.  Open up books.js and notice that the FK references are properly configured (publisherID, formatID, locationName):

module.exports = function(sequelize, DataTypes) {
 return sequelize.define('books', {
  bookID: {
   type: DataTypes.INTEGER(11),
   allowNull: false,
   primaryKey: true,
   autoIncrement: true
  },
  title: {
   type: DataTypes.STRING(100),
   allowNull: false
  },
  publisherID: {
   type: DataTypes.INTEGER(11),
   allowNull: true,
   references: {
    model: 'publishers',
    key: 'publisherID'
   }
  },
  ISBN: {
   type: DataTypes.INTEGER(13),
   allowNull: true
  },
  printingYear: {
   type: DataTypes.INTEGER(11),
   allowNull: true
  },
  printingNum: {
   type: DataTypes.INTEGER(11),
   allowNull: true
  },
  formatID: {
   type: DataTypes.INTEGER(11),
   allowNull: true,
   references: {
    model: 'formats',
    key: 'formatID'
   }
  },
  rating: {
   type: DataTypes.INTEGER(11),
   allowNull: true
  },
  notes: {
   type: DataTypes.STRING(500),
   allowNull: true
  },
  locationName: {
   type: DataTypes.STRING(45),
   allowNull: false,
   references: {
    model: 'locations',
    key: 'locationName'
   }
  }
 }, {
  tableName: 'books',
  timestamps: false
 });
};

Setting up relationships in the sequelize models

Unfortunately sequelize ORM does not directly interpret these foreign keys as relationships between objects.   We will need to use the association methods to do that.  Sequelize exposes the several methods to setup the relations.   For the relations to work correctly on deep operations we will have to implement both sides, represented in the information below as the paired operation. Also note that relations only work for models that have been imported to the sequelize instance.  More on that later, here are the methods:

  • Methods:
    • belongsTo: BelongsTo associations are associations where the foreign key for the one-to-one relation exists on the source model. It also denotes half of a one to many relationship.
    • hasOne: HasOne associations are associations where the foreign key for the one-to-one relation exists on the target.
    • belongsToMany: BelongsToMany is used to denote half of a one to many relationship.
    • hasMany: HasMany is used to specify both halfs of a many to many relationship and always requires the through attribute specifying the junction table.
  • One to One Paired method calls:
    • KeymasterModel.hasOne(GatekeeperModel);  // FK in KeymasterModel
      GatekeeperModel.belongsTo(KeymasterModel);  // FK in KeymasterModel
  • One to Many Paired method calls:
    • publishers.hasMany(books);  // FK in books
      books.hasOne(publishers);  // FK in books
  • Many to Many Paired method calls:
    • books.hasMany(authors, { through: books_authors });
      authors.hasMany(books, { through: books_authors });

There are a couple of ways to do this, we could load up all the models and then denote the relationships, or we could denote the relationships for each model as part of that model.   We will be doing the latter.   So bring up your favorite editor and let’s edit the authors model.   Basically we’re adding a template function that will then be used to add the relationships to the sequelize instance.  If you create the association without denoting the keys, sequelize will auto generate the key(s), which is not useful when we’ve established our data model before writing the ORM models.  I’ve denoted the changes in the model with strike through on deleted test and red text (green doesn’t show up too well on my screen) for added text:

module.exports = function(sequelize, DataTypes) {
  returnconst authors = sequelize.define('authors', {
    authorID: {
      type: DataTypes.INTEGER(11),
      allowNull: false,
      primaryKey: true,
      autoIncrement: true
    },
    lastName: {
      type: DataTypes.STRING(45),
      allowNull: true
    },
    firstName: {
      type: DataTypes.STRING(45),
      allowNull: true
    }
  }, {
    tableName: 'authors',
    timestamps: false
  });

  authors.associate = function (models) {
    models.authors.belongsToMany(models.books, { through: 'books_authors', foreignKey: 'authorID', otherKey: 'bookID' });
  };

  return authors;
};

Rinse and repeat for each model.   Here are the associate functions for each model:

books.associate = function (models) {
  models.books.belongsTo(models.publishers, { foreignKey: 'publisherID', targetKey: 'publisherID' });
  models.books.belongsTo(models.formats, { foreignKey: 'formatID', targetKey: 'formatID' });
  models.books.belongsTo(models.locations, { foreignKey: 'locationName', targetKey: 'locationName' });
  models.books.belongsToMany(models.authors, { through: 'books_authors', foreignKey: 'bookID', otherKey: 'authorID' });
};
/*
 * books_authors represents the junction table for the many to many relationship between books
 * and authors. The relationships are defined in the respective models instead of in books_authors.
 */
formats.associate = function (models) {
  models.formats.hasMany( models.books, {foreignKey: 'publisherID', sourceKey: 'publisherID'});
};
locations.associate = function (models) {
  models.locations.hasMany( models.books, {foreignKey: 'locationName', sourceKey: 'locationName'});
};
publishers.associate = function (models) {
  models.publishers.hasMany( models.books, {foreignKey: 'publisherID', sourceKey: 'publisherID'});
};

Add support for environment settings (this is where we learn about NODE_ENV 😉 )

Before we get to using the models, let’s talk about connecting to the database.  It’s never a good idea to store username/password information in a file on a production server.   There are lots of different ways to handle this, but for our purposes we’ll be using environment variables in production.   But a dev box is a different matter, and often it’s dead simple when we use a local environment file for test and development.   Just remember to add your environment file names to .gitignore so you don’t share them with the world ;).  We’re using dotenv to load the environment based on the NODE_ENV (“development” or “test”).   We’ll use our own env import to wrap dotenv.   Create the “config” directory at the root of the node project and add env.js:

const dotenv = require('dotenv');

// Load the environment vars for the correct environment.   Default to dev.
// YOU WILL NEED TO CREATE A .env.dev AND A .env.test file in the config directory!
if (process.env.DOTENV_LOADED !== 'true') {
  let envPath = null;
  switch (process.env.NODE_ENV) {
    case 'test': {
      envPath = { path: './config/.env.test' };
      break;
    }
    case 'development':
    default : {
      envPath = { path: './config/.env.dev' };
      break;
    }
  }
  dotenv.config(envPath);
  process.env.DOTENV_LOADED = true;
}

const env = {
  /*
   Set NODE_ENV as part of the task so its loaded BEFORE the env files are processed.
   e. g.
   "scripts": {
     "tests": "NODE_ENV=tests mocha"
     "start": "NODE_ENV=development node app.js"
   },
  */
  NODE_ENV: process.env.NODE_ENV || 'development',
  DATABASE_NAME: process.env.DATABASE_NAME || 'mybooks',
  DATABASE_HOST: process.env.DATABASE_HOST || 'localhost',
  DATABASE_PORT: process.env.DATABASE_PORT || 3306,
  DATABASE_USERNAME: process.env.DATABASE_USERNAME,
  DATABASE_PASSWORD: process.env.DATABASE_PASSWORD || '',
  DATABASE_DIALECT: process.env.DATABASE_DIALECT || 'mysql',
  DATABASE_POOL_MAX: process.env.DATABASE_POOL_MAX || 10,
  DATABASE_POOL_MIN: process.env.DATABASE_POOL_MIN || 1,
  DATABASE_POOL_IDLE: process.env.DATABASE_POOL_IDLE || 5000,
  DATABASE_QUERY_LOGGING: process.env.DATABASE_QUERY_LOGGING || false,
  API_VERSION: process.env.API_VERSION || 1.0,
  API_NAME: process.env.API_NAME || 'mybook-api',
  API_PORT: process.env.API_PORT || 8080,
};

module.exports = env;

In the project there are is example of .env file: env.example.   Use them it as a template for your very own .env.dev and .env.test:

NODE_ENV= either development or test
DATABASE_HOST=your_host
DATABASE_USERNAME=your_db_username
DATABASE_PASSWORD=your_db_password
DATABASE_SCHEMA= either mybooks-dev or mybooks-test
DOTENV_LOADED=true

Load the models and relationships into sequelize

Now we need to write the code to have sequelize import the models and define the relationships so we can use it to query the database.    In the models directory, add orm.js.

const fs = require('fs');
const path = require('path');
const Sequelize = require('sequelize');

const basename = path.basename(module.filename);
const env = require('../config/env');

const orm = {};

// connect to the database.
const sequelize = new Sequelize(env.DATABASE_NAME, env.DATABASE_USERNAME, env.DATABASE_PASSWORD, {
  host: env.DATABASE_HOST,
  port: env.DATABASE_PORT,
  dialect: env.DATABASE_DIALECT,
  pool: {
    max: env.DATABASE_POOL_MAX,
    min: env.DATABASE_POOL_MIN,
    idle: env.DATABASE_POOL_IDLE,
  },
  define: {
    timestamps: false,
  },
  logging: env.DATABASE_QUERY_LOGGING,
});

// Import the models
fs.readdirSync(__dirname)
  .filter(file => (file.indexOf('.') !== 0) && (file !== basename) && (file.slice(-3) === '.js'))
  .forEach((file) => {
    let model = sequelize.import(path.join(__dirname, file));
    orm[model.name] = model;
});

// Associate the models
Object.keys(orm).forEach((modelName) => {
  if (orm[modelName].associate) {
    orm[modelName].associate(orm);
  }
});


orm.sequelize = sequelize;
orm.Sequelize = Sequelize;

module.exports = orm;

Almost ready to write our first test

Before we write our first test, it’s always best to keep a dev and test database separate from production.   We should be able to run every db test from a known db state.   For EVERY test.   The tests should be atomic and not depend on the results of any other tests.   To do that we’re going to try and use sequelize to setup and teardown the mysql database down to the schema.   I’m not using an in memory db because part of what I want to test is that some of the unique indexes/key constraints are not violated.   In a strict environment, we might consider doing this via sql scripts.   Anyhoo, we need to go back to mysql and setup a test and a dev database.   Eventually we’ll add some seed data, but for our first test we just need to create the db so sequelize has something to work with.   Start up the cloud_sql_proxy and login to mysql, then create database mybook_dev; create database_mysql_test; :

$ mysql -u root -p --host 127.0.0.1
Enter password: 
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 154175
Server version: 5.7.14-google-log (Google)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create DATABASE mybook_test;
Query OK, 1 row affected (0.09 sec)

mysql> create DATABASE mybook_dev;
Query OK, 1 row affected (0.05 sec)

mysql>exit;
Bye
$

Check the Database Connectivity from sequilize…

Before we right the test, let’s make sure our settings are all in place correctly.   Open up app.js again and let’s put in some code to test the sequelize connection:

const env = require('./config/env');
const restify = require('restify');
const orm = require('./models/orm');

// NOTE: we are using the sequelize instance, not the class definition
orm.sequelize.authenticate()
.then(() => {
  console.log('DB Connection Established');
})
.catch((err) => {
  console.error('DB Connection Failed: ', err);
});

Make sure the cloud_sql_proxy is running, then run it either via your IDE or on the command line with npm start.   If you get an error, loop back and confirm all your settings.   Something is wrong somewhere along the line.   Once the connection is confirmed we can write our first test and we’re done for this step 🙂

FINALLY!  Tests!

Testing the ORM can be controversial.  Some will say running tests of the queries used by features is sufficient.   Others want to use a stubbed in memory db representation and test the features of the model against that.   Any way you slice it, you will probably find someone that says what you’re doing is not a “pure” unit test.   They’re right.   And it doesn’t matter.

<OPINION> (and you know what “they” say about opinions!)

I want the tests I write to uncover as many possible bugs while I’m still developing.   After blessing my code and committing it, I don’t want to find a bug during integration testing, automated or not. If the team I’m working with has dedicated QA, I really don’t want to get something back from QA.   And the last stop is production.   I’ve been on call for production issues.   Trust me, you don’t want to get called out in the middle of the night to find out why the services aren’t running.   So we’ll write tests that might be testing some of the internal ORM, might be testing some of the database constraints.   But it WILL be testing the code and the functionality of the app so that I feel comfortable about going camping and still being within my on call time requirements.   I like sleeping uninterrupted 🙂

</OPINION>

Alright, now that I’ve climbed off my soapbox, here’s our first test.   From the root of our node project, create a tests directory and a models subdirectory.   In this directory let’s add an authors-test.js file:

require('../../config/env');

const expect = require('chai').expect;
const orm = require('../../models/orm');

describe('author model - create and select to verify', () => {
  let author = null;
  before(async() => {
    expect(orm).to.have.own.property('authors');

    // drops the tables imported into the db and recreates them
    await orm.sequelize.sync({ force: true });
    author = await orm.authors.create({ lastName: 'Fred', firstName: 'Flintstone' });
  });

  it('should have created an author', async () => {
    expect(author.authorID).to.equal(1);
    const checkAuthor = await orm.authors.findById(author.authorID);
    expect(checkAuthor.firstName).to.equal(author.firstName);
    expect(checkAuthor.lastName).to.equal(author.lastName);
  });
});

I also wrote a test for the env.js file and dynamic env loading. You can take a look at that one by cloning the github repository and switching to branch step003.

We need to update the npm script so that mocha can run the test.   In package.json scripts, replace the test placeholder with the following:

"test": "NODE_ENV=test mocha --colors -t 30000 --reporter spec \"tests/**/*.js\"",

From the command line or your IDE:

npm test

> mybook-api@1.0.0 test /Users/rcarver/work/mybooks/mybook-api
> NODE_ENV=test mocha --colors -t 30000 --reporter spec "tests/**/*.js"



  test env
    ✓ should not have been loaded before test
    ✓ should set DOTENV_LOADED on load
    ✓ should set the correct properties

  dev env
    ✓ should not have been loaded before test
    ✓ should set DOTENV_LOADED on load
    ✓ should set the correct properties

  author model - create and select to verify
    ✓ should have created an author (48ms)


  7 passing (3s)

Wrap up

This should give you a pretty good start towards an understanding of sequelize models, how to import them from an existing mysql database, using the models, and testing them.   Additionally we’ve added an environment loader to allow align us with standard practices in securing our database authentication information.   Next step will be to flesh out the tests and start adding in the restful webservices.   As always, if you have questions, don’t hesitate to ask!

One thought on “Creating a Software Ecosphere: MyBooks Tutorial: Step 3 – Node RESTful Webservices – Overview and Setup”

Leave a Reply

Your email address will not be published. Required fields are marked *