Build a Node API Client - Part 2: Encapsulation, Resources, & Architecture

Welcome to Part Two of our series on Node.js Client Libraries. This post serves as our guide to REST client design and architecture. Be sure to check out Part One on Need-To-Know RESTful Concepts before reading on.

API Encapsulation

Before sinking our teeth into resources and architecture, let’s talk about encapsulation. At Stormpath, we like to clearly separate the public and private portions of our API client libraries, aka ‘SDKs’ (Software Development Kit).

All private functionality is intentionally encapsulated, or hidden from the library user. This allows the project maintainers to make frequent changes like bug fixes, design and performance enhancements, all while not impacting users. This leads to a much greater level of maintainability, allowing the team to deliver better quality software, faster, to our user community. And of course, an easier-to-maintain client results in less friction during software upgrades and users stay happy.

To achieve this, your Node.js client should only expose users to the public version of your API and never the private, internal implementation. If you’re coming from a more traditional Object Oriented world, you can think of the public API as behavior interfaces. Concrete implementations of those interfaces are encapsulated in the private API. In Node.js too, functions and their inputs and output should rarely change. Otherwise you risk breaking backwards compatibility.

Encapsulation creates lot of flexibility to make changes in the underlying implementation. That being said, semantic versioning is still required to keep your users informed of how updates to the public will affect their own code. Most developers will already be familiar semantic versioning, so it’s an easy usability win.

Encapsulation In Practice

We ensure encapsulation primarily with two techniques: Node.js module.exports and the ‘underscore prefix’ convention.

module.exports

Node.js gives you the ability to expose only what you want via its module.exports capability: any object or function in a module’s module.exports object will be available to anyone that calls.

This is a big benefit to the Node.js ecosystem and helps improve encaspsulation goals better than traditional JavaScript environments.

Underscore Names

Additionally we use the ‘underscore prefix’ convention for objects or functions that are considered private by the development team but still accessible at runtime because of JavaScript’s weak encapsulation behavior. That is, any object or function that starts with the underscore _ character is considered private and its state or behavior can change, without warning or documentation, on any given release.

The takeaway is that external developers should never explicitly code against anything that has a name that starts with an underscore. If they see a name that starts with an underscore, it’s simply ‘hands off’.

Alternatively, other libraries use @public and @private annotations in their JS docs as a way of indicating what is public/allowed vs. private/disallowed. However, we strongly prefer the underscore convention because anyone reading or writing code that does not have immediate access to the documentation can still see what is public vs private. For example it is common when browsing code in GitHub or Gists that annotations in documentation are not easily available. However, you can still always tell that underscore-prefixed methods are to be considered private.

Either way, you need to consistently convey which functions to use and which to leave alone. You may want to omit the private API from publicly hosted docs to prevent confusion.

Public API

The public API consists of all non-private functions, variables, classes, and builder/factory functions.

This may be surprising to some, but object literals used as part of configuration are also part of the public API. Think of it like this: if you tell people to use a function that requires an object literal, you are making a contract with them about what you support. It’s better to just maintain backwards and forwards compatibility with any changes to these object literals whenever possible.

Prototypical OO Classes

We use prototypical inheritance and constructor functions throughout the client, but the design reflects a more traditional OO style. We’ve found this makes sense to most of our customers of all skill/experience levels.

Stormpath is a User Management API, so our classes represent common user objects like Account, in addition to more generic classes, like ApiKey. A few classes used as examples in this post:

• Client
• ApiKey
• Application
• Directory
• Account

Builder Functions

Node.js and other APIs often use method chaining syntax to produce a more readable experience. You may have also heard of this referred to as a Fluent Interface.

In our client, it’s possible to perform any API operation using a client instance. For example, getApplications obtains all Applications by using the client and method chaining:

client.getApplications()
.where(name).startsWith(‘foo’)
.orderBy(name).asc()
.limit(10)
.execute(function (err, apps){
  ...
});

There are two important things to note from this getApplications example:

1. Query construction with where, startsWith and orderBy functions is synchronous. These are extremely lightweight functions that merely set a variable, so there is no I/O overhead and as such, do not need to be asynchronous.
2. The execute function at the end is asynchronous and actually does the work and real I/O behavior. This is always asynchronous to comply with Node.js performance best practices.

Did you notice getApplications does not actually return an applications list but instead returns a builder object?

A consistent convention we’ve added to our client library is that get* methods will either make an asynchronous call or they will return a builder that is used to make an asynchronous call.

But we also support direct field access, like client.foo, and this implies a normal property lookup on a dictionary and a server request will not be made.

So, calling a getter function does something more substantial. Both still retain familiar dot notation to access internal properties. This convention creates a clear distinction between asynchronous behavior and simple property access, and the library user knows clearly what to expect in all cases.

Writing code this way helps with readability too – code becomes more simple and succinct, and you always know what is going on.

Base Resource Implementation

The base resource class has four primary responsibilities:

1. Property manipulation methods – Methods (functions) with complicated interactions
2. Dirty Checking – Determines whether properties have changed or not
3. Reference to DataStore – All our resource implementations represent an internal DataStore object (we’ll cover this soon)
4. Lazy Loading – Loads linked resources

Resource and all of its subclasses are actually lightweight proxies around a DataStore instance, which is why the constructor function below takes two inputs:

1. data (an object of name/value pairs)

2. A DataStore object.

    var utils = require('utils');
   
    function Resource(data, dataStore) {
   
      var DataStore = require('../ds/DataStore');
   
      if (!dataStore && data instanceof DataStore){
        dataStore = data;
        data = null;
      }
   
      data = data || {};
   
      for (var key in data) {
        if (data.hasOwnProperty(key)) {
          this[key] = data[key];
        }
      }
   
      var ds = null; //private var, not enumerable
      Object.defineProperty(this, 'dataStore', {
        get: function getDataStore() {
          return ds;
        },
        set: function setDataStore(dataStore) {
          ds = dataStore;
        }
      });
   
      if (dataStore) {
        this.dataStore = dataStore;
      }
    }
    utils.inherits(Resource, Object);
   
    module.exports = Resource;
   

When CRUD operations are performed against these resource classes, they just delegate work to the backend DataStore. As the DataStore is a crucial component of the private API, we keep it hidden using object-defined private property semantics. You can see this in practice with the public getters and setters around the private attribute above. This is one of the few ways to implement proper encapsulation in JavaScript.

If you remember to do just two things when implementing base resource classes, let them be:

1. Copy properties over one-to-one
2. Create a reference to a DataStore object to use later

Base Instance Resource Implementation

InstanceResource is a subclass of Resource. The base instance resource class prototypically defines functions such as save and delete, making them available to every concrete instance resource.

Note that the saveResource and deleteResource functions delegate work to the DataStore.

var utils = require('utils');
var Resource = require('./Resource');

function InstanceResource() {
  InstanceResource.super_.apply(this, arguments);
}
utils.inherits(InstanceResource, Resource);

InstanceResource.prototype.save = function saveResource(callback) {
  this.dataStore.saveResource(this, callback);
};

InstanceResource.prototype.delete = function deleteResource(callback) {
  this.dataStore.deleteResource(this, callback);
};

In traditional object oriented programming, the base instance resource class would be an abstract. It isn’t meant to be instantiated directly, but instead should be used to create concrete instance resources like Application:

var utils = require('utils');
var InstanceResource = require('./InstanceResource');

function Application() {
  Application.super_.apply(this, arguments);
}
utils.inherits(Application, InstanceResource);

Application.prototype.getAccounts = function
getApplicationAccounts(/* [options,] callback */) {
  var self = this;
  var args = Array.prototype.slice.call(arguments);
  var callback = args.pop();
  var options = (args.length > 0) ? args.shift() : null;
  return self.dataStore.getResource(self.accounts.href, options,
                                    require('./Account'), callback);
};

How do you support variable arguments in a language with no native support for function overloading? If you look at the getAccounts function on Applications, you’ll see we’re inspecting the argument stack as it comes into the function.

The comment notation indicates what the signature could be and brackets represent optional arguments. These signal to the client’s maintainer(s) (the dev team) what the arguments are supposed to represent. It’s a handy documentation syntax that makes things clearer.

...
Application.prototype.getAccounts = function
getApplicationAccounts(/* [options,] callback */) {
  ...
}
...

options is an object literal of name/value pairs and callback is the function to be invoked. The client ultimately directs the work to the DataStore by passing in an href. The DataStore uses the href to know which resource it’s interacting with server-side.

Usage Paradigm

Let’s take a quick look at an example JSON resource returned by Stormpath:

{
  “href”: “https://api.stormpath.com/v1/accounts/x7y8z9”,
  “givenName”: “Tony”,
  “surname”: “Stark”,
  ...,
  “directory”: {
    “href”: “https://api.stormpath.com/v1/directories/g4h5i6”
  }
}

Every JSON document has an href field that exists in all resources, everywhere. JSON is exposed as data via the resource and can be referenced via standard dot notation like any other JavaScript object.

Note: Check out this blog post on linking and resource expansion if you’re wondering how we handle linking in JSON.

Proxy Pattern

Applications using a client will often have an href for one concrete resource and need access to many others. In this case, the client should support a method (e.g. getAccount) that takes in the href they have, to obtain the ones they need.

String href = 'https://api.stormpath.com/v1/...etc...';

client.getAccount(href, function(err, acct) {
  if (err) throw err;

  account.getDirectory(function(err, dir) {
    if (err) throw err;
    console.log(dir);
  });
});

In the above code sample,getAccount returns the corresponding Account instance, and then the account can be immediately used to obtain its parent Directory object. Notice that you did not have to use the client again!

The reason this works is that the Account instance is not a simple object literal. It is instead a proxy, that wraps a set of data and the underlying DataStore instance. Whenever it needs to do something more complicated than direct property access, it can automatically delegate work to the datastore to do the heavy lifting.

This proxy pattern is popular because it allows for many benefits, such as programmatic interaction between references, linked references, and resources. In fact, you can traverse the entire object graph with just the initial href! That’s awfully close to HATEOS! And it dramatically reduces boilerplate in your code by alleviating the need to repeat client interaction all the time.

So how does this work under the hood? When your code calls account.getDirectory, the underlying (wrapped) DataStore performs a series of operations under the hood:

1. Create the HTTP request
2. Execute the request
3. Receive a response
4. Marshal the data into an object
5. Instantiate the resource
6. Return it to the caller

Client Component Architecture

Clearly, the DataStore does most of the heavy lifting for the client. There’s actually a really good reason for this model: future enhancements.

Your client will potentially handle a lot of complexity that is simpler in the long run to decouple from resource implementations. Because the DataStore is part of the private API, we can leverage it to plugin new functionality and add new features without changing the Public API at all. The client will just immediately see the benefits.

Here is a really good example of this point. The first release of our SDK Client did not have caching built in. Any time a Stormpath-backed app called getAccount, getDirectory, or any number of other methods, the client always had to execute an HTTP request to our servers. This obviously introduced latency to the application and incurred an unnecessary bandwidth hit.

However our DataStore-centric component architecture allowed us to go back in and plug in a cache manager. The instant this was enabled, caching became a new feature available to everyone and no one had to change their source code. That’s huge.

Anyway, let’s walk through the sequence of steps in a request, to see how the pieces work together.

First, the DataStore looks up the cache manager, finds a particular region in that cache, and checks if the requested resource is in cache. If it is, the client returns the object from the cache immediately.

If the object is not in cache, the DataStore interacts with the RequestExecutor. The RequestExecutor is another DataStore component that in turn delegates to two other components: an AuthenticationStrategy and the RequestAuthenticator.

REST clients generally authenticate by setting values in the authorization header. This approach is incredibly convenient because it means swapping authentication strategies is a simple matter of changing out the header. All that is required is to change out the AuthenticationStrategy implementation and that’s it – no internal code changes required!

Many clients additionally support multiple/optional authentication schemes. More on this topic in part 3.

After authentication, the RequestExecutor communicates the outgoing request to the API server.

Finally, the ResourceFactory takes the raw JSON returned by the API server and invokes a constructor function to create the instance resource that wraps (proxies) this data, and again, the DataStore.

All of the client components represented in this diagram should be pluggable and swappable based on your particular implementation. To make this a reality as you architect the client, try to adhere to the Single Responsibility Principle: ensure that your functions and classes do one and only one thing so you can swap them out or remove them without impacting other parts of your library. If you have too many branching statements in your code, you might be breaking SRP and this could cause you pain in the future.

And there you have it! Our approach to designing a user-friendly and extremely maintainable client to your REST API. Check back for Part Three and a look at querying, authentication, and plugins!

API Management with Stormpath

Stormpath makes it easy to manage your API keys and authenticate developers to your API service. Learn more in our API Key Management Guide and try it for free!