In describing the ConcurrencyMaster, I referred to Bluebird promisify as a magic function. Of course, it’s not really magical. It’s just software. It only seemed magical because I didn’t understand how it worked. So this week, I’ve taken the opportunity to fill this hole in my knowledge by studying the internal workings of promisify. I like to learn about projects by fixing bugs in them, but Bluebird has no open bugs. Instead, we’ll just run through the working code and see what makes the magic happen. Here goes!
Bluebird promisify
promisify takes a function whose last argument is a callback, and it returns a new function. The new function takes the same arguments as the original function, except it doesn’t take a callback. The new function returns a promise that resolves when the original function would have called the callback successfully and rejects when the original function would have called the callback with an error. Here it is in action:
var Promise = require('bluebird');
var fs = require('fs');
// this is how you read a file without promisify
fs.readFile('/etc/profile', function(err, buffer) {
console.log('fs.readFile: ' + buffer.toString());
});
// this is the promisified version
var promisifiedRead = Promise.promisify(fs.readFile);
promisifiedRead('/etc/profile')
.then(function(buffer) {
console.log('promisified readFile: ' + buffer.toString());
});
Both of those will open up the /etc/profile file and print its contents. The promisified version is better for reasons that have been elaborately discussed elsewhere. promisify is defined in the aptly-named promisify.js file. Here it is:
var makeNodePromisified = canEvaluate
? makeNodePromisifiedEval
: makeNodePromisifiedClosure;
function promisify(callback, receiver, multiArgs) {
return makeNodePromisified(callback, receiver, undefined,
callback, null, multiArgs);
}
Promise.promisify = function (fn, options) {
if (isPromisified(fn)) {
return fn;
}
options = Object(options);
var receiver = options.context === undefined ? THIS : options.context;
var multiArgs = !!options.multiArgs;
var ret = promisify(fn, receiver, multiArgs);
util.copyDescriptors(fn, ret, propsFilter);
return ret;
};
Promise.promisify calls the function promisify with receiver and multiArgs arguments taken from the options object. receiver specifies an object to bind as this in executing the promisified function.
- receiver is used when promisifying a method on an object like a database client. It makes the promisified function still bind the client as the this object.
- multiArgs triggers special handling for functions like fs.open whose callbacks take multiple values. With multiArgs set, the arguments to the callback are passed as an array.
promisify has two different implementations based on the value of canEvaluate. canEvaluate is true when Bluebird is running in Node.js, and it is false when running in a browser. First, let’s check out the browser implementation, makeNodePromisifiedClosure.
makeNodePromisifiedClosure
Here’s makeNodePromisifiedClosure‘s implementation:
function makeNodePromisifiedClosure(callback, receiver, _, fn, __, multiArgs) {
function promisified() {
var _receiver = receiver;
if (receiver === THIS) _receiver = this;
var promise = new Promise(INTERNAL);
promise._captureStackTrace();
var fn = nodebackForPromise(promise, multiArgs);
try {
callback.apply(_receiver, withAppended(arguments, fn));
} catch(e) {
promise._rejectCallback(maybeWrapAsError(e), true, true);
}
return promise;
}
util.notEnumerableProp(promisified, "__isPromisified__", true);
return promisified;
}
makeNodePromisifiedClosure returns a function called promisified. promisified starts by determining the right object to use as this based on the context passed to Promise.promisify. Promise.promisify used THIS as the default receiver argument if no context was specified. THIS is actually a flag that says to use the local this created for the call to promisified.
Then, promisified instantiates a Promise object and uses the _captureStackTrace method to add a stack trace, which is useful for debugging. It uses this promise to construct a function fn using nodebackForPromise. Here’s nodebackForPromise:
function nodebackForPromise(promise, multiArgs) {
return function(err, value) {
if (promise === null) return;
if (err) {
var wrapped = wrapAsOperationalError(maybeWrapAsError(err));
promise._attachExtraTrace(wrapped);
promise._reject(wrapped);
} else if (!multiArgs) {
promise._fulfill(value);
} else {
INLINE_SLICE(args, arguments, 1);
promise._fulfill(args);
}
promise = null;
};
}
nodebackForPromise takes a promise argument and returns a function that resolves that promise with the value passed to the function if the err argument is falsy. If the err argument is truthy, it rejects the promise with the error value instead. It also handles the multiArgs logic we mentioned earlier.
So in our example, promisified calls fs.readFile with first argument '/etc/profile' and second argument the promise-resolving function fn returned by nodebackForPromise. It returns the promise that it passed to nodebackForPromise. fs.readFile‘s completion causes fn to be called, resolving or rejecting the promise that promisified had returned. Cool!
makeNodePromisifiedEval
Now that we’re warmed up, we can tackle the really crazy one, makeNodePromisifiedEval. makeNodePromisifiedEval has a similar idea to makeNodePromisifiedClosure, but it uses some elaborate hacks to take advantage of the optimization capabilities of the V8 Javascript engine that Node.js is built on. In particular, it wants to take advantage of function inlining for callback, which saves the (very high) cost of creating a closure when callback is called. Inlining doesn’t work on invocations of callback.apply, except in the case where the second argument to apply is the arguments object. Since we invoke apply on a new array, it can’t be inlined.
callback.call, on the other hand, can be inlined. The difference between apply and call is that apply takes an array of arbitrary length to use as the arguments to the function, whereas call has to explicitly name each argument. makeNodePromisifiedEval wants to use callback.call instead of callback.apply to reap massive inlining performance rewards. This means it has to know the number of arguments that callback takes and generate different code based on this number. Here’s how it looks:
makeNodePromisifiedEval =
function(callback, receiver, originalName, fn, _, multiArgs) {
var newParameterCount = Math.max(0, parameterCount(fn) - 1);
var body = "'use strict'; \n\
var ret = function (Parameters) { \n\
'use strict'; \n\
var len = arguments.length; \n\
var promise = new Promise(INTERNAL); \n\
promise._captureStackTrace(); \n\
var nodeback = nodebackForPromise(promise, " + multiArgs + "); \n\
var ret; \n\
var callback = tryCatch(fn); \n\
switch(len) { \n\
[CodeForSwitchCase] \n\
} \n\
if (ret === errorObj) { \n\
promise._rejectCallback(maybeWrapAsError(ret.e), true, true);\n\
} \n\
if (!promise._isFateSealed()) promise._setAsyncGuaranteed(); \n\
return promise; \n\
}; \n\
notEnumerableProp(ret, '__isPromisified__', true); \n\
return ret; \n\
".replace("[CodeForSwitchCase]", generateArgumentSwitchCase())
.replace("Parameters", parameterDeclaration(newParameterCount));
return new Function("Promise", "fn", "receiver", "withAppended", "maybeWrapAsError", "nodebackForPromise", "tryCatch", "errorObj", "notEnumerableProp", "INTERNAL", body)(Promise, fn, receiver, withAppended, maybeWrapAsError, nodebackForPromise, util.tryCatch, util.errorObj, util.notEnumerableProp, INTERNAL);
};
In order to create different functions depending on callback, makeNodePromisifiedEval uses the Function constructor. Function takes a list of strings enumerating the argument names of the function to be constructed. Our function here has a lot of arguments, starting with Promise and ending with INTERNAL. These arguments are all used to inject promisify.js’s global variables into the context of the created function, since otherwise they wouldn’t be available.
The last argument to Function is a string body that actually consists of Javascript source code. This source code forms the body of the constructed function. Our function here begins in the same way as the function returned by makeNodePromisifiedClosure, creating the new Promise(INTERNAL) and building a nodeback with it. The difference comes in the CodeForSwitchCase section. CodeForSwitchCase is a switch statement that invokes callback.call with varying numbers of arguments. If the promisified function is called with one of the numbers of arguments in this switch statement, we get to use callback.call instead of callback.apply.
Here’s how CodeForSwitchCase is calculated in the generateArgumentSwitchCase function:
var parameterCount = function(fn) {
if (typeof fn.length === "number") {
return Math.max(Math.min(fn.length, MAX_PARAM_COUNT + 1), 0);
}
//Unsupported .length for functions
return 0;
};
var argumentSequence = function(argumentCount) {
return util.filledRange(argumentCount, "_arg", "");
};
function generateCallForArgumentCount(count) {
var args = argumentSequence(count).join(", ");
var comma = count > 0 ? ", " : "";
var ret = "ret = callback({{args}}, nodeback); break;\n";
return ret.replace("{{args}}", args).replace(", ", comma);
}
var switchCaseArgumentOrder = function(likelyArgumentCount) {
var ret = [likelyArgumentCount];
var min = Math.max(0, likelyArgumentCount - 1 - PARAM_COUNTS_TO_TRY);
for(var i = likelyArgumentCount - 1; i >= min; --i) {
ret.push(i);
}
for(var i = likelyArgumentCount + 1; i <= PARAM_COUNTS_TO_TRY; ++i) {
ret.push(i);
}
return ret;
};
function generateArgumentSwitchCase() {
var newParameterCount = Math.max(0, parameterCount(fn) - 1);
var argumentOrder = switchCaseArgumentOrder(newParameterCount);
var ret = "";
for (var i = 0; i < argumentOrder.length; ++i) {
ret += "case " + argumentOrder[i] +":" + generateCallForArgumentCount(argumentOrder[i]);
}
ret += " \n\
default: \n\
var args = new Array(len + 1); \n\
var i = 0; \n\
for (var i = 0; i < len; ++i) { \n\
args[i] = arguments[i]; \n\
} \n\
args[i] = nodeback; \n\
ret = callback.apply(receiver, args); \n\
break; \n\
";
return ret;
}
Deep stuff! First generateArgumentSwitchCase gets an approximation of the number of arguments to the function using parameterCount, which just checks the length property of the function. The length property is the number of arguments that appear in the function’s declaration. For instance, fs.readFile‘s signature is fs.readFile = function(path, options, callback_), so fs.readFile.length === 3. This is a good estimator of the actual number of arguments a function takes, but it’s not perfect. For instance, fs.readFile‘s second argument, options, is optional, so fs.readFile('some_file', some_callback) is a valid invocation despite having just 2 arguments.
Using newParameterCount, we calculate argumentOrder, which is an array of numbers of arguments in order of likelihood. For fs.readFile, argumentOrder is [2, 1, 0, 3], since PARAM_COUNTS_TO_TRY is 3 and arguments.length - 1 is 2. The goal of argumentOrder is to minimize the number of cases we have to check: if there are 2 arguments, we only have to check the first case. generateArgumentSwitchCase then iterates over argumentOrder, adding an invocation of callback.call for each element of argumentOrder. Here’s the final generated code for the non-default portion of the switch statement:
switch(len) {
case 2:ret = callback.call(this, _arg0, _arg1, nodeback); break;
case 1:ret = callback.call(this, _arg0, nodeback); break;
case 0:ret = callback.call(this, nodeback); break;
case 3:ret = callback.call(this, _arg0, _arg1, _arg2, nodeback); break;
Finally, there’s the default case, which is where we admit defeat and fall through to the apply method as in makeNodePromisifiedClosure. When our original script calls promisifiedRead on '/etc/profile' (no options), that hits the case 1 and we get to use the inlined callback.call version. Nice!
With that, we’ve completed the picture of how Bluebird promisify works. I hope you learned something! I know I did.