Detecting the End of a Rainbow Inside a Writable Stream

I was implementing a custom writable stream in Node.js the other day when I ran into this issue where I wanted to know whether more data was coming or that we were actually done writing stuff. When looking at the recently revised API for stream implementers, the only thing that gets mentioned in the docs is the _write method.

After some intensive DuckDuckGoing (yes, this might become an actual word in the future), I ran into this thread on the Node.js user group and this issue on GitHub. In short, it is possible and even advisable to listen to the ‘finish’ event inside a Writable stream. Apparently, this happens all the time in the Node.js library itself. This means that by catching the ‘finish’ event, I was able to implement the flush functionality that I was looking for.

So without further ado, here’s a simple example of a custom Readable and Writable stream that are piped together. The Writable stream listen to the ‘finish’ event in order to flush stuff down the drain.

var stream = require('stream'),
    util = require('util');

//
// Reading stuff
//

var ReadingStuff = function() {
    this._data = [1, 2, 3, 4, 5];

    stream.Readable.call(this);
};

util.inherits(ReadingStuff, stream.Readable);

ReadingStuff.prototype._read = function() {
    if(0 === this._data.length) {
        this.push(null);
        return;
    }

    this.push(this._data[0].toString());
    this._data.shift();
};


//
// Writing stuff
//

var WritingStuff = function() {
    stream.Writable.call(this);

    this.on('finish', function() {
        console.log('Finished writing stuff!!');
    });
};

util.inherits(WritingStuff, stream.Writable);

WritingStuff.prototype._write = function(chunk, encoding, next) {
    console.log(chunk.toString(encoding));
    next();
};

//
// Application
//

var readingStuff = new ReadingStuff();
var writingStuff = new WritingStuff();

readingStuff.pipe(writingStuff);

 

Notice that inside the _read method of our custom Readable stream, we call this.push(null) in order signal the end of the data.

Hope this helps. 

Taking Toddler Steps with Node.js – The Towering Inferno Revisited

Soon after I started using Node.js, I ran into the phenomenon of multiple nested callbacks that create some kind of horizontal tower effect. The solution I came up with in order to improve the readability of my code was using a library called step, as described in this blog post that I wrote at that time.

Over the past years I switched over to a couple of other control flow libraries that solve the same problem as step, but  eventually I settled on using the async library.

Let’s look back at the problem I used in my original blog post:

image

Here’s the slightly refactored equivalent using async:

http.createServer(function(request, response) {
    async.waterfall([
        assembleFilePath,
        readFavoritePodcastsFromFile,
        addNewFavoritePodcastToFile
        ], 
        function(error, favoritePodcasts) {
            if(error)
                return response.end(error);

            response.writeHead(200, {
                'Content-Type': 'text/html', 
                'Content-Length': favoritePodcasts.length
            });

            response.end(favoritePodcasts); 
        }
    );
})
.listen(2000);

function assembleFilePath(callback) {
    var filePath = path.join(__dirname, 'podcasts.txt');
    callback(null, filePath);
}

function readFavoritePodcastsFromFile(podcastsFilePath, callback) {
    fileSystem.readFile(podcastsFilePath, 'utf8', function(error, data) {
        if(error)
            return callback(error);

        callback(null, podcastsFilePath, data);
    });                     
}

function addNewFavoritePodcastToFile(podcastsFilePath, favoritePodcastData, callback) {
    var favoritePodcasts = favoritePodcastData;

    if(-1 == favoritePodcasts.indexOf('Astronomy Podcast')) {
        favoritePodcasts = favoritePodcasts + '\n' + 'Astronomy Podcast';       
        fileSystem.writeFile(podcastsFilePath, favoritePodcasts, function(error) {
            if(error)
                return callback(error);

            callback(null, favoritePodcasts);
        });                     
    }
    else {
        process.nextTick(function() {
            callback(null, favoritePodcasts);
        });     
    }
}

 

Here I’ve used the waterfall method of the async library in order to pass results from one function to the next. Other functions that I often use are series and parallel. Notice that in the addNewFavoritePodcastToFile function I used process.nextTick instead of just invoking the callback. This is done in order to prevent inconsistent behavior of the function. I also wrote about this in the past.

There has been a lot of buzz lately around promises, so I decided to drink some of this kool-aid. Basically, we can achieve the same kind of solution as with the async library.

http.createServer(function(request, response) {

    assembleFilePath()
    .then(readFavoritePodcastsFromFile)
    .then(addNewFavoritePodcastToFile)
    .then(function(favoritePodcasts) {
        response.writeHead(200, {
            'Content-Type': 'text/html', 
            'Content-Length': favoritePodcasts.length
        });

        response.end(favoritePodcasts); 
    })
    .done();
})
.listen(2000);

function assembleFilePath() {
    return Q.fcall(function() {
        return path.join(__dirname, 'podcasts.txt');
    });
}

function readFavoritePodcastsFromFile(podcastsFilePath) {
    var deferred = Q.defer();

    fileSystem.readFile(podcastsFilePath, 'utf8', function(error, favoritePodcasts) {
        if(error)
            return deferred.reject(new Error(error));

        deferred.resolve({
            favoritePodcasts: favoritePodcasts,
            podcastsFilePath: podcastsFilePath
        });
    });

    return deferred.promise;
}

function addNewFavoritePodcastToFile(data) {
    var deferred = Q.defer(),
    favoritePodcasts = data.favoritePodcasts;

    if(-1 == favoritePodcasts.indexOf('Astronomy Podcast')) {
        favoritePodcasts = favoritePodcasts + '\n' + 'Astronomy Podcast';       
        fileSystem.writeFile(data.podcastsFilePath, favoritePodcasts, 
        function(error) {
            if(error)
                return deferred.reject(new Error(error));

            deferred.resolve(favoritePodcasts);
        });                     
    }
    else {
        process.nextTick(function() {
            deferred.resolve(favoritePodcasts);
        });
    }

    return deferred.promise;
}

 

I’ve used the Q library for this code sample. For an excellent introduction to promises and the Q library, check out this great article on the StrongLoop blog. I think the approach using promises looks, uhm … promising as well.

Are you, dear reader, using a control flow library, which one and why?

Until next time.

Byte Magazine Archives

Earlier this week I accidentally stumbled on archive.org where they are hosting an extensive backlog of old issues of Byte magazine. This magazine was an American microcomputer magazine, influential in the late 1970s and throughout the 1980s (and well before my time).

I’ve been skimming through a couple of these old magazines, and I got quite intrigued by some of these old topics. Probably one of the more popular issues was the one on Smalltalk, but there are others on Lisp, Pascal and several varying topics as well. These old issues have been scanned in and can be downloaded as a PDF, ePub, Mobi, etc. … .

If you’re as interested in computer history as I am, then you should definitely check these out.

Seven Languages in Seven Weeks

A while ago, I was thinking about which programming language I wanted to learn next. At first I was doubting between Scala, Ruby or perhaps Clojure. But I couldn’t really make up my mind. So I decided to pick up Seven Languages in Seven Weeks. This way I was able to take a small bite from a couple different cakes in order to decide which flavor I liked the most.

The seven languages that are discussed  in this book are Ruby, Io, Prolog, Scala, Erlang, Clojure and Haskell.

I really liked Ruby and it’s nice syntax as this programming language is not a complete stranger to me.

I had great hopes for Scala, but it kind of left me disappointed. While exploring the syntax of this academic newcomer, I had this constant nagging feeling that Scala reminded me too much of C++. Don’t get me wrong, I do like C++. But while C++ was intended to bridge the gap between procedural programming (C) and object-oriented programming, Scala does seem to bring the same kind of awkwardness as it tries to bridge the gap between object-oriented programming and functional programming. I still find it a really fascinating language to learn, but it kind of fell down in my personal ranking.

The big surprise for me was Erlang. Although it’s syntax has its quirks, this is a truly powerful and fascinating programming language that piqued my interest.

Clojure was kind of was I expected and brought back some fine memories from the small, entry-level Lisp programs I wrote back when I was in college.

While I was reading this book, I briefly jumped into some other programming languages as well. Go, D and yes, even Smalltalk are other programming languages that sparked my interest.

I very much enjoyed reading this book and had a lot of fun trying to do those small exercises at the end of each day. The only down-side now is that the list of programming languages that I want to take a closer look at basically doubled. Oh well, nothing that flipping a coin can’t handle.

I want to end this blog post with the question: What programming language are you learning at the moment?

Until next time.

Taking Baby Steps with ScriptCS

I’ve been following the ScriptCS project with great interest over the last couple of months. As you may know by now, I’ve been searching for a more lightweight .NET development experience for quite some time. The ScriptCS project is here to fulfill this desperate need. Driven by the open-source .NET community, this wonderful initiative promises to unleash C# from Visual Studio which is exactly what I’ve been looking for.

Being heavily inspired by the node.js development workflow, ScriptCS is built on top of Roslyn and makes heavy use of NuGet for installing packages and script packs. You can get up and running in no time by installing ScriptCS using Chocolatey. If you didn’t have Chocolatey installed already (like me), then it can be easily done by running the following Powershell command:

@powershell -NoProfile -ExecutionPolicy Unrestricted -Command "iex ((New-Object Net.WebClient).DownloadString(‘https://chocolatey.org/install.ps1′))" && SET PATH=%PATH%;%systemdrive%\chocolatey\bin 

Now we can install ScriptCS by simply running the following command:

cinst scriptcs

That’s all we need!

Now in order to do something useful with it, we can clone the samples repository to our dev machine and try to run the Nancy sample application. Navigate to the Nancy folder in a command prompt and run the command “scriptcs -install” to install all package dependencies defined in the packages.config file. Next run the following command to start the web application:

image

Now open up your favorite browser, navigate to http://localhost:1234 and et voilà:

image

Amazing isn’t it?

I do recognize that the project is still very much a work in progress. Also something that’s not entirely clear for me is the deployment story. The road map mentions some kind of export functionality to an .exe or a Visual Studio Project. In any case, there should be an easy way to convert the code files into binary form.

Something that I also would love to see is support for Mono. How cool would it be to develop and run C# code using ScriptCS on Mac and Linux?

The future for ScriptCS certainly looks bright and I’m very much looking forward to where this particular road is taking the .NET universe. Take a look at the wiki and start familiarize yourself with ScriptCS as it’s going to open up very interesting opportunities for future .NET development.

Until next time.

Introducing node-validation

Some time ago I was looking for a validation library/module for use in a small Express application that I was writing at the time. I couldn’t find anything that suited my taste so I decided to write one myself just for kicks. The goal was learning how to publish a module to npm and making a futile attempt to contribute something back to the vibrant Node.js community. node-validation is a minimal but slightly opinionated validation library for Node.js.

Installing node-validation can be done using the canonical package manager:

$ npm install node-validation

Validation rules must be defined in a custom validator by deriving from the base Validator.

var MyObjectValidator = function() {
    Validator.call(this);

    this.ruleFor('stringProperty').isNotEmpty();
    this.ruleFor('otherStringProperty').hasMaximumLength(10);

    this.ruleFor('numericStringProperty').isNumber()
        .withMessage('Oops, something is wrong ...');
    this.ruleFor('dateStringProperty')
        .matches(/^(19|20)\d\d[-](0[1-9]|1[012])[-](0[1-9]|[12][0-9]|3[01])$/);

    this.ruleFor('numberProperty').isInteger();
    this.ruleFor('otherNumberProperty').isMaximum(5);

    this.ruleFor('exoticProperty').is(function(value) {
        return 3 === value.propertyA + value.propertyB;
    }).withMessage('Either propertyA or propertyB has an incorrect value.');
};

util.inherits(MyObjectValidator, Validator);

 

After creating a validator object, an object that needs to be validated (the subject) can be passed to the validate method. The validate method returns an array of validation errors specifying a message and the name of the violating property.

 

//
// Validation subject
//
var subject = {
    stringProperty: '',
    otherStringProperty: 'Some string value that is too long ...',

    numericStringProperty: '65.85 invalid',
    dateStringProperty: '2013-04-30 invalid',

    numberProperty: 'Some invalid number',
    otherNumberProperty: 48,

    exoticProperty: {
        propertyA: 1,
        propertyB: 1
    }
};

//
// Now it's time to validate
//
var validator = new MyObjectValidator();
var validationErrors = validator.validate(subject);

for(var i=0; i < validationErrors.length; i++) {
    console.log('Property name: ' + validationErrors[i].propertyName 
                + ', Message: ' + validationErrors[i].message);
}

 

There you go. Head over to the GitHub repository and give it a try. I’m definitely looking forward to hear your feedback.

Basic JavaScript: Prototypical Inheritance vs. Functional Inheritance

Inheritance in JavaScript has been the topic of many discussions in the past and will continue to be the source of future debates and arguments. While we do value composition over inheritance, we don’t want to throw the baby out with the bathwater either. So, from time to time, we run into these cases where we want some notion of inheritance in JavaScript. Now what?

As with many things in JavaScript, there is not a single straight answer. We can choose between a couple of options and many different variations of these solutions. But one thing’s for sure: we can’t have it all!

In this blog post I want to discuss two different styles of inheritance that I have a hard time choosing from when programming JavaScript. And as with everything in life, both styles have their own pros and cons. 

Prototypical inheritance

In ‘classical’ programming languages, one class can directly inherit from another class. JavaScript doesn’t have this notion of classes (yet). Instead, JavaScript has prototypes which you can augment to fit your own needs. This means that having a single augmented object as the prototype for other objects, which ‘inherit’ all members of the augmented prototype object, kind of simulates a pseudo-classical inheritance pattern. Let’s talk code in order to demystify this concept.

// validator.js
var Validator = exports.Validator = function() {
    this._rules = [];
};

Validator.prototype.addRule = function(rule) {
    this._rules.push(rule)
};

Validator.prototype.validate = function(instance) {
    ...
};

// specificValidator.js
var util = require('util');

var SpecificValidator = function() {
    Validator.call(this);
};

util.inherits(SpecificValidator, Validator);

SpecificValidator.prototype.filter = function(instance) {
    ...
};

// client.js
var validator = new SpecificValidator();

// Calls function on derived object
validator.filter( { ... } );        

// Calls function on base object
validator.validate( { ... } );        

Here we have a constructor function named Validator which is the base object for other ‘derived’ objects. We augment the prototype with two functions (addRule and validate). Next we define another constructor function named SpecificValidator. We ‘derive’ this new  constructor function by calling the base constructor function and wiring the prototype by using the util.inherits() function from the Node.js core library.

We have to use the new keyword in order to instantiate a SpecificValidator object. Now we can use the functions that we added to the prototype.

Functional inheritance

This pattern is advocated by Douglas Crockford in his book JavaScript, The Good Parts. There he offers this particular style as the way to go for inheriting objects. Let’s look at an example.

// validator.js
module.exports = function() {
    var rules = [], my = {};

    my.addRule = function(rule) {
        rules.push(rule);
    };

    my.validate = function(instance) {
        ...
    };

    return my;
};

// specificValidator.js
var validator = require('...').validator;

var specificValidator = function() {
    var my = validator();

    my.filter = function(instance) {
        ...
    };
    
    return my;
};

// client.js
var validator = specificValidator();

// Calls function on derived object
validator.filter( { ... } );    

// Calls function on base object
validator.validate( { ... } );        

The base constructor function returns an object that is augmented with functions and is returned at the end. The derived constructor function simple calls the base constructor function and further augments the retrieved object before returning it to the calling code. Here we don’t have to use the new keyword to instantiate anything. Just calling the right constructor function gives us an object which we can use in our client code.

Conclusion

The most important benefit of prototypical inheritance, at least in my humble opinion, is performance. By augmenting the prototype with functions, we only create these functions once. Not matter how many times we instantiate a constructor function, the same functions get (re)used every single time. Functional inheritance on the other hand creates new functions every time a constructor function is called, which is several orders of magnitude slower compared to the prototypical inheritance pattern.

On the other hand, the prototypical approach doesn’t come with encapsulation. Looking at the example shown earlier, the ‘_rules’ property is publicly available to the client code and can be manipulated at will. By using a simple convention, like prefixing with an underscore, we can indicate that these private members should not be touched in order to guarantee a correct behavior. But again, nothing can be enforced. Using functional constructors, we can have private variables and functions that cannot be manipulated by the calling code.    

There are more pros and cons, but for me, these are the most important ones to be aware of. You can see that both styles have their strengths and weaknesses. I usually tend to go with prototypical inheritance as this is the ‘JavaScript way’, but I like using the functional approach as well for those cases were I know in advance that not too many objects are created or when I don’t care about performance.

I would love to hear other takes on this. What particular styles do you use? When do you use them and why?

Until next time

Confessions of a Sublime Text-aholic

It’s true. I’m a Sublime Text addict. It’s by far my favorite development tool. End of story!

Just to illustrate, earlier this week, a member of our development team asked how to quickly remove all empty lines from a very large text file. I quickly came up with the following:

  1. Press CTRL-F.
  2. Enable regular expressions (the button entirely in the bottom-left corner).
  3. Search for ^\s*$
  4. Press ALT-ENTER (click on the “Find all” button).
  5. Hit the backspace button.
  6. Done!
  7. Be merry …
    Don’t just take my word for it. Just start using it!