Solving the upload progress bar problem–The History of Node.js

If you would have asked me how Node.js came to be I probably would have guessed something along the lines of a Stanford research lab. That is, until I stumbled across this youtube video ‘Ryan Dahl – History of Node.js’ from a Pheonix user group back in Oct 11. I intended to watch just a few minutes, but it was really interesting and I ended up watching the whole thing. If you are into Node I recommend you check it out too.

A few points from the video:

  • Ryan was working on a math PhD (Algebraic Topology), dropped out and moved to South America with no money
  • Ryan thinks Ruby has the most beautiful syntax, but all the assumptions about it are based on lies (Ruby is f*cked because it can only run on one thread)
  • Node was heavily influenced by Zed Shaw’s Mongrel. The idea that a webserver can be just a simple lib which does request/response (not necessarily a directory of files tied to file system)
  • Node was started trying to find the best way of notifying a user, real time, about the status of file upload over the web (who would have thought?)
  • The best thing about javascript on the server is that no one had used it. No way way to open a file, open a socket etc. This is good, because if there was, they would have done it wrong
  • Ryan quit his job and worked on node for 6 mos straight convinced it was ‘a thing’. Begged for a slot at JSConf in Berlin. Blew people away with an IRC chat server written in 400 lines of JS (had to demo on a different laptop and no idea if the demo would even work)
  • Some new up-and-coming node features like Node domains
  • Ryan and his team makes finger gestures and sound effects when talking about things like latency and load balancing Winking smile
    Ryan tells a great story, and as with any great story comes great lessons. The ones that really struck home with me:
  1. Sometimes you have to quit good things in order to achieve great things
  2. The obvious solution to your problem might be right in front of your face

    “Don’t be a meme, but a movement”

    Scott Hanselman wrote blog post earlier this year titled “Your Blog is The Engine of Community”, in which he encourages us to blog more, and tweet less.

    I hear you Scott.

    Social Media is junk food, and my brain is overweight! I used to have a healthy technical blog and really enjoyed contributing to the community through writing. That was over 21 mos ago. Since then I have become a complacent retweeter (with occasional joke). Looking around at some of my favorite blogs, I can see many of you have fallen into the same trap (you know who you are).

    Hey we all need a break, and that’s cool, but it’s time to start blogging again. Now don’t’ run out and write one of those “My blog has been really quiet lately so this is the first post in an amazing series I am going to do” posts, because we all know how that one turns out. Instead, I just make an honest effort by writing one post, and have a good time doing it.

    Scott shares some good pointers-

    “Blog your opinions. Blog your cool project, or your latest useful function or library. Don’t blog if it feels like work. Blog and get excited when someone comments. Often the comments are more fun and more useful than the post itself. Be passionate, but not rude. Point out failings, but suggest solutions. Organize. Invent.

    I have also started a personal blog over at jarodferguson.com where I am sharing my not-so-technical stuff. I have been meaning to do this for years. You know what? So far I really like it! It feels great because its MY content, MY brand. Twitter, Facebook, Path, Pinterest etc etc can take a back seat. Sure I will still enjoy them, but going forward I plan on doing a lot less ‘managing’ content across my networks. Blog first, social media site of the day second (besides Posterous makes it pretty easy to publish to all these other sites).

    So how about you? Why not brush off that keyboard and share with us? You never know who might really enjoy your thoughts.

    SqlBulkCopy for Generic List<T> (useful for Entity Framework & NHibernate)

    A common complaint of the Entity Framework is slow insert times for larger datasets. Last night I was trying to insert a catalog of 15k products and it was taking a very long time (I gave up after 5 minutes). I recalled this post a while back from Mikael Eliasson demonstrating SqlBulkCopy using .NET. I had used BCP in SQL server, but not from .NET. I took Mikael’s example and roughed out a reusable generic version below, which produced 15k inserts in 2.4s or +- 6200 rows per second. I upped it to 4 catalogs, 224392 rows in 39s, for +- 5750 rps (changing between 4 files). These are pretty decent records too, 41 columns and a few of the fields have a meaty char count. Good enough I say.

    public static void BulkInsert<T>(string connection, string tableName, IList<T> list)
    {
         using (var bulkCopy = new SqlBulkCopy(connection))
         {
              bulkCopy.BatchSize = list.Count;
              bulkCopy.DestinationTableName = tableName;
    
              var table = new DataTable();
              var props = TypeDescriptor.GetProperties(typeof(T))
                                         //Dirty hack to make sure we only have system data types 
                                         //i.e. filter out the relationships/collections
                                         .Cast<PropertyDescriptor>()
                                         .Where(propertyInfo => propertyInfo.PropertyType.Namespace.Equals("System"))
                                         .ToArray();
                  
              foreach (var propertyInfo in props)
              {             
                   bulkCopy.ColumnMappings.Add(propertyInfo.Name, propertyInfo.Name);
                   table.Columns.Add(propertyInfo.Name, Nullable.GetUnderlyingType(propertyInfo.PropertyType) ?? propertyInfo.PropertyType);
              }
    
              var values = new object[props.Length];
              foreach (var item in list)
              {
                   for (var i = 0; i < values.Length; i++)
                   {
                        values[i] = props[i].GetValue(item);
                   }
    
                   table.Rows.Add(values);
              }
    
              bulkCopy.WriteToServer(table);
         }
    }

    This works off just a basic list of items which property names match the table column names. Given that most POCO based ORM’s generally match the schema exactly, it works great with EF code first objects.

    To use I just build up a list of objects, pick the connection string off the DbContext and then call BulkInsert to save to the DB. Note that in this case I am just adding items to a List<T>, not the EF DbSet<T>.

    var imports = new List<Product>();
    //Load up the imports
    
    //Pass in cnx, tablename, and list of imports
    BulkInsert(context.Database.Connection.ConnectionString, "Products", imports);

    Resources for getting started with Backbone.js

    The current product I am building makes heavy use of HTML5 & javascript to give the user a rich experience, both on and offline, across a variety of screens. Before I even started this application I knew I wanted a clean and proven approach to help manage the complexity on the client. After reviewing several js frameworks I decided to go with backbone.js (compare yourself). I have been working with backbone for several months and I am really happy with it, and would recommend to anyone building similar types of javascript applications.

    As per the project description:

    Backbone supplies structure to JavaScript-heavy applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing application over a RESTful JSON interface.”

    There are a ton of good ‘getting started’ posts out there, so rather than duplicate those I thought I would put together a few of my favorite resources. Hopefully some of you might find them useful.

    • Backbone.js : The main github site- Here you will find valuable reference documentation as well a simple examples to introduce you to backbone. Be prepared to spend a little time here 😉
    • Underscore.js : “It’s the tie to go along with jQuery‘s tux, and Backbone.js‘s suspenders.”  If you haven’t heard about underscore do yourself a favor on go check it out. There is nothing magical about it, yet I can’t say enough. Even if you aren’t using backbone, I highly recommend this library (some folks are using it on the server with node.js as well).
    • Todo.js : A simple Todo list example using backbone.js
    • TodoMVC : Compare a truck load of JS MVC frameworks – “a project which offers the same Todo application implemented using MVC concepts in most of the popular JavaScript MVC frameworks of today.”
    • Backbone.js Fundamentals : A book on Backbone.js targeted at beginners and advanced users alike – by Addy Osmani (and contributors). This guide is quickly becoming the ‘mini-bible’ for backbone. Chalked full of nuggets, from beginning concepts to the more advanced.
    • The Backbone Boilerplate : “Backbone Boilerplate is a set of best practices and utilities for building Backbone.js applications”
    • Organizing Your Backbone.js Application With Modules : Some guidance around how you might structure your application using modules (esp valuable for larger applications).
    • Organizing your application using Modules (require.js) : Similar to the previous article, but makes use of AMD & Require.js to wire up the application.
    • Derick Baileys blog on Lostechies : Derick has been blogging on backbone for 6 mos or so. I appreciate reading Dericks blog because he has done such a good job of cataloging his experiences as he has learned the framework. When I found myself stumbling with backbone, my searches often led over to Dericks blog.
    • Backbone.js Screencast – Introduction and Views : A nice “basic introduction on how to bootstrap a new Backbone.js application and go in-depth on how to use Backbone Views in particular”. Joey now also has screencasts available over at his site http://backbonescreencasts.com/
    • Backbone.js walkthrough of Models and Views (Part 1/2) Screencast : Enough said!
    • Using node.js + backbone.js + socket.io for real-time goodness : I haven’t actually implemented this yet, but am excited about the possibilities. Check out here here and here for some examples. (seeing how the middle tier in this app is MVC.net, I may opt to swap socket.io for signalR. Hoping to take the node.js plunge at some point).

    Updated:

    That’s all for now. Please feel free to reach out with any other resources you found helpful and Ill add them on here. I am still very much learning and would love to read your posts. More to come-

    Social Media as a Software Development Tool?

    One thing about software development I have always struggled with is inter-team communication. As I develop, I am often thinking things like “I need to make sure I tell Elvis I moved this method” or “The team is really going to like the reduced friction of this new implementation”. I have used various methodologies to try and address this, most of which you have probably tried too:

    • Detailed check-in notes (no one reads unless they are tracking down who broke something)
    • Code-reviews (*yawn*)
    • Stand-ups (wrong medium, too high level)
    • Pair programming (exclusive to 2 or 3 developers, and is too expensive to do at a high frequency)
    • IM (Too exclusive, temporary, can be distracting)
    • Shouting “Hey dude check this out”
      All of these can be valuable, however none of them offer the signal to noise balance I am looking for. I want a mechanism which promotes and enables a different form of communication. So what about a form of social media? Wikipedia tells us:

    “Social media is media designed to be disseminated through social interaction… Social media uses…technologies to transform and broadcast media monologues into social media dialogues. It supports the democratization of knowledge and information and transforms people from content consumers to content producers.”

    Personally I have gotten really adept at tracking and participating in twitter conversations, IM, email etc while I work. I am ready for a social client that lives in my development environment. A tool which enables me to informally talk to my team as I am writing code.

    I have a bunch of ideas around what this could look like, and I think implemented properly it would make developing software better.

    So is anyone out there doing anything like this? What sort of features would make it useful?

    Event Driven Architecture: Publishing Events using an IOC container

    In my last post I talked about advanced IOC usage and how it is possible to use an IOC container to resolve an open generic from a closed implementation. This is technically cool, but it does not explain why this is important. In this post I want to show some additional code that will demonstrate a basic concept of Event Driven Architecture: Publishing an Event.

     

    Why is Event Driven Architecture good?

    Event driven architecture is extremely extensible. In my mind, it the perfect solution to address the Open Closed Principal and enable Single Responsibility in code.

    Consider a system that processes orders. There is a method called SubmitOrder(order) on an OrderController. It just validates and saves the new order to the database today.

    In a following story, the product owner says “I need the system to send an email to the customer when the order is submitted”. No problem, inject the MailService into the OrderController, send the mail in the SubmitOrder(order) method. great.

    The next story the product owner says “I need you to deduct the ordered quantities from the OnHand inventory”. Check, so we inject in the inventory service into our OrderController, and we call it from the SubmitOrder(order) method.

    More stories…. the product owner says “We need to send a message to the warehouse fulfillment system to let them know a new order has been submitted

    Ugh, another service, another method. The tests for SubmitOrder become out of control. Even when extracting methods and encapsulating more logic into services, there is still violation of SRP & OCP. Not to mention there could be performance degradation from performing all of these tasks synchronously while the user waits.

    Its not the job of SubmitOrder to do all of this, there has to be a better way.

     

    Publishing an Event Example

    What if I “tell” the system that an event had occurred, and anyone interested could take action? (like an old school observer right?) In the following code sample I am going to inject in an ‘EventPublisher’ into my OrderController, then use that object to notify the rest of the system by publishing an event using this line:

    _eventPublisher.Publish(new OrderSubmittedEvent{OrderId = order.Id});

     
    The OrderController
    public class OrderController : Controller
    {
       private readonly IRepository<Order> _orderRepository;
       private readonly IEventPublisher _eventPublisher;
     
       public OrderController(IRepository<Order> orderRepository, 
                              IEventPublisher eventPublisher)
       {
           _orderRepository = orderRepository;
           _eventPublisher = eventPublisher;
       }
     
       public ActionResult SubmitOrder(OrderViewModel viewModel)
       {
           try
           {
               if (ModelState.IsValid)
               {
                   var order = MapOrder(viewModel);
                   _orderRepository.Add(order);
                   _orderRepository.SaveChanges();
     
                   _eventPublisher.Publish(new OrderSubmittedEvent{OrderId = order.Id});
                 
                   //Display success message
                   //ViewInfo.AddSuccessMessage(Language.SubmitOrderSuccess);
               }
           }
           catch
           {
               ModelState.AddModelError("__Form", Language.SubmitOrderError);
           }
      
           return View(viewModel);
       }
     
       //other
    }

     

    Now, using this simple Consumer (Handler) interface:

    public interface IConsumer<T>
    {
        void Handle(T eventMessage);
    }

     

    I can independently implement each of the system requirements, as needed:

    public class EmailOrderConfirmation : IConsumer<OrderSubmittedEvent>
    {
       public void Handle(OrderSubmittedEvent eventMessage)
       {
          //send email
       }
    }
     
    public class NotifyWarehouse : IConsumer<OrderSubmittedEvent>
    {
       public void Handle(OrderSubmittedEvent eventMessage)
       {
           //notify warehouse
       }
    }
     
    public class DeductOnHandInventory : IConsumer<OrderSubmittedEvent>
    {
       public void Handle(OrderSubmittedEvent eventMessage)
       {
           //deduct inventory
       }
    }

     

    One of my favorite parts about this code: the container is building up all of these event consumers, which makes it is easy to satisfy all of the dependencies using constructor injection. This makes testing each of these handlers a breeze, and everything follows the same patterns and conventions keeping the code base understandable and clean. (Services, Controllers, Consumers all work the same)

    Just plug in the dependencies in the EmailOrderConfirmation consumer ctor* like so:

    public class EmailOrderConfirmation : IConsumer<OrderSubmittedEvent>
    {
       private readonly IRepository<Order> _orderRepository;
       private readonly ISmtpService _smtpService;
       private readonly ILogger _logger;
     
       public EmailOrderConfirmation(IRepository<Order> orderRepository, 
                                     ISmtpService smtpService,
                                     ILogger logger)
       {
           _orderRepository = orderRepository;
           _smtpService = smtpService;
           _logger = logger;
       }
     
       public void Handle(OrderSubmittedEvent eventMessage)
       {
           var order = _orderRepository.Single(x => x.Id == eventMessage.OrderId);
           var message = new SmtpMessage();
           //get customer info from order  & populate message
           _smtpService.SendMessage(message);
       }
    }

    * Note that you could also inject the IEventPublisher into a Consumer and publish more events.

     

    How Does it Work?

    Under the hood the IoC container is doing most of the work. It keeps track of the event subscriptions and also provides the consumer instantiation.

    Event Subscriptions

    Here is how the event subscriptions are added to the container. In this example I use an interface so that I can enable dependency injection into components, and then mock the GetSubscriptions<T> dependency for easy testing. I also use a static method for adding subscriptions since the registration process operation usually happens at bootstrapping time, which is generally all static. The IoC.Container here is thread safe.

    public interface ISubscriptionService
    {
       IEnumerable<IConsumer<T>> GetSubscriptions<T>();
    }
     
    public class EventSubscriptions : ISubscriptionService
    {
       public static void Add<T>()
       {
           var consumerType = typeof(T);
     
           consumerType.GetInterfaces()
                       .Where(x => x.IsGenericType)
                       .Where(x => x.GetGenericTypeDefinition() == typeof(IConsumer<>))
                       .ToList()
                       .ForEach(x => IoC.Container.RegisterType(x, 
                                                                consumerType, 
                                                                consumerType.FullName));
       }
     
       public IEnumerable<IConsumer<T>> GetSubscriptions<T>()
       {
           var consumers =  IoC.Container.ResolveAll(typeof(IConsumer<T>));
           return consumers.Cast<IConsumer<T>>();
       }
    }

     

    Event Publishing

    Now that the subscriptions are setup, the publisher can read those subscriptions, get the consumers from the container, and pass the event message instance to each of them. Again I am using an Interface so I can enable dependency injection (and mocking capabilities) for the components which need to do publishing (controllers, services)

    public interface IEventPublisher
    {
       void Publish<T>(T eventMessage);
    }
     
    public class EventPublisher : IEventPublisher
    {
       private readonly ISubscriptionService _subscriptionService;
     
       public EventPublisher(ISubscriptionService subscriptionService)
       {
           _subscriptionService = subscriptionService;
       }
     
       public void Publish<T>(T eventMessage)
       {
           var subscriptions = _subscriptionService.GetSubscriptions<T>();
           subscriptions.ToList().ForEach(x => PublishToConsumer(x, eventMessage));
       }
     
       private static void PublishToConsumer<T>(IConsumer<T> x, T eventMessage)
       {
           try
           {
               x.Handle(eventMessage);
           }
           catch(Exception e)
           {
               //log and handle internally
           }
           finally
           {
               var instance = x as IDisposable;
               if (instance != null)
               {
                   instance.Dispose();
               }
           }
       }
    }

     

    Keep in mind in this example is all running on the same thread, therefore even though semantically we are handling events as if they are asynchronous, it is a blocking operation

     

    Additional Benefits of this Architecture

    – Events are in messages, which can be serialized and processed on different threads, or even different nodes in a cluster. In a web environment, this might mean offloading the workload out of your IIS processes freeing web-server threads for performance & scalability (see Service Bus section)

    – Get multicast delegate observer like functionality without having to deal with static events, event registration & deregistration (the whole += –= biz)

    – Each event handler has its own class, each having 1 responsibility

    – Very extensible, very maintainable (wait, I said that already?)

     

    This is not a Service Bus

    This code above could work great in a simple MVC.NET application for implementing clean separation and structuring your code base into events. Though it gets a lot of influence from the OSS services busses, it is greatly simplified and merely demonstrates a simple concept of publishing & consuming domain events using an IoC container.

    A service bus does quite a bit more than just publishing local messages on single thread. If you find yourself enjoying this posts and want to take it to the next level, check out MassTransit (Chris Patterson & Dru Sellers) & nServiceBus (Udi Dahan). I would also recommend the EAI patterns books by Gregor Hohpe and recognize the reference patterns. (one of these days Ill make it through it instead of using it for reference!)

    Some additional concepts and common patterns you might find:

      The Specification

      Normally I don’t make a habit of testing the container, but in this case its so close to the container I feel its appropriate vs mocking out the subscriptions. Plus since this is a blog post demonstrating advanced IOC usage, I want explicitly show all of the dependencies required at bootstrap time.

      Please note the LoggerSpy for assertions. In this spec we are really only concerned whether or not they get called. Because of the nature of eventing, using a Test Spy is really the only good way I have found to assert whether or not your handlers get invoked. (If your processing on multiple threads you will have to make sure they are all done before you check assertions, perhaps a future blog post)

      [TestFixture]
      public class When_An_Order_Is_Submitted : Specification_Context
      {
         private LoggerSpy _loggerSpy;
         private Mock<IRepository<Order>> _orderRepositoryMock;
         readonly IUnityContainer _container = IoC.Container;
       
         protected override void Context()
         {
             //setup EF repository mock
             _orderRepositoryMock = new Mock<IRepository<Order>>();
       
             //logger spy to capture handler messages
             _loggerSpy = new LoggerSpy();
       
             //register with container
             _container.RegisterInstance<ILogger>(_loggerSpy);
             _container.RegisterInstance(_orderRepositoryMock.Object);
             _container.RegisterType<IEventPublisher, EventPublisher>();
             _container.RegisterType<ISubscriptionService, EventSubscriptions>();
             _container.RegisterType<OrderController>();
       
             //add event subscriptions
             EventSubscriptions.Add<EmailOrderConfirmation>();
             EventSubscriptions.Add<NotifyWarehouse>();
             EventSubscriptions.Add<DeductOnHandInventory>();
         }
       
         protected override void Because()
         {
             _container.Resolve<OrderController>().SubmitOrder(new OrderViewModel());
         }
       
         [Test]
         public void Order_Should_Be_Persisted()
         {
             _orderRepositoryMock.Verify(x => x.Add(It.IsAny<Order>()));
             _orderRepositoryMock.Verify(x => x.SaveChanges());
         }
       
         [Test]
         public void Email_Notification_Should_Be_Sent_To_Customer()
         {
             _loggerSpy.LoggedMessages.ShouldContain(Language.OrderConfirmationEmailSuccess);
         }
       
         [Test]
         public void On_Hand_Inventory_Should_Be_Reduced()
         {
             _loggerSpy.LoggedMessages.ShouldContain(Language.DeductOnHandInventorySuccess);
         }
       
         [Test]
         public void Warehouse_Should_Be_Notified()
         {
             _loggerSpy.LoggedMessages.ShouldContain(Language.NotifyWarehouseSuccess);
         }
      }

      Advanced Unity: Connecting Implementations to Open Generic Types

      Jimmy Bogard has an excellent post called “Advanced StructureMap: connecting implementations to open generic types” which he uses the StructureMap IOC container to connect messages to handlers.

      This is something I have been using in my codebase to handle domain events, as well as a publish/subscribe mechanism for WCF message handling. I learned about this Handler<T>(or Consumer<T>) approach from the MassTransit codebase. For those that don’t know about MassTransit, it is lean service bus implementation for building loosely coupled applications using the .NET framework. I highly recommend checking it out.

      Though I am a big fan of StructureMap, I have also use the Unity dependency injection container to resolve my handlers/consumers.

       

      The Open Generic Handler

      As Jimmy described in his post, there is an generic interface describing a contract

      public interface IHandler<TEvent> 
      {
          void Handle(TEvent args);
      }

      The Implementation

      The interface is implemented specifying the type (the Domain Event) for the interface contract

      public class OrderCanceledEvent
          : IHandler<OrderCanceledMessage>
      {
          public void Handle(OrderCanceledMessage args)
          {
              // send an email or something
          }
      }

      Resolving the Handler

      When a given domain event occurs, the handler(s) is resolved for the given event, the “T”, or OrderCancelledMessage.

      var handler = container.Resolve<IHandler<OrderCanceledMessage>>();

      Configuring Unity

      There are a few ways to do this with Unity. I will show how to do it using a Unity Extension. Since this is fairly trivial, you could easily do it with a regular old Extension method, which would result in less setup code.

      Basically the extension going to do is drill into the impl type and extract the interface that matches the passed in generic, and register it in the container.

      The Unity Extension
      public class OpenGenericExtension : UnityContainerExtension, IOpenGenericExtension
      {
          protected override void Initialize()
          {
             
          }
       
          public void RegisterClosedImpl<T>(Type openGenericInterface)
          {
              var closedType = typeof(T);
       
              closedType.GetInterfaces()
                        .Where(x => x.IsGenericType)
                        .Where(x => x.GetGenericTypeDefinition() == openGenericInterface)
                        .ToList()
                        .ForEach(x => Container.RegisterType(x, closedType));
          }
      }

      public interface IOpenGenericExtension : IUnityContainerExtensionConfigurator
      {
          void RegisterClosedImpl<T>(Type openInterface);
      }

       

      The Test

      Make it go.

      [Test]
      public void should_connect_types()
      {
          var container = new UnityContainer();
          container.AddNewExtension<OpenGenericExtension>()
                   .Configure<IOpenGenericExtension>()
                   .RegisterClosedImpl<OrderCanceledEvent>(typeof(IHandler<>));
       
          var handler = container.Resolve<IHandler<OrderCanceledMessage>>();
       
          Assert.AreEqual(handler.GetType(), typeof(OrderCanceledEvent));
      }