Saturday, September 24, 2011

How to build Large Web Applications, part I

I've been busy the last two years building fairly intricate web apps for my customers and while it's a fun job it's also by necessity not public in the same way an open-source project is. The only comments I get and discussion I'm involved in are generally from my customers, so it's a bit of a closed loop.

When I met Paul Irish in May this year in San Francisco he asked me,' But what do you actually do?', which was a fair question but took me by surprise. I'm mostly known for .. being that Swedish guy. With some luck people know I prefer Dojo as well, but I guess that's about it.

So what do I do? I build web apps as an independent consultant. Sure, I arrange conferences and user group meetings too, but that's where my beer comes from, so to speak.

This series of posts is both written to share and discuss what I feel are best practices around building large web apps, from a developer perspective, but also to let people know what I do. Possibly with the secretly evil intent of getting new customers if anything I write seems to be useful :)

Let's begin with defining what a large web app is.

A large web app (according to me, but please challenge and/or add to the list of items);

  1. Have composite views (showing both a static status area and a changing subview of items and lists, for example)
  2. Have widgets that are variants of each other.
  3. Have widgets that need to show up in dynamic configurations (e.g. a list of n, or each a always have a b except on Saturdays, ..)
  4. Initializes communication with the server or service and caches some or all information.
  5. Have widgets that need to change states, remember states and react to state changes in other widgets.

This list could go on and on, probably, but these are the most important things off the top of my head.
One of my customers had an existing, successful product that was built in .Net C#. It was a client-server system where the client had the following features;

  • Personal view of things going on, of various sorts.
  • List of personal things, arranged per time period in a tree structure, which showed a grid with the selected things.
  • Library of things of other people, arranged in a tree by both time period, user names and project names.
  • *very* complex search view where list of search queries could be build up dynamically and combined like date, project names but also other domain-specific (and visual) properties.
  • A thing editor which let the user compose a thing that was composed of various parts which could be dragged from a tab to the left and then changed according to their type and rearranged. There were parts like annotatable image, rich text, project reference, and so on.
  • Some of the parts needed access to the system clipboard for reading and writing, and some parts needed to start with and communicate with Excel. Several parts needed file access.
  • Lots of other, minor stuff.

And now, two years later, the Dojo/HTML5 version of the client have about 80% of the functionality of the old client (and some that it didn't) being sold to clients and everything. It's been quite a ride!

I realize that I both want to cover the story of the programs creation and 'how to make large web apps' at the same time. It probably is a bad idea, but let's anyway.

First of all, before digging into specific implementation I would like to talk about bare necessities. Stuff you just can't do without, stuff that if you are indeed left without it you will have to reinvent, poorly, and too late and botch things up completely. Yes, I have, and probably you too. But I didn't on this project(s) :)

Anyhoo, If we take the first list of things that define a large web app, the items leads to certain demands on your tooling and libraries of choice. First of all you need to have a clean widget abstraction to work with, one that clearly modularize your code and your markup. It also help tremendously if you have a widget lifecycle metaphor, which is handled for you and that you can plug in to when and if you need to.

Widget Abstraction

The widget abstraction should, at minimum do the following (lots of lists, sorry);

  1. Separate code and markup, so you have a separate file or string for the markup of the widget
  2. Make it possible to create widgets in classes that derive from each other, so you can have one grid baseclass, then a librarygrid that inherits from the basegrid, et.c.
  3. Completely hide id allocation to elements and sub-widgets. By this I mean that the abstraction need to let you use symbolic names inside the markup (
    ) that are then used inside the widget class in lieu of id lookups.
  4. Use markup replacement, so that parts of the markup is replaced with data in the widget class (
    will be replaced with in the widget before the widget is placed in the DOM)
  5. Have a lifecycle so that when you create a widget, the widget "subsystem" / superclasses will call a number of functions on it that corresponds to the different event like a) init, b) before markup creation, c) after markup creation, d) before widget inserted into the DOM, et.c.

Dojo does all of this (as does several if it's peers (JavascriptMVC, Sproutcore, ExtJS, et.c.), but since the 'everything in a  box' approach is a conceptual hard-sell, people generally want to use smaller and separate libraries that does just one thing, but well.

A good, but a little bit dated list of a couple of markup libraries can be found here. But markup libraries (like Mustache.js) only solves one of the problems.


I'm going to cover a bit how Dojo works here, but please let me know in the comments if you know about stand-alone libraries for these functions. i would think that some are probably part of a larger library's class-system, but still.

When I create a widget using Dojo, I create two files. One is the JavaScript class (obv) whic inherits from another custom widget or directly from dijit._Widget (widget subsystem) and dijit._Templated (markup replacement and symbolic element/subwidget naming). It could look like this;

In the directory "my/custom" the file "widget.js";

dojo.declare(""my.custom.widget", [dijit._Widget, dijit._Templated],
    templatePath: dojo.cache("my.custom", "widget.html"),
    username: "nisse",
    postCreate: function()    
    dojo.connect(this.mybutton, this, function(e)
                       console.log("Someone pressed me button, eh?");        

The file "widget.html";

< div>
     < button dojoAttachPoint="mybutton" >${username}< button >
<div / >

The 'postCreate' function on a widget is called (if present) just before the widget is placed in the DOM, but after all markup replacement have been done. We now have markup that contains a div with a button in it, and the content of the button reads 'nisse', since that was what the class variable 'username' contained.

To use the widget you just have the following things in you code;

var foo = new my.custom.widget();
// Not mentioning argument passing or automatic placement in page stuff. it's just a widget in the wind right now

There a re lots of other things you can do, like automatic event wiring, but I hope this small example is just enough to make you think 'Hmm.. that could actually be kind of useful'.

I am not really touching on the namespacing either, which is a simple things that have had ridiculous positive effects on the development speed and stability. We'll get to that later.

What the widget will look like in the actual page is something like this;

< div>
    <button id="mycustomwidget_0_mybutton">nisse</ button >
<div/ > 

What's actually in the id is not important, the important thing is that the id is never visible to you (unless you really want to, but you don't. Why would you? The only thing you need is to get a reference to the node in question (in this case, to attach an event handler to it).

Also, what may be less apparent, is that when you customer want another widget just like it beside the first, you don't have to worry about widget id lookup code, you know that you abstraction will handle those details and just create a new one and smack it up - no problem.

There are tons of more things I'd like to fit in here, but I have to clean the house while parts of the family is playing Deus Ex (the original) and others are off galivanting with horses.  Oh well..

Oh, right. I'll be doing a series of free, half-day Dojo-training events at Valtech in Sweden beginning the 13/10 2011. I'll post here (and on twitter etc) if you live nearby and want to learn the basics of creating custom Dojo widgets.

See you soon!

Saturday, April 9, 2011

Bad, hairy, dangerous and irresponsible node+now+couch example

What I feel is lacking many times when I want to learn a new piece of technology is smallest possible complete examples. I find many time how to interact with a new fancy database, or maybe how to communicate between client and server in a simpler, cooler way.

Very seldom do I find examples that do everything I need. What I want, ideally, when toying with a new piece of tech is the following;

1. Full instructions on how to set up any requirements, or at least pointers to good sources.
2. Example of client code, or at least how to get a basic web page up through or in parallel to this new something.
3. Example of how to call the service from whatever client should be used.
4. Example of how to take care of the client requests send on the server-side
5. Example on how to persist information and get it back again, even if that is not really part of the new tech.

What I like to convey, I think,  is that there is need for examples that give a newbie a fully working service for all examples, since even a very simple assumption can be to hard to grok if you are new to a certain scene.

In general I'd think it is better to have working, fully stand-alone examples that one does not understand one bit, than to have an example that is a partial service that is easy to understand. The newbie is a coder after all, and as long he or she have a working example that can be thrown rocks at, eventually he or she will figure the stuff out.

As it happens I've just manages to create (what I hope is) such a thing. What I wanted was to use the new now.js client->server and server->client communications proxy.  It's really sweet since it makes JSON-RPC calls really simple. it's like this;

1. Your web page loads the now.js script from your now-powered node.js serverside JavaScript server.
2. The server-script on the node side adds any property to the global ' object.
3. JavaScript in your page gets any property changes propagated to a proxy object called just 'now' on the client.
4. If the properties are functions, the client can call them, and the function call will be executed on th server. No setup code, no nothing. Just call the bloody function, thanks.
5. If the client (your webpage JS code) adds a function of its own the server can call that.
6. This is what you really, really want.

If the browser doesn't support, a flash wrapper will be used, but all else remain equal.

But then again, you really would want to have some database running wouldn't you? On the server-side, I mean. Couchdb, for example.

What I just did was create a very small, not very clean example that uses now.js, couchdb and cradle on node and a small index.html file which load the now.js script and shows how to call functions exposed back and forth.

The code looks like this;

 var url = require("url");  
 var sys = require("sys");  
 var path = require("path");  
 var fs = require("fs");  
 var inspect = require('eyes').inspector({styles: {all: 'magenta'}});  
 var cradle = require("cradle");  
 cradle.setup({host: "", port: 5984});  
 var conn = new(cradle.Connection)();  
 var db = conn.database("abc");  
 var yourHttpServer = require('http').createServer(function(req, response)  
  /* Serve your static files */  
  var uri = url.parse(req.url).pathname;  
   var filename = path.join(process.cwd(), uri);  
   path.exists(filename, function(exists) {  
     if(!exists) {  
       response.writeHead(404, {"Content-Type": "text/plain"});  
       response.write("404 Not Found\n");  
     fs.readFile(filename, "binary", function(err, file) {  
       if(err) {  
         response.writeHead(500, {"Content-Type": "text/plain"});  
         response.write(err + "\n");  
       response.statusCode =200;  
       response.write(file, "binary");  
 var everyone = require("now").initialize(yourHttpServer); = "Welcome to TEH wicked nowjs JSON-RPC"; = function(arg,cb)  
     sys.puts("Foo called.. arg and is...\n");  
     db.get('clientlog', function(err, doc)  
   'clientlog', {log: []});  
             var arr = doc.log || [];  
             arr.push({date: new Date(), message: arg});  
             doc.log = arr;  
   'clientlog', doc,  
             function(err, res)  
             // Handle success  
                 sys.puts("Error after save was : "+err);  
     cb( "server callback yo!");  
 } = function(arg)  
     sys.puts("now.js function 'bar' called from client. args are..");  

What you need to do to get this working is;

0. Don't have a lossy operating system.
1. Install couchdb and create a database named 'abc'
2. git clone node.js + npm;

git clone
./configure and make install first node, then npm

3. npm install cradle (couchdb helper library)
4. npm install now (that thing I just talked about, sheesh)
5. npm install eyes (logging/inspect library)
6. Unzip the nowtest example zipfile somwhere and do 'node nowtest.js'

Ta-daa! :)

Or possibly not, since I just slammed this together. It's bound to have ugly, warty, beer-guzzling bugs. Please tell me about them so I can update the code and links. Thx!