How to build Large Web Applications, part I

I've been busy the last two years building fairly intricate web apps for my customers and while it's a fun job it's also by necessity not public in the same way an open-source project is. The only comments I get and discussion I'm involved in are generally from my customers, so it's a bit of a closed loop.

When I met Paul Irish in May this year in San Francisco he asked me,' But what do you actually do?', which was a fair question but took me by surprise. I'm mostly known for .. being that Swedish guy. With some luck people know I prefer Dojo as well, but I guess that's about it.

So what do I do? I build web apps as an independent consultant. Sure, I arrange conferences and user group meetings too, but that's where my beer comes from, so to speak.

This series of posts is both written to share and discuss what I feel are best practices around building large web apps, from a developer perspective, but also to let people know what I do. Possibly with the secretly evil intent of getting new customers if anything I write seems to be useful :)

Let's begin with defining what a large web app is.

A large web app (according to me, but please challenge and/or add to the list of items);

1. Have composite views (showing both a static status area and a changing subview of items and lists, for example)
2. Have widgets that are variants of each other.
3. Have widgets that need to show up in dynamic configurations (e.g. a list of n, or each a always have a b except on Saturdays, ..)
4. Initializes communication with the server or service and caches some or all information.
5. Have widgets that need to change states, remember states and react to state changes in other widgets.

This list could go on and on, probably, but these are the most important things off the top of my head.
One of my customers had an existing, successful product that was built in .Net C#. It was a client-server system where the client had the following features;

• Personal view of things going on, of various sorts.
• List of personal things, arranged per time period in a tree structure, which showed a grid with the selected things.
• Library of things of other people, arranged in a tree by both time period, user names and project names.
• *very* complex search view where list of search queries could be build up dynamically and combined like date, project names but also other domain-specific (and visual) properties.
• A thing editor which let the user compose a thing that was composed of various parts which could be dragged from a tab to the left and then changed according to their type and rearranged. There were parts like annotatable image, rich text, project reference, and so on.
• Some of the parts needed access to the system clipboard for reading and writing, and some parts needed to start with and communicate with Excel. Several parts needed file access.
• Lots of other, minor stuff.

And now, two years later, the Dojo/HTML5 version of the client have about 80% of the functionality of the old client (and some that it didn't) being sold to clients and everything. It's been quite a ride!

I realize that I both want to cover the story of the programs creation and 'how to make large web apps' at the same time. It probably is a bad idea, but let's anyway.

First of all, before digging into specific implementation I would like to talk about bare necessities. Stuff you just can't do without, stuff that if you are indeed left without it you will have to reinvent, poorly, and too late and botch things up completely. Yes, I have, and probably you too. But I didn't on this project(s) :)

Anyhoo, If we take the first list of things that define a large web app, the items leads to certain demands on your tooling and libraries of choice. First of all you need to have a clean widget abstraction to work with, one that clearly modularize your code and your markup. It also help tremendously if you have a widget lifecycle metaphor, which is handled for you and that you can plug in to when and if you need to.


Widget Abstraction



The widget abstraction should, at minimum do the following (lots of lists, sorry);

1. Separate code and markup, so you have a separate file or string for the markup of the widget
2. Make it possible to create widgets in classes that derive from each other, so you can have one grid baseclass, then a librarygrid that inherits from the basegrid, et.c.
3. Completely hide id allocation to elements and sub-widgets. By this I mean that the abstraction need to let you use symbolic names inside the markup (
   ) that are then used inside the widget class in lieu of id lookups.
4. Use markup replacement, so that parts of the markup is replaced with data in the widget class (
   ${foo}
   will be replaced with this.foo in the widget before the widget is placed in the DOM)
5. Have a lifecycle so that when you create a widget, the widget "subsystem" / superclasses will call a number of functions on it that corresponds to the different event like a) init, b) before markup creation, c) after markup creation, d) before widget inserted into the DOM, et.c.

Dojo does all of this (as does several if it's peers (JavascriptMVC, Sproutcore, ExtJS, et.c.), but since the 'everything in a  box' approach is a conceptual hard-sell, people generally want to use smaller and separate libraries that does just one thing, but well.

A good, but a little bit dated list of a couple of markup libraries can be found here. But markup libraries (like Mustache.js) only solves one of the problems.



Dojo

I'm going to cover a bit how Dojo works here, but please let me know in the comments if you know about stand-alone libraries for these functions. i would think that some are probably part of a larger library's class-system, but still.

When I create a widget using Dojo, I create two files. One is the JavaScript class (obv) whic inherits from another custom widget or directly from dijit._Widget (widget subsystem) and dijit._Templated (markup replacement and symbolic element/subwidget naming). It could look like this;

In the directory "my/custom" the file "widget.js";

dojo.provide("my.custom.widget");
dojo.require("dijit._Widget");
dojo.require("dijit._Templated");
dojo.declare(""my.custom.widget", [dijit._Widget, dijit._Templated],
{
    templatePath: dojo.cache("my.custom", "widget.html"),
    username: "nisse",
    postCreate: function()    

  {       

    dojo.connect(this.mybutton, this, function(e)

    {

                       console.log("Someone pressed me button, eh?");        
              }    
            }
           });


The file "widget.html";



< div>

     < button dojoAttachPoint="mybutton" >${username}< button >

<div / >

The 'postCreate' function on a widget is called (if present) just before the widget is placed in the DOM, but after all markup replacement have been done. We now have markup that contains a div with a button in it, and the content of the button reads 'nisse', since that was what the class variable 'username' contained.

To use the widget you just have the following things in you code;

dojo.require("my.custom.widget");
...
var foo = new my.custom.widget();
// Not mentioning argument passing or automatic placement in page stuff. it's just a widget in the wind right now

There a re lots of other things you can do, like automatic event wiring, but I hope this small example is just enough to make you think 'Hmm.. that could actually be kind of useful'.

I am not really touching on the namespacing either, which is a simple things that have had ridiculous positive effects on the development speed and stability. We'll get to that later.

What the widget will look like in the actual page is something like this;

< div>
    <button id="mycustomwidget_0_mybutton">nisse</ button >

<div/ > 

What's actually in the id is not important, the important thing is that the id is never visible to you (unless you really want to, but you don't. Why would you? The only thing you need is to get a reference to the node in question (in this case, to attach an event handler to it).

Also, what may be less apparent, is that when you customer want another widget just like it beside the first, you don't have to worry about widget id lookup code, you know that you abstraction will handle those details and just create a new one and smack it up - no problem.

There are tons of more things I'd like to fit in here, but I have to clean the house while parts of the family is playing Deus Ex (the original) and others are off galivanting with horses.  Oh well..

Oh, right. I'll be doing a series of free, half-day Dojo-training events at Valtech in Sweden beginning the 13/10 2011. I'll post here (and on twitter etc) if you live nearby and want to learn the basics of creating custom Dojo widgets.

See you soon!