Shindig Architecture: Java gadget Server 3 – Util

More raw Shindig notes. This time, looking at org.apache.shindig.util. See Shindigging tag. This is just a quickie for completeness sake as it’s a few generic util classes. This post completes the listing of all Java classes in the Shindig architecture at this time.

Check – Runs some standard assertions (empty null etc).

InputStreamConsumer – Input stream -> String

ResourceLoader – Loads some files within a path. Will trawl through path and also open up any JARs.

Shindig Architecture: Java Gadget Server 2 – Servlets

More raw Shindig notes. This time, looking at org.apache.shindig.gadgets.http. See Shindigging tag. I’ll structure them just a little more this time.

Main Servlet – data struct for country/language/locale – The servlet that accepts gadget spec URL and prefs, and outputs the gadget content (typically in an iframe). Delegates heavily to GadgetServer, in order to get a Gadget, and then serialises the Gadget itself with outputGadget(). outputGadget() will output the gadget as either URL or HTML type, depending on the content type. (I expect those output methods will probably be extracted to a seperate class, or to their own strategy classes.)

  1. gadget = servletState.getGadgetServer().processGadget(gadgetId,
  2.           getPrefsFromRequest(req), context.getLocale(),
  3.           RenderingContext.GADGET, options);
  4.       outputGadget(gadget, view, options, contentFilters, resp); extends ProcessingOptions – Allows URL params to override default options, e.g. to allow caller to suppress caching

Javascript Servlet – Outputs Javascript content

Proxy Servlet – Provides implementation for ProxyServlet, which is a thin wrapper around this class Handles Fetch commands for gadgets, i.e. allowing them to get remote content. Delegates everything to ProxyHandler


RpcServlet is a “meta” servlet. Initially I thought it was just for debugging/administering the container, but it plays a more important role as it lets the browser-side gadget container issue a query to find out about the gadgets it’s hosting. (The gadgets are of course in an iframe, so due to security restrictions, it can’t directly inspect the gadget content to find out, for example, its name, which it needs to know in order to show the gadget chrome/wrapper). – Context for JsonRpc stuff. Used by RpcServlet – Meta-model of a Gadget (ie just its defining features – URL and moduleId – and not fields to populate it as in Gadget). Used by RpcServlet – Used by RpcServlet – Used by RpcServlet – Used by RpcServlet – Boring exception class – Provides Gadget meta-info – allows a programmer/tester to get info about the gadget server and list its gadgets. See[email protected]/msg00317.html.

Used by All Servlets – Servlet scoped state (ie instances of the same servlet always get this state object) - – Defines accessors for globals such as the GadgetServer, so that each Gadget can get a handle on them. implements CrossServerState – creates globals such as the GadgetServer (and defines accessors for servlets to access them). Also includes some utility methods for the servlet (which could really go elsewhere).

Misc implements ContentFilter – Caja filter – adaptor/bridge to Caja project, which sanitises JS, intended for inlined gadget.

Shindig Architecture: Java Gadget Classes

This is the first of an open series on the architecture of Shindig, the new open-source gadget/widget framework project. As mentioned here earlier, this project is building something similar to iGoogle, i.e. an environment for serving gadgets, a run-time environment for the gadgets to operate in, and a gadget container (as well as OpenSocial support).

I’m currently digging into Shindig’s architecture and will document my progress.

For the record, there’s not much discussion of Shindig’s architecture to date. The most useful summaries I’ve seen are a couple of notes on the mailing list:

  •[email protected]%3E
  •[email protected]/msg00369.html

Also, be aware that Shindig has server-side implementations in both Java and PHP, and potentially more languages in the future. I’m focusing on Java at this time.

I’ll be tagging each of these articles with “shindigging” (as well as “shindig”, a general tag for anything on this blog about shindig). Thus, you’ll be able to find a full list of articles from

Java Gadget Server

I’ve walked through each file in the Java gadget server, in the main package – org.apache.shindig.gadgets and taken a very raw set of notes on each file / public class, as well as sketched a quick summary of the process. I’ll refine all this later.

Java Gadget Server – Tracing from gadget spec to page content

A gadget server takes an XML file on a server somewhere and converts it to some HTML/JS/etc content inside an iframe. After looking at org.apache.shindig.gadgets, the Java gadget server achieves this task as follows.

  • GadgetServer is invoked from the web app to render a gadget whose spec sits at a URL
  • GadgetServer uses CacheLoadTask to load the _Gadget_ object if possible
  • If not found, GadgetServer uses SpecLoadTask, which uses RemoteContentFetcher, to grab the Spec.
  • GadgetSpecParser converts the XML string into a GadgetSpec, which is a Java representation of the XML spec.
  • Gadget constructs itself from a combination of the GadgetSpec and the preferences.
  • GadgetServer passes Gadget to each required GadgetFeature (going by the required features declared in the spec). These GadgetFeature objects perform some kind of transformation on the Gadget – typically they add one of more JS libs to it (a gadget has a list of JS libs).
  • At this point, classes in the http package kick in to render the Gadget object, of which more in a different blog post.

Java Gadget Server – Files / Classes in org.apache.shindig.gadgets (raw notes) – [part of GadgetServerConfig] dumb implementation of GadgetBlacklist – file based – dumb implementation of GadgetDataCache – Just a hashmap – dumb implementation of GadgetSigner “Provides dummmy data to satisfy tests and API calls” – dumb (String) implementation of GadgetToken – server-side remoting proxy implements GadgetFeatureFactory – Bidirectional language support (i18n). Performs “hangman” substitutions (MSG_foo). Builds up a Substitutions and executes it. – It’s a gadget! This object is created from a GadgetSpec and ultimately serialised to a string representing the HTML/JS/etc content that sits on the page. Prior to serialisation, the object is subject to a set of transformations, one for each GadgetFeature it requires. interface – [part of GadgetServerConfig] persists blacklist and lets you query if a given URL is blacklist interface – String->String filter interface to transform the HTML/JS/etc widget content for the browser, e.g. for Caja sanitisation – This object is passed to each GadgetFeature in the processing sequence to tell it what’s going on and help modify its behaviour, since it contains info about gadget server options – ProcessingOptions – as well as Locale, RenderingContext and ServerConfig. – [part of GadgetServerConfig] Cache interface. Simply a map from string ID -> Type T. – Exception base class – Transforms a Gadget so it will implement a particular feature. prepare() on initial call and void process(Gadget) later on. TODO more – Simply an interface to create Gadgets “GadgetFeature create()” – [part of GadgetServerConfig] A map of gadget features in this gadget server. Essentially Gadget ID string -> {feature object, other features it depends on}

  • Includes processGadget(), which is called by gadget servlet. GadgetID [ie gadget URL] -> Gadget object ready for rendering
  • processGadget() adds a sequence of task objects (commands) and executes them:
    • CacheLoadTask – load gadget from cache instead of fetching/constructing it
    • SpecLoadTask – load gadget from remote URL (using low-level class, RemoteContentFetcher)
    • EnqueueFeaturesTask – popalate Gadget’s list of required gadget feature objects
  • Uses a workflow process: Works iteratively – each cycle, it works out which tasks need to be performed. Keeps iterating until all tasks completed or no new tasks can be added. Meanwhile, accumulates all gadget exceptions for all iterations so they can be bundled together in a big exception option that’s thrown if any exceptions occurred. [Note: I'm not sure why this complicated workflow algorithm is required, when afaict only 3 task objects are present. Maybe more will be added later on.] Configuration options for the gadget server. Composed of java.util.concurrent.Executor, FeatureRegistry, GadgetDataCache, MessageBundleCache, RemoteContentFetcher, GadgetBlacklist, SyndicatorConfig Nothing much right now. You’d think it parses a config file or something, but it just ~replicates GadgetServerConfig interface – defines interface for mapping token ID string -> GadgetToken – Dumb data structure encapsulating the spec (xml) ie user prefs, required features, gadget URI, HTML content data, random info-garbles (author etc.) – String xml -> GadgetSpec.

  1. GadgetSpecParser specParser = new GadgetSpecParser();
  2.       GadgetSpec spec = specParser.parse(gadgetId, xml.getResponseAsString());
  3.       wc.gadget = new Gadget(gadgetId, spec, prefs);
  4.       (ie xml file becomes spec, spec becomes gadget) – Effectively a token string, with a method to sign URLs interface – An immutable view of the gadget – Goes into a directory and recursively finds all files matching “feature.xml” Reads each file into a GadgetFeatureRegistry.Entry and registers it into registry (e.g. feature.containerJs.add(JsLibrary) (remember a GadgetFeature modifies the gadget in some way. In the case of a JsFeature (defined in JsLibraryFeatureFactory), the modification is simply to add some JS libraries) [jsLibraries is part of Gadget] – Represents a JS library – holds its source u.g. URL/file) and capable of reading it to get a string of the JS. The source may be a string representing the JS itself, which is useful if the client simply wants to construct the script text programatically. implements GadgetFeatureFactory – Provides GadgetFeatures in the case where the gadget feature is simply a JS file (or a list of container JS files and a list of gadget JS files). In this case, the feature’s process() method is simply to add all the libraries to the gadget (gadget.addJsLibrary). JsFeatureLoader uses this after trawling through to find the feature.xml for each gadget, since that file simply identifies a bunch of JS libraries. [part of GadgetServerConfig] String ID -> Message map. XML file -> MessageBundle implements GadgetFeatureFactory – Provides MessageBundleSubstituterFeature. This feature is a Javascript library that “compiles” the MessageBundle to Javascript, for a particular locale. It sets up language and country preference (String setLangFmt = "gadgets.prefs_.setLanguage(%d, "%s");"; String setCountryFmt = "gadgets.prefs_.setCountry(%d, "%s");";), and then sets up, for each message, the JS mapping from ID -> Message ( String setMsgFmt = "gadgets.prefs_.setMsg(%d, %s);" ); – Includes ModuleSubstituterFeature which simply replaces MODULE hangman string with the module ID. – Provides OpenSocialFeature – Tweaks GadgetServer.processGadget algorithm (methinks this seems like a weird pattern – should instead be attributes of GadgetServer). – Encapsulates results of HTTP call – the content as well as status code, size, etc. [part of GadgetServerConfig] – HTTP client to grab gadget spec (nb IMO too much BDUF abstraction going on here) – Encapsulates request for HTTP call – headers etc. – enum { GADGET | CONTAINER } – boring exception class [part of Gadget] – A collection of Substitutions – each Gadget has a Substitutions object, which it uses for get() queries, e.g. “public String getTitle() { return substitutions.substitute(baseSpec.getTitle()); }”. * Several substitution types MSG BIDI UP(user-prefs) MODULE * A map for each substitution type, mapping substitution key -> substitution string * Runs the sequence of substitutions on a given string [part of GadgetServerConfig] Unclear – related to OpenSocial and JSON. [part of Gadget] – Builds up JSON object with preference values, using Substitutions to perform any substitutions (???) – preference ID -> string (value of preference)

Updated Tags

I’ve updated the tags on this blog.

Since July, 2005, I’ve been using SimpleTags by Burble, which was just a simple <tags> tag embedded in the content. Having upgraded to WP 2.3+, I can use WordPress’s built-in tags and the ecosystem around them. This mean I can use the Simple Tags plugin by Amaury Balmer (same name but no relation). This is good as I can now use tags as they’re intended, i.e. as categories, so if I blog about “xyz”, you can click on the “xyz” tag and see all posts about “xyz”. There’s also a tag cloud avaiable in the sidebar. Good.

Dual-Side Templating

Ajax, Ajax Patterns, Javascript, Server-Side Javascript

As server-side Javascript continues to gather momentum, patterns will start to emerge. Dual-side templating, which I’ll explain below, is a pattern I’ve been harping on about for a while because you can kinda sorta use it already with a product like Rails. It will be a lot more powerful with OFL (our favourite language) on both sides of the wire.

The timeline looks like this (with milestone times neatly accelerating towards the singularity :):

  • c. 1995: Server-Side Templating. This is the standard templating used in Java’s JSP, Perl’s Mason, PHP, ASP, etc. ie some html code with <?= “language” ?> code embedded in it.
  • c. 2005: Browser-Side Templating. This is an Ajax pattern where you have a block of HTML that includes some custom syntax (e.g. <% ${} %>) which are then processed via Javascript.
  • c. 2010: Dual-Side Templating A single template is used on both browser and server, to render content wherever it’s appropriate – typically the server as the page loads and the browser as the app progresses. For example, blog comments. You output all existing comments from the server, using your server-side template. Then, when the user makes a new comment, you render a preview of it – and the final version – using browser-side templating.

I continue to be bullish on server-side Javascript and am expecting a lot of design patterns to emerge in the next couple years. AppJet and Jaxer are already available, but the real impact will be (a) enterprise-friendly stack, probably Java-based; (b) commodity hosting stack, probably Jaxer based.

SSH Tip: Remember Last Path

ssh, Unix

Lately, I’ve been living dangerously and doing a lot of coding on the live server, via ssh. This is all proof-of-concept stuff at Ajaxify. The reason I’m coding live is that it’s all widget development and it’s just easier that way, since environments like iGoogle and Facebook need to reach my apps live, in order to test the app. None of the alternatives take my fancy: open up a port into my development PC (difficult due to dynamic DNS and less secure); upload code each change (slow even if it’s fully scripted); use Shindig on my local machine (unreliable as Shindig is new and in rapid growth mode).

So I’ve been ssh’ing and no matter how many times I’ve tried, I can’t remove ssh timeout, so I sometimes get timed out. So anyway, I wanted an easy way to remember the most recent directory I was in, each time I log back in. This is it – in my .bash_profile:

function cd { builtin cd $1 ; pwd > $HOME/.path_history; }
cd cat $HOME/.path_history

Coding Standards Suck

Coding Standards, Software

… because they are dogmatic. Coding guidelines are just fine, but coding standards imply:

  • There will be an automated tool that checks your conformity to guidelines, leading to (a) inability to check in your code; (b) public shame; (c) much head-banging on keyboards
  • A smug sense of unjustified satisfaction from the kinds of people who add little value to the development process
  • Developers are thick-heads

As a devotee of literate and self-documenting software, I frequently find cases which break my own default preference. A simple example is a data structure where geometrically splendid code layout will aid comprehension, e.g.:

  var fibs    = [ 0, 1, 1, 2,  3,  5,  8, 13, 21];
  var squares = [ 0, 1, 4, 9, 25, 36, 49, 64, 81];
This is of course laid out so you can easily compare the value at each position in the array. But under some dogmatic coding standards, you are incited to make the code considerably less comprehensible:
  var fibs = [0,1,1,2,3,5,8,13,21];
  var squares=[0,1,4,9,25,36,49,64,81]
That’s a simple data structure example, but the same thing can happen with control flows as well, where you might want to sometimes use whitespace, sometimes use a curly brace at a different place, etc.

Another example – which I’ve mentioned for years on this blog – is the craziness of setter and getter dogmatism. Instead of:

  public void setName(name) { = name; }
  public void getName() { return name; }
  public ...

… you end up with:

  /* Sets the name. Duh. */
  public void setName(name) { = name;

/* Ehm. Well I suppose it gets the ^$()* name .Bozo!*/ public void getName() { return name; }

public ...

… How’s that for a five-fold increase in your LOC, if that’s what you’re getting paid for!

Competent programmers can make these decisions and decide when it’s reasonable to deviate from coding guidelines, and if you’re relying on an automated system to check code, maybe it’s time to start running code reviews or pair programming. The problem is, they’re not allowed to by heavy-handed Coding Standards.

In Extreme Programming, coding standards are defended on the basis of collective ownership; the argument is that each team member has their own taste, so code would forever be thrashed back and forth as each programmer rotates on to it. Once again, though, this is fine as long as coding guidelines are in place. The example above is something all programmers can agree on, but would still fail some automated systems, or fail to be generated correctly by a tool like Eclipse. There will still be arguments, but probably not many if the team is thinking alike, and those will be arguments worth having.

There’s also a view that coding standards support faster development, by letting you focus more on cranking code out and not on hand-crafting each line of code. Well, I’d rather hand-craft each line of code and make it as valuable as possible, refactoring and refactoring it; IMO a good team and environment can produce code that’s one-tenth the size of equivalent code created by a mediocre team. That’s ten times less code to maintain and ten times less code to confuse. To wit, 37Signals took TadaList – a reasonably successful app – to production with just 579 lines of code. With tight code like that, you ought to treat it more like prose and less like a can of dog food on an assembly line.

BlingText and Banner

Ajax, AjaxPatterns

As foretweeted last week, I created a little Ajax app called BlingText.

As you can see, it takes a message and provides some ASCII renderings. In particular, it includes a port of the old UNIX/C Banner utility.

If I do more work on it, the main improvements will be:

  • Options. Let the user specify, for each transformation, parameters such as the fill character (“*”) and amount of spacing.
  • Better OO (internal change). Each of the transformations is at present a terse strategy object, which is good. However, there’s no inheritance going on, so it could be better.

All my p0dc@st are Belong to Me!!!

Hosting, Podcast

In the past day, I’ve pulled down all my podcast mp3 files from libsyn and updated all the posts to point to my domain instead of libsyn. Hence the major feed refresh on the blog and podcast, sorry if that hit you subscribers. (Is anyone still subscribed to the podcast?)

This is a big relief.

Having the podcasts hosted on my own domain gives me slightly more incentive to start podcasting again. Also, having PodPress installed has been a great help too. My next priority, though, is to upgrade WordPress, so I can get some modern features like auto-backup.

Libsyn has been a fine service, but really my Dreamhost account is all I really need now – given how much storage and bandwidth modern hosting accounts provide – and it’s nice to have direct file system access to all the podcasts too.

Widget/Gadget Containers: What are they good for?

Ajax, AjaxPatterns, Gadgets, OpenSocial, Shindig, Web, Web 2.0, Widgets


Widgets are small “mini websites”, typically self-contained blocks of content, on a larger web page (with Ajax Design Patterns, I referred to them by the nom du jour Portlets). They are used in a couple of ways:

  • Embedded in a normal web page. For example, my blog currently contains a BBC weather widget and a “Twitter Badge” showing my latest tweets. Widgets embedded in this way are combined by the publisher, often with some manual HTML coding (script tags), and are usually a sideshow affair.
  • Combined within a widget container (aka “widget portal”, “Ajax homepage”). Websites such as iGoogle and NetVibes are primarily designed as widget aggregators, allowing an end-user to construct a personalised page for themselves. They are the Ajax/Web 2.0 successors of the “My <whatever>” hype (My Yahoo!, My Excite!, etc.) of the mid-to-late ’90s. To some extent, a social networks like Facebook fits into this category too, with the proliferation of applications available to users to embed on their homepages. And as Facebook in particular illustrates, widget containers are not always private “productivity tools”, but may also be available to a user’s friends or the general public. Indeed, the more conventional widget containers have recently started allowing users to make their public portals, which starts to move them in the territories of CMS and social networking.

Well, this is an article is about widget containers. Specifically, I’m currently compiling a list of typical features you’d expect to see in widget containers, so if this post sounds stream-of-consciousness, well, it sort of is.

Anatomy of a Widget (well, Gadget and OpenSocial) Container

A good place to start is the breakdown of functionality for Shindig, the new Google-supported Apache project to build a reference implementation for the OpenSocial standard. OpenSocial is heavily intertwined with the whole idea of widgets and widget containers, since it’s basically Google Gadgets + standardised social networking APIs. Hence, Shindig is essentially the high-profile open-source project involving a widget container. Shindig has been broken into four parts:

  • Gadget Container JavaScript — core JavaScript foundation for general gadget functionality (read more about gadget functionality). This JavaScript manages security, communication, UI layout, and feature extensions, such as the OpenSocial API.
  • Gadget Server — an open source version of, which is used to render the gadget xml into JavaScript and HTML for the container to expose via the container JavaScript.
  • OpenSocial Container JavaScript — JavaScript environment that sits on top of the Gadget Container JS and provides OpenSocial specific functionality (profiles, friends, activities).
  • OpenSocial Gateway Server — an open source implementation of the server interface to container-specific information, including the OpenSocial REST APIs, with clear extension points so others can connect it to their own backends.

That’s a very useful overview, though I’m looking more at specific features which generally cut across at least some of these parts. For example, gadget preferences. These are part of the container Javascript because there is a UI to change the preferences, they are part of the gadget server because it must initialise the gadget according to preferences, and they are part of the server because they must be persisted.

Feature List

First cut at a feature list (# indicates not directly available in iGoogle)

Gadget Preferences

  • Preference defaults
  • User can set preferences
  • Publisher can set preferences when embedding widget on a page
  • Preference variable types: string, enumerable (set by dropdown, set by combobox), boolean, list, location, etc
  • Preference persistence: by database against session vs. in cookie
  • Preferences persisted for anonymous user (Lazy Registration)
  • Gadgets can access preferences via API, consistent access regardless of content type and embedding model
  • # Gadget can be notified of preference changes, so that it’s not necessary to reload the entire gadget/page after each change

Gadget Appearance

  • Gadget appearance customisable by publisher
  • Gadget appearance customisable by user
  • # Round corners, shadows, background images (e.g. Schmedley has all of these)

Gadget Content Type

See Widget Content Type article * HTML – from container provider’s domain * URL – from gadget provider’s own domain * # Inlined – embedded on container. This implies a security model, e.g. Caja

Container Appearance

  • User can change gadget skin, header, footer, background, widget preferences via constrained mechanism (ie can’t change everything; e.g. config file or UI)
  • Publisher can change look and feel via HTML/CSS
  • Gadget themes available, with gallery, for pre-defined configurations
  • Animation used for features such as preference setting and expand/collapse of widgets
  • Drag-and-drop shows preview of page appearance after drop

Container Layout

  • Multi-column (usually 3) vs. freestyle
  • Gadgets can be dragged around the page, displacing other gadgets
  • Tabs allow for multiple layouts
    • Tabs can be renamed
    • Gadgets can be dragged into tabs

Gadget Manipulation

  • Gadgets can be added
    • by URL (with security warnings etc)
    • from Gallery (see Gallery below)
    • by cloning a gadget on user’s page or someone else’s public page or external page
  • Gadgets can be removed
  • Can limit singleton gadgets to one instance per container

Gadget Gallery

  • Gadgets displayed and rendered with metadata embedded in gadget spec, e.g. thumbnail image, author, etc.
  • Gadgets can be browsed by category, date added, etc.
  • # Users can tag gadgets and gadgets can be browsed by tags; tag cloud; etc
  • Users can rate gadgets
  • # Users can recommend gadgets to their friends
  • Users can comment on gadgets

Gadget-To-Gadget Communication

  • Gadgets can communicate with each other (Google PubSub)

Gadget Size

  • # Gadget content can be expanded and collapsed (BBC Beta Homepage)
  • Gadgets can be dynamically resized

Content Sharing

  • Container can be made public (what happens to personalised widgets?)
  • Container can be shared with “friends” (how are friends decided)
  • Users can invite/permit/disallow friends to use their container

Gadget Services

  • Proxying service
  • Caching service (extends proxying service)
  • OpenSocial (or generic social networking) service
  • # Advertising service – Gadgets can serve as ads with revenue model
  • # Financial service – Gadgets can charge for services (subscription, one-off, etc.)

Admin Functions

  • Admin function to tweak gadget/theme rankings, scrutinise/moderate/eliminate gadget/theme submissions, etc
  • Admin function to view metrics, e.g. number of page views, popularity of gadgets, back-end service usage (e.g. proxying and OpenSocial calls)
  • Admin function to manage users (provide support, ban, etc.)