This is the first of an open series on the architecture of Shindig, the new open-source gadget/widget framework project. As mentioned here earlier, this project is building something similar to iGoogle, i.e. an environment for serving gadgets, a run-time environment for the gadgets to operate in, and a gadget container (as well as OpenSocial support).

I’m currently digging into Shindig’s architecture and will document my progress.

For the record, there’s not much discussion of Shindig’s architecture to date. The most useful summaries I’ve seen are a couple of notes on the mailing list:

  • http://mail-archives.apache.org/mod_mbox/incubator-shindig-dev/200801.mbox/%[email protected]%3E
  • http://www.mail-archive.com/[email protected]/msg00369.html
  • http://trac.hyves-api.nl/hyves-api/wiki/ShindigStarted

Also, be aware that Shindig has server-side implementations in both Java and PHP, and potentially more languages in the future. I’m focusing on Java at this time.

I’ll be tagging each of these articles with “shindigging” (as well as “shindig”, a general tag for anything on this blog about shindig). Thus, you’ll be able to find a full list of articles from http://softwareas.com/tag/shindigging.

Java Gadget Server

I’ve walked through each file in the Java gadget server, in the main package - org.apache.shindig.gadgets and taken a very raw set of notes on each file / public class, as well as sketched a quick summary of the process. I’ll refine all this later.

Java Gadget Server - Tracing from gadget spec to page content

A gadget server takes an XML file on a server somewhere and converts it to some HTML/JS/etc content inside an iframe. After looking at org.apache.shindig.gadgets, the Java gadget server achieves this task as follows.

  • GadgetServer is invoked from the web app to render a gadget whose spec sits at a URL
  • GadgetServer uses CacheLoadTask to load the _Gadget_ object if possible
  • If not found, GadgetServer uses SpecLoadTask, which uses RemoteContentFetcher, to grab the Spec.
  • GadgetSpecParser converts the XML string into a GadgetSpec, which is a Java representation of the XML spec.
  • Gadget constructs itself from a combination of the GadgetSpec and the preferences.
  • GadgetServer passes Gadget to each required GadgetFeature (going by the required features declared in the spec). These GadgetFeature objects perform some kind of transformation on the Gadget - typically they add one of more JS libs to it (a gadget has a list of JS libs).
  • At this point, classes in the http package kick in to render the Gadget object, of which more in a different blog post.

Java Gadget Server - Files / Classes in org.apache.shindig.gadgets (raw notes)

BasicGadgetBlacklist.java - [part of GadgetServerConfig] dumb implementation of GadgetBlacklist - file based

BasicGadgetDataCache.java - dumb implementation of GadgetDataCache - Just a hashmap

BasicGadgetSigner.java - dumb implementation of GadgetSigner “Provides dummmy data to satisfy tests and API calls”

BasicGadgetToken.java - dumb (String) implementation of GadgetToken

BasicRemoteContentFetcher.java - server-side remoting proxy

BidiSubstituter.java implements GadgetFeatureFactory - Bidirectional language support (i18n). Performs “hangman” substitutions (MSG_foo). Builds up a Substitutions and executes it.

Gadget.java - It’s a gadget! This object is created from a GadgetSpec and ultimately serialised to a string representing the HTML/JS/etc content that sits on the page. Prior to serialisation, the object is subject to a set of transformations, one for each GadgetFeature it requires.

GadgetBlacklist.java interface - [part of GadgetServerConfig] persists blacklist and lets you query if a given URL is blacklist

GadgetContentFilter.java interface - String->String filter interface to transform the HTML/JS/etc widget content for the browser, e.g. for Caja sanitisation

GadgetContext.java - This object is passed to each GadgetFeature in the processing sequence to tell it what’s going on and help modify its behaviour, since it contains info about gadget server options - ProcessingOptions - as well as Locale, RenderingContext and ServerConfig.

GadgetDataCache.java - [part of GadgetServerConfig] Cache interface. Simply a map from string ID -> Type T.

GadgetException.java - Exception base class

GadgetFeature.java - Transforms a Gadget so it will implement a particular feature. prepare() on initial call and void process(Gadget) later on. TODO more

GadgetFeatureFactory.java - Simply an interface to create Gadgets “GadgetFeature create()”

GadgetFeatureRegistry.java - [part of GadgetServerConfig] A map of gadget features in this gadget server. Essentially Gadget ID string -> {feature object, other features it depends on}

GadgetServer.java

  • Includes processGadget(), which is called by gadget servlet. GadgetID [ie gadget URL] -> Gadget object ready for rendering
  • processGadget() adds a sequence of task objects (commands) and executes them:
  • CacheLoadTask - load gadget from cache instead of fetching/constructing it
  • SpecLoadTask - load gadget from remote URL (using low-level class, RemoteContentFetcher)
  • EnqueueFeaturesTask - popalate Gadget’s list of required gadget feature objects
  • Uses a workflow process: Works iteratively - each cycle, it works out which tasks need to be performed. Keeps iterating until all tasks completed or no new tasks can be added. Meanwhile, accumulates all gadget exceptions for all iterations so they can be bundled together in a big exception option that’s thrown if any exceptions occurred. [Note: I’m not sure why this complicated workflow algorithm is required, when afaict only 3 task objects are present. Maybe more will be added later on.]

GadgetServerConfig.java Configuration options for the gadget server. Composed of java.util.concurrent.Executor, FeatureRegistry, GadgetDataCache, MessageBundleCache, RemoteContentFetcher, GadgetBlacklist, SyndicatorConfig

GadgetServerConfigReader.java Nothing much right now. You’d think it parses a config file or something, but it just ~replicates GadgetServerConfig

GadgetSigner.java interface - defines interface for mapping token ID string -> GadgetToken

GadgetSpec.java - Dumb data structure encapsulating the spec (xml) ie user prefs, required features, gadget URI, HTML content data, random info-garbles (author etc.)

GadgetSpecParser.java - String xml -> GadgetSpec. [java] GadgetSpecParser specParser = new GadgetSpecParser(); GadgetSpec spec = specParser.parse(gadgetId, xml.getResponseAsString()); wc.gadget = new Gadget(gadgetId, spec, prefs); (ie xml file becomes spec, spec becomes gadget) [/java]

GadgetToken.java - Effectively a token string, with a method to sign URLs

GadgetView.java interface - An immutable view of the gadget

JsFeatureLoader.java - Goes into a directory and recursively finds all files matching “feature.xml” Reads each file into a GadgetFeatureRegistry.Entry and registers it into registry (e.g. feature.containerJs.add(JsLibrary) (remember a GadgetFeature modifies the gadget in some way. In the case of a JsFeature (defined in JsLibraryFeatureFactory), the modification is simply to add some JS libraries)

JsLibrary.java [jsLibraries is part of Gadget] - Represents a JS library - holds its source u.g. URL/file) and capable of reading it to get a string of the JS. The source may be a string representing the JS itself, which is useful if the client simply wants to construct the script text programatically.

JsLibraryFeatureFactory.java implements GadgetFeatureFactory - Provides GadgetFeatures in the case where the gadget feature is simply a JS file (or a list of container JS files and a list of gadget JS files). In this case, the feature’s process() method is simply to add all the libraries to the gadget (gadget.addJsLibrary). JsFeatureLoader uses this after trawling through to find the feature.xml for each gadget, since that file simply identifies a bunch of JS libraries.

MessageBundle.java [part of GadgetServerConfig] String ID -> Message map.

MessageBundleParser.java XML file -> MessageBundle

MessageBundleSubstituter.java implements GadgetFeatureFactory - Provides MessageBundleSubstituterFeature. This feature is a Javascript library that “compiles” the MessageBundle to Javascript, for a particular locale. It sets up language and country preference (String setLangFmt = “gadgets.prefs_.setLanguage(%d, “%s”);”; String setCountryFmt = “gadgets.prefs_.setCountry(%d, “%s”);”;), and then sets up, for each message, the JS mapping from ID -> Message ( String setMsgFmt = “gadgets.prefs_.setMsg(%d, %s);” );

ModuleSubstituter.java - Includes ModuleSubstituterFeature which simply replaces MODULE hangman string with the module ID.

OpenSocialFeatureFactory.java - Provides OpenSocialFeature

ProcessingOptions.java - Tweaks GadgetServer.processGadget algorithm (methinks this seems like a weird pattern - should instead be attributes of GadgetServer).

RemoteContent.java - Encapsulates results of HTTP call - the content as well as status code, size, etc.

RemoteContentFetcher.java [part of GadgetServerConfig] - HTTP client to grab gadget spec (nb IMO too much BDUF abstraction going on here)

RemoteContentRequest.java - Encapsulates request for HTTP call - headers etc.

RenderingContext.java - enum { GADGET CONTAINER }

SpecParserException.java - boring exception class

Substitutions.java [part of Gadget] - A collection of Substitutions - each Gadget has a Substitutions object, which it uses for get() queries, e.g. “public String getTitle() { return substitutions.substitute(baseSpec.getTitle()); }”.

  • Several substitution types MSG BIDI UP(user-prefs) MODULE
  • A map for each substitution type, mapping substitution key -> substitution string
  • Runs the sequence of substitutions on a given string

SyndicatorConfig.java [part of GadgetServerConfig] Unclear - related to OpenSocial and JSON.

UserPrefSubstituter.java [part of Gadget] - Builds up JSON object with preference values, using Substitutions to perform any substitutions (???)

UserPrefs.java - preference ID -> string (value of preference)