Reflections from a TiddlyWiki Tiddler and Thoughts on a Guide for Web App Development with TiddlyWiki

I’ve recently begun working on a project with Osmosoft, which I’ll announce Real Soon Now, and have got my hands dirty with TiddlyWiki to the point where I’m now able to make at least some useful functionality. You effectively get an MVC framework for free with TiddlyWiki, so as a power developer, I can see how it could let you build a certain type of app quite rapidly. Even without too much knowledge, you could use it for prototyping quite easily, just by knowing the key extension points. I figured it would be worth capturing my reflections to date.

There’s a lot of documentation around on TiddlyWiki, and also some good support resources in the form of Groups and IRC. What I’d like to see in addition to that is a guide for people writing “verticals” – web apps building on TiddlyWiki, rather than incremental enhancements. And also it would be good to see a cookbook or pattern collection covering the main types of plugins.

The other issue I’ve had, while I’m brain-dumping, is that plugin exceptions are caught at some point, with an error message shown, and not propagated, so they don’t turn up in Firebug. I need to look into this more as I think there are ways to get better diagnostics. I would also like to get a better understanding of the startup and shutdown lifecycle. As explained in my last post, there are some techniques you can use to hook in at certain stages, though no direct support.

I’ll have an initial stab here at the kind of thing I’d like to see in a developer guide, and refine it massively at a time when I’m less of a green Tiddler, maybe on a wiki. I’m really not qualified to say too much at this stage, but I want to capture it while it’s fresh on my mind. Note there’s already a lot of good material along these lines on the TiddlyWiki wiki.

Development Process

The simplest way to work is directly inside the TiddlyWiki. Locate shadow tiddlers under the “more” tab and edit them. And create new tiddlers, tag them as “systemConfig”, and stick your plugin code inside them. Each time you change it, hit reload. That’s a nice, simple, way to start off.

For serious development, there are better options. (Which I’m about to start learning.)

Extending TiddlyWiki into a Custom Web App

The way to extend TiddlyWiki is not to take a TiddlyWiki and start hacking the source. You can do anything you want to do via the following techniques:

  • Edit Shadow Tiddlers Shadow tiddlers are special tiddlers used to configure the TiddlyWiki. When you open a shadow tiddler and edit it, it saves as a regular tiddler. The original shadow tiddler remains, mainly as a safeguard, but becomes irrelevant. (I mention this because you might think there’s a cascading effect, where the regular tiddler is run before or after the shadow. In fact, the regular tiddler simply replaces its shadow from the perspective of configuration.
  • Make Plugins A plugin is a tiddler tagged “systemConfig” and containing some code. The code will be executed when the wiki loads, since TiddlyWiki’s built-in startup sequence looks for tiddlers marked SystemConfig and executes their content.
  • Include Plugins Include plugins from elsewhere if they are useful. You import a plugin via the PluginManager tool, accessible from Backstage. Or by cutting-and-pasting into a new “systemConfig” tiddler.
  • Change Options Options such as Auto-Save also affect your application’s behaviour. You can set them programatically using config.options at startup. Alternatively, if you have retained the options control in the sidebar, set options there and save the page.

In addition, you may find you need more than one TiddlyWiki file to build up your web app. You might end up with a separate file for each key functional area. (I haven’t seen this done, but a guide could explain how to pull in plugins and tiddlers from a common place. Possibly building on TiddlyWeb.)

Which Shadow Tiddlers to Customise?

Here’s a list of shadow tiddlers you might want to customise for your app:

  • SiteTitle – the contents of this tiddler are used in the header. This is an example of an uber-simple shadow tiddler. When you begin to build a vertical, you will want a header that reflects the name of your app. Similarly, you can set SiteSubtitle and SiteURL tiddlers.
  • StyleSheet – the “StyleSheet” tiddlers lets you add some CSS rules to enhance or redefine the existing CSS based look and feel. This is a slightly more complex form of appearance customisation than the previous point. You probably don’t want your app to look like a carbon copy of the original TiddlyWiki, so use these tiddlers to theme it. There are some other style-related tiddlers, like StyleSheetColors and StyleSheetLayout, but it’s not recommended that you change them, because if you ever upgrade the TiddlyWiki core, any new styling will be ignored. So if there are things you want to change in those tiddlers, just create the CSS rule inside StyleSheet and it will take precedence. There is also a ColorPallette tiddler referenced by StyleSheetColors, which you *can* modify safely, and it lets you change the color scheme. In general, you won’t need to override StyleSheetColors directly – ColorPallette should be sufficient.
  • PageTemplate – the HTML for the entire page. An even more complex way to customise your app, by changing the raw page structure. Note the way this tiddler “pulls in” content other tiddlers. You can therefore change the appearance not just by changing PageTemplate, but by changing those included tiddlers. We’ve already met a couple of them – SiteTitle and SiteSubtitle – which simply contain a word or few in most cases. Others you’ll noticed here are MainMenu and Sidebar to reflect further options which are always present in your app.
  • DefaultTiddlers – a list of the initial tiddlers that show up when the page is loaded. In a web app, this would typically be a blurb, maybe embedding a video or image, and links to other tiddlers. The kind of thing you’d see in any web app, really. In this sense, TiddlyWiki provides you with a basic, customisable, layout scheme.
  • etc.

What Kind of Plugins to Create?

Here are some examples of plugin styles:

  • Macro – Your plugin defines a macro. Users include the macro as <<macroname>> inside a tiddler, and your macro is invoked, and it typically spits out some content. You can easily make your macro apply inside all tiddlers by including it in ViewTemplate.
  • Hijacking (aka Monkey Patching) – changing the behaviour of some code that’s already in the application – either core TiddlyWiki code or another plugin you’ve imported. (Could expand this with specific things that are changed)
  • Startup behaviour – Some code executed on startup.

Architectural Principles

  • Reuse – where a plugin already exists, don’t reinvent the wheel. Reuse it. Likewise for core TiddlyWiki components. If possible, hijack (aka monkey patch) to extend it rather than directly hacking it. (Open-Close Principle).
  • Modularity – Plugins build on each other. Don’t write a single “big bang” plugin for your entire app. Break things down into logical units.
  • Flexibility – keep the app open to further customisation from users. For example:
    • Content inside special tiddlers – Instead of hard-coding values, consider pulling them in from special Tiddlers, just like the way SiteTitle and SiteSubtitle shadow tiddlers are used. This way, a power user could easily change the value by modifying the tiddler. If it’s content to be included somewhere, you would usually want to run wikify(tiddler.text) against the tiddler content, so the user can include TiddlyWiki markup. A useful concept here is “slices” – this mechanism lets you include content from just part of a tiddler, not the whole thing. See how StyleSheetColors uses ColorPallette for an example.
    • Javascript inside special tiddlers – Most apps have critical algorithms which power users – if sufficiently authorised – might like to update (“strategy pattern” type modules). Isolate such code into special tiddlers, so someone can change those critical parts of your code without having to dive into the whole thing. If your core code has the right functions available, the tiddler content might be a kind of internal domain-specific language.
    • Favour macros over plugins which just charge in and change stuff. For example, if you were writing a comments plugin. The brute force approach would be to bolt on a comments area to every tiddler. But you could achieve the same thing by creating a comments macro, and then updating ViewTemplate to run the macro (ie adding <> to ViewTemplate). Power users could then have more flexibility in several ways: (a) they could remove comments or alter where they appear; (b) they could use a plugin like TaggedTemplateTweak to ensure comments only appear for certain tiddlers; (c) they could customise the way comments are handled using parameters to the comments tag (assuming your macro accepted parameters).

Key Classes and Functions

Here’s a guide to the most important classes ( (More complete guide on TiddlyWiki internals).):

  • Tiddler – the data model for a single tiddler – its title (tiddler.title), content (tiddler.text), last modifier, etc.
  • TiddlyWiki (~aka “store”) – a collection of tiddlers. Typically, there is just one TiddlyWiki on the page containing all Tiddlers, called “store”. Due to certain hard-codedness, it’s not worth creating a separate TiddlyWiki – just use “store” to create, retrieve, update, and delete Tiddlers (the “CRUD” functions).
  • Story (~aka “story”) – a view showing zero or more tiddlers. The main display area you see on the standard TiddlyWiki page is a Story called “story”. So use “story” to add and remove tiddlers being viewed.
  • (possibly others)
  • window – for completeness sake, I’ll note that there are certain “global” (i.e. window) functions worth knowing about.

Let’s look at how those classes give our web app a model and view.

First, the model. How is CRUD handled?

Tiddlers are the atomic model unit in TiddlyWiki. You can inspect the contents using tiddler attributes:

  • tiddler.title, tiddler.text, tiddler.fields, tiddler.tags. These are the title (which also acts as ID), content, custom fields map, and tags for a tiddler. To access slices, use getTiddlerSlice(). Note that shadow tiddlers aren’t real instances of “Tiddler”, so use getTiddlerText() to retrieve them and inspect their text. (There are also properties tracking creation, modifier, etc.)
  • tiddler.set()
  • -update several features of a tiddler at once.

So the “U” in CRUD – Updating – is handled primarily by setting its values.

As for creation, reading, and deleting – these go against the collection of tiddlers on the page, viz. the “store” object (an instance of TiddlyWiki). The key functions are:

  • Creation – store.createTiddler(various, args, mostly, optional) – makes a new tiddler. You might then customise it further with tiddler.set(). You might also want to display the tiddler, once created, with story.displayTiddler(title).
  • Reading – store.getTiddler(title) – retrieves a tiddler by title.
  • Updating – see above
  • Deleting – store.deleteTiddler(title) – deletes a tiddler by title.

You persist these changes by calling:

  • saveChanges() – save changes to disk (remember, TiddlyWiki runs on a file:/// URL – using cunning manipulations normally exclusive to file:/// URLs, it modifies itself with the updated data)

Next, the view. This is mainly handled by manipulating the “story” global (as well as customising general structure and look-and-feel using the many appearance-related shadow tiddlers). You will probably want to show and hide tiddlers dynamically:

Some important global functions:

  • wikify() – converts a string of marked-up TiddlyWiki content to HTML. (e.g. wikify(''xyz'') becomes <b>xyz</b>.)
  • createTiddlyElement – just a generic utility function to help create DOM elements (notTiddlyWiki-specific, but appears a lot if you’re creating content via DOM manipulation).

Cookbook

  • Introducing a New Tiddler
  • Using Slices
  • Running Code on Startup
  • Turning Auto-Save on
  • etc etc

Testing

Using JSpec, building a test vertical etc.

Tiddlywiki Plugin Authoring: Detecting onload

Tiddlywiki plugins load on startup, early on in the whole boot sequence. I was wondering how to detect when elements are on the page and got some good advice in #tiddlywiki.

Turns out there’s no hooks, but a kind-of-hook is macros’ init() function. I didn’t want to use this, since I’m not creating a macro, though it would have been possible to create a “fake” macro – ie one that’s not intended to be called, but just does something in its init(). However, I went for the more general solution – hijacking (monkey-patching) restart():

javascript
< view plain text >
  1. var origRestart = restart;
  2. window.restart = function() {
  3.   origRestart();
  4.   ... some code which can assume elements are on the page ...
  5. }

mmahemoff: hi, i’m writing a plugin that needs to load as soon as the page is ready…is there an event handler for that?
mmahemoff: specifically, my PageTemplate contains an element and I want my plugin to do document.getElementById(“element”)
Ace_NoOne: mmahemoff: well, there’s a ticket for those kinda hooks – but no implementation so far AFAIK
Ace_NoOne: a hacky way would be to write a macro that’s hidden in (executed by) some PageTemplate element
jayfresh: mmahemoff: there is this:
jayfresh: var startingUp = false;
jayfresh: that’s in the global space and is true during main()
jayfresh: and false the rest of the time
Ace_NoOne: FWIW, http://trac.tiddlywiki.org/ticket/484
Ace_NoOne: has some links
Ace_NoOne: specifically: http://www.tiddlywiki.org/wiki/Dev:Startup_Phases mmahemoff: great tips, thanks :)
Ace_NoOne: mmahemoff: macros’ init function run after refreshDisplay
mmahemoff: ok, that ought to do it in my case
mmahemoff: in the more general case, it sounds like hijacking restart() is the typical pattern?
Ace_NoOne: mmahemoff: saqimtiaz1 probably knows best
saqimtiaz1: mmahemoff: standard practice is to hijack restart. Not pretty but the best we have right now

(Snipped some discussion related to a separate thread.)

Frame-Busting Gadgets

In the questions after my @media ajax talk, Simon Willison asked about frame busting. If gadgets sit inside iframes, what’s to stop them from busting the frame, i.e. replacing the container with another website. I notice he made a similar comment when OpenSocial came out. If a gadget can cause iGoogle to go away in place of another website, it could for example launch a phishing attack by presenting a mock iGoogle login page (“iGoogle has timed out. Please re-enter your username and password.”).

At a technical level, there is actually nothing to prevent this. There’s no magic in the container. IFrames are about as good as it gets in modern browser for taming the gadgets, but there are still problems with that model, and one of those is the ability for iframes to bust out. I decided to create a demo of this. The gadget is here. You can drop it into iGoogle and see for yourself. Video below:

[kml_flashembed movie="http://picupper.com/video/framebust.swf" height="700" width="400" /]

(Parenthetically, the video is vertical. Why are videos always horizontal on the internets? It’s a computer, not a TV! It should be just as easy to embed an dodecahedron video if that’s what I fancy at the time.)

At this stage, there’s no technical way around this. One possibility would be for the container to do something in the rarely-used onexit() method, which runs just before the new page is loaded. However, as I learned with WebWait, which is also vulnerable to frame-busting, there’s nothing you can really do at that stage, and you can’t determine if the exit is happening because of a frame-busting event as opposed to the user just typing in a new URL. I suppose onexit() could still be used to log info to the server – if you knew all the times the container exits and which gadgets were present at that time, you might find certain (malicious) gadgets were present more times than you’d expect if they were present randomly. Although, I’m not sure about how reliable it is to make an Ajax call in the onexit(). (In this case, though, it would only need to work some of the time.)

In the future, the technical solution is Caja, if it proves to be production-ready. Caja ensures a safe subset of Javascript, so a container like iGoogle would only ever serve a gadget if it was verified to be safe.

Until then, we have to rely on “social” mechanisms. i.e. user comments in the catalogue and manual checking by container staff. A while ago, iGoogle made it harder to add a gadget directly by URL – you basically have to use the catalogue unless you’re a developer. So the catalogue is effectively acting as a whitelist – once found to be malicious, a gadget could be removed. This isn’t simple though – gadgets are dynamic and the catalogue is effectively just a list of URLs. A malicious author could have their gadget added to the catalogue and then change it, so the gadgets need constant review – a highly ineffective process.

I also mentioned in the presentation that “html” type gadgets are more recommended than “url” gadgets. This is partly because with html gadgets, the container at least has some idea of what’s happening inside the gadget – though not entirely, since the code can still reference external scripts and services. With “url” gadgets, what happens is entirely controlled by an external server.

OpenSocial-Tiddlywiki Integration

Go straight to the demo.

Google Gadget TiddlyWiki plugin - a tiddlywiki macro to embed google/opensocial gadgets

I’m at the Osmosoft hackathon and finally got an opportunity to experiment with OpenSocial-Tiddlywiki integration. Specifically, how to embed OpenSocial gadgets in Tiddlywiki. I paired (trio’d?) up with Stuart Race and Tom D’Roza on it. We managed to get something useful up in an hour or two, which is testimony to the simplicity of both OpenSocial and Tiddlywiki. It’s only proof-of-concept for now and for some reason doesn’t work in IE.

Technically, it’s iGoogle integration, not OpenSocial. The gadgets are iframes pointing to the iGoogle gadget server (gmodules.com). That’s because right now, it’s not clear there are any OpenSocial containers that will serve out gadgets externally in the same way. Of the various live OpenSocial containers, there is really scant information about how to embed the gadgets in an external page. Suffice to say, the code here is virtually identical to what you’d need to embed an OpenSocial gadget – it’s simply a matter of changing the endpoint URL from whence gadgets are served.

One concern about this is whether or not Google will be cool with us pointing directly to the iframe. Technically, you’re supposed to embed gadgets using a script tag, which will document.writeln() the iframe and its source. I vaguely recall a time last year where we were playing around with direct calls to the gadget, and gmodules saying no. This is possible in theory, if the server was attempting to match each iframe request with a previous corresponding script call. However, I am probably dreaming. It seems to be working fine. In any event, Shindig has no constraint like that, so if we extend this to point to other opensocial servers, or something spun up by an individual, it should work fine.

Usage

With the plugin installed, it’s simple to use:

<<gadget http://abowman.googlepages.com/spider.xml >>

or with options:

<<gadget http://www.btinternet.com/~tdroza/gadgets/twitter/index.xml height:400 width:500 border:0 prefs:"up_max_items=5&up_username=downingstreet&up_feed=http://twitter.com/statuses/user_timeline/" >>

If you’re familiar with opensocial you’ll realise that the height, width, and border are options to the gadget server telling it how to render the gadget and its chrome; whereas the prefs string contains preferences which the gadget itself gets. All this is optional.

Plugin Code

We began with the Shout plugin whose code I’ve already documented. It was only a matter of creating an iframe and pointing it – by setting its src property – to “http://www.gmodules.com/ig/ifr?url=”+params[0]. And that worked fine, and we had gadgets inside tiddlywiki.

The hard part was parameters, took a little investigation to work out how to handle key-value pair parameters. The plugin code below serves as a simple example for how it’s done. It’s very similar to the very cool Ruby/Rails idiom which has also been adopted by Javascript libraries like Prototype (obviously, given its Rails association) and JQuery. i.e. mandatory params followed by a hash of optional params. Here, the URL is mandatory, and its proceeded by a set of key-value pairs.

The plugin code:

javascript
< view plain text >
  1. config.macros.gadget = {};
  2. config.macros.gadget .handler = function(place,macroName,params, wikifier, paramString) {
  3. var elem = createTiddlyElement(place,"iframe",null,"greeting","");
  4. var p = paramString.parseParams(null, null, true);
  5.  
  6. elem.src= "http://www.gmodules.com/ig/ifr?url=" + params[0] + "&" + getParam(p,"prefs","");
  7. elem.height = getParam(p,"height","200");
  8. elem.width= getParam(p,"width","300");
  9. elem.style.border= getParam(p,"border","1");
  10.  
  11. place.appendChild(elem);
  12. }

That’s it.

Applications

Why would you want to embed a gadget in a tiddlywiki, aside from proof of concept? Well, tiddlywiki is a platform whose applications are many and varied; OpenSocial integration simply enhances the platform so that developers can select from a rich palette of existing gadgets. For example, if your tiddlywiki involves trip planning, pull in a Google Maps gadget.

It also makes for an extremely easy way to collect up a bunch of related gadgets, and optionally have a conversation around them. For example, you could compile your favourite game and toy gadgets into a game tiddlywiki. Remember tiddlywikis are single files, so this becomes a single file game.html which you can mail around, keep on your hard drive, or post to a server. A related usage would be a single tiddlywiki where all gadgets are the same type, but with different preference strings. For example, 10 weather gadgets with different “location” settings.