Offline Sound: No Flash, No File

I just did some tinkering with offline sound and it turns out you can embed an audio clip in an HTML file, and play it without using Flash. I could have demo’d this in a raw HTML file, but it was just as easy to stick it in a Tiddlywiki file. So I made a TiddlyWiki called JinglyWiki. Click on the button and it will play a sound. If you grab the file (safest using a tool like Curl or wget, as the browser will sometimes Save As something different), you can run it offline and it will still play the sound.

JinglyWiki started after an office conversation yesterday. We had a flaky wifi connection and were pleased to be able to play a beat on http://instantrimshot.com to signify a re-connection. This got us thinking about playing beats in Tiddlywiki. Jeremy mentioned the possibility of data: URIs, and I thought back to Reinier’s great effort in playing sound without Flash. I wondered if it would work with data: URIs and to my pleasant surprise, it did. At least in FF3, which is all I’ve tested so far. I also forgot how tiny the code is to play audio without Flash – you just add an element to the page.

The point of JinglyWiki is you can stick a 300K file on your local file system or a USB stick and play gratuitous sounds without being online. Your iPod cost you several hundred bucks, whereas JinglyWiki is entirely free. Free as in you can spend your money on beer instead. JinglyWiki may well be the best portable music player you never paid a penny for. Groovy.

It would be so much more useful with some code to dynamically generate WAVs, but that’s a project for another day.

As for the name, it’s a play on the much more important project Phil Hawksworth has initiated to build a JQuery based Tiddlywiki framework: JigglyWiki. I think this nascent effort is going to produce some wonderful plugins for the JQuery community and be an important enough project to derive novelty joke names out of :), so I’m setting the trend here.

The data itself is 15KB for about a second of voice content.

Here’s the plugin code. Trivial or wot?!!!

javascript

  1. config.macros.jingle = {}
  2. config.macros.jingle.handler = function(place,macroName,params,wikifier,paramString,tiddler) {
  3.  
  4.   button = document.createElement("button");
  5.   button.style.fontSize = "4em";
  6.   button.innerHTML = "Play Me! Play Me!"
  7.   button.onclick = function(ev) {
  8.     startWav(SOUND_URL);
  9.   }
  10.   place.appendChild(button);
  11.  
  12.   // cancel dbl-click so we can follow our natural urge to click on the button incessantly
  13.   story.getTiddler(tiddler.title).ondblclick = function() {}
  14.  
  15. }
  16.  
  17. // non-flash sound handling adapted from http://www.zwitserloot.com/files/soundkit/soundcheck.html
  18.  
  19. var embedEl;
  20.  
  21. function startWav(uri) {
  22.         stopWav();
  23.         embedEl = document.createElement("embed");
  24.         embedEl.setAttribute("src", uri);
  25.         embedEl.setAttribute("hidden", true);
  26.         embedEl.setAttribute("autostart", true);
  27.         document.body.appendChild(embedEl);
  28. }
  29.  
  30. function stopWav() {
  31.   if (embedEl) document.body.removeChild(embedEl);
  32.   embedEl = null;
  33. }
  34.  
  35. // To make a new sound, cut and paste into hixie's handy Data URI Kitchen
  36. // http://software.hixie.ch/utilities/cgi/data/data.pl
  37.  
  38. var SOUND_URL="data:audio/x-wav,RIFF%17%1C%00%00WAVEfmt%20%10%00%00%00%01%00%01%00%40%1F%00%00%40 ........................" // snipped

Reflections from a TiddlyWiki Tiddler and Thoughts on a Guide for Web App Development with TiddlyWiki

I’ve recently begun working on a project with Osmosoft, which I’ll announce Real Soon Now, and have got my hands dirty with TiddlyWiki to the point where I’m now able to make at least some useful functionality. You effectively get an MVC framework for free with TiddlyWiki, so as a power developer, I can see how it could let you build a certain type of app quite rapidly. Even without too much knowledge, you could use it for prototyping quite easily, just by knowing the key extension points. I figured it would be worth capturing my reflections to date.

There’s a lot of documentation around on TiddlyWiki, and also some good support resources in the form of Groups and IRC. What I’d like to see in addition to that is a guide for people writing “verticals” – web apps building on TiddlyWiki, rather than incremental enhancements. And also it would be good to see a cookbook or pattern collection covering the main types of plugins.

The other issue I’ve had, while I’m brain-dumping, is that plugin exceptions are caught at some point, with an error message shown, and not propagated, so they don’t turn up in Firebug. I need to look into this more as I think there are ways to get better diagnostics. I would also like to get a better understanding of the startup and shutdown lifecycle. As explained in my last post, there are some techniques you can use to hook in at certain stages, though no direct support.

I’ll have an initial stab here at the kind of thing I’d like to see in a developer guide, and refine it massively at a time when I’m less of a green Tiddler, maybe on a wiki. I’m really not qualified to say too much at this stage, but I want to capture it while it’s fresh on my mind. Note there’s already a lot of good material along these lines on the TiddlyWiki wiki.

Development Process

The simplest way to work is directly inside the TiddlyWiki. Locate shadow tiddlers under the “more” tab and edit them. And create new tiddlers, tag them as “systemConfig”, and stick your plugin code inside them. Each time you change it, hit reload. That’s a nice, simple, way to start off.

For serious development, there are better options. (Which I’m about to start learning.)

Extending TiddlyWiki into a Custom Web App

The way to extend TiddlyWiki is not to take a TiddlyWiki and start hacking the source. You can do anything you want to do via the following techniques:

  • Edit Shadow Tiddlers Shadow tiddlers are special tiddlers used to configure the TiddlyWiki. When you open a shadow tiddler and edit it, it saves as a regular tiddler. The original shadow tiddler remains, mainly as a safeguard, but becomes irrelevant. (I mention this because you might think there’s a cascading effect, where the regular tiddler is run before or after the shadow. In fact, the regular tiddler simply replaces its shadow from the perspective of configuration.
  • Make Plugins A plugin is a tiddler tagged “systemConfig” and containing some code. The code will be executed when the wiki loads, since TiddlyWiki’s built-in startup sequence looks for tiddlers marked SystemConfig and executes their content.
  • Include Plugins Include plugins from elsewhere if they are useful. You import a plugin via the PluginManager tool, accessible from Backstage. Or by cutting-and-pasting into a new “systemConfig” tiddler.
  • Change Options Options such as Auto-Save also affect your application’s behaviour. You can set them programatically using config.options at startup. Alternatively, if you have retained the options control in the sidebar, set options there and save the page.

In addition, you may find you need more than one TiddlyWiki file to build up your web app. You might end up with a separate file for each key functional area. (I haven’t seen this done, but a guide could explain how to pull in plugins and tiddlers from a common place. Possibly building on TiddlyWeb.)

Which Shadow Tiddlers to Customise?

Here’s a list of shadow tiddlers you might want to customise for your app:

  • SiteTitle – the contents of this tiddler are used in the header. This is an example of an uber-simple shadow tiddler. When you begin to build a vertical, you will want a header that reflects the name of your app. Similarly, you can set SiteSubtitle and SiteURL tiddlers.
  • StyleSheet – the “StyleSheet” tiddlers lets you add some CSS rules to enhance or redefine the existing CSS based look and feel. This is a slightly more complex form of appearance customisation than the previous point. You probably don’t want your app to look like a carbon copy of the original TiddlyWiki, so use these tiddlers to theme it. There are some other style-related tiddlers, like StyleSheetColors and StyleSheetLayout, but it’s not recommended that you change them, because if you ever upgrade the TiddlyWiki core, any new styling will be ignored. So if there are things you want to change in those tiddlers, just create the CSS rule inside StyleSheet and it will take precedence. There is also a ColorPallette tiddler referenced by StyleSheetColors, which you *can* modify safely, and it lets you change the color scheme. In general, you won’t need to override StyleSheetColors directly – ColorPallette should be sufficient.
  • PageTemplate – the HTML for the entire page. An even more complex way to customise your app, by changing the raw page structure. Note the way this tiddler “pulls in” content other tiddlers. You can therefore change the appearance not just by changing PageTemplate, but by changing those included tiddlers. We’ve already met a couple of them – SiteTitle and SiteSubtitle – which simply contain a word or few in most cases. Others you’ll noticed here are MainMenu and Sidebar to reflect further options which are always present in your app.
  • DefaultTiddlers – a list of the initial tiddlers that show up when the page is loaded. In a web app, this would typically be a blurb, maybe embedding a video or image, and links to other tiddlers. The kind of thing you’d see in any web app, really. In this sense, TiddlyWiki provides you with a basic, customisable, layout scheme.
  • etc.

What Kind of Plugins to Create?

Here are some examples of plugin styles:

  • Macro – Your plugin defines a macro. Users include the macro as <<macroname>> inside a tiddler, and your macro is invoked, and it typically spits out some content. You can easily make your macro apply inside all tiddlers by including it in ViewTemplate.
  • Hijacking (aka Monkey Patching) – changing the behaviour of some code that’s already in the application – either core TiddlyWiki code or another plugin you’ve imported. (Could expand this with specific things that are changed)
  • Startup behaviour – Some code executed on startup.

Architectural Principles

  • Reuse – where a plugin already exists, don’t reinvent the wheel. Reuse it. Likewise for core TiddlyWiki components. If possible, hijack (aka monkey patch) to extend it rather than directly hacking it. (Open-Close Principle).
  • Modularity – Plugins build on each other. Don’t write a single “big bang” plugin for your entire app. Break things down into logical units.
  • Flexibility – keep the app open to further customisation from users. For example:
    • Content inside special tiddlers – Instead of hard-coding values, consider pulling them in from special Tiddlers, just like the way SiteTitle and SiteSubtitle shadow tiddlers are used. This way, a power user could easily change the value by modifying the tiddler. If it’s content to be included somewhere, you would usually want to run wikify(tiddler.text) against the tiddler content, so the user can include TiddlyWiki markup. A useful concept here is “slices” – this mechanism lets you include content from just part of a tiddler, not the whole thing. See how StyleSheetColors uses ColorPallette for an example.
    • Javascript inside special tiddlers – Most apps have critical algorithms which power users – if sufficiently authorised – might like to update (“strategy pattern” type modules). Isolate such code into special tiddlers, so someone can change those critical parts of your code without having to dive into the whole thing. If your core code has the right functions available, the tiddler content might be a kind of internal domain-specific language.
    • Favour macros over plugins which just charge in and change stuff. For example, if you were writing a comments plugin. The brute force approach would be to bolt on a comments area to every tiddler. But you could achieve the same thing by creating a comments macro, and then updating ViewTemplate to run the macro (ie adding <> to ViewTemplate). Power users could then have more flexibility in several ways: (a) they could remove comments or alter where they appear; (b) they could use a plugin like TaggedTemplateTweak to ensure comments only appear for certain tiddlers; (c) they could customise the way comments are handled using parameters to the comments tag (assuming your macro accepted parameters).

Key Classes and Functions

Here’s a guide to the most important classes ( (More complete guide on TiddlyWiki internals).):

  • Tiddler – the data model for a single tiddler – its title (tiddler.title), content (tiddler.text), last modifier, etc.
  • TiddlyWiki (~aka “store”) – a collection of tiddlers. Typically, there is just one TiddlyWiki on the page containing all Tiddlers, called “store”. Due to certain hard-codedness, it’s not worth creating a separate TiddlyWiki – just use “store” to create, retrieve, update, and delete Tiddlers (the “CRUD” functions).
  • Story (~aka “story”) – a view showing zero or more tiddlers. The main display area you see on the standard TiddlyWiki page is a Story called “story”. So use “story” to add and remove tiddlers being viewed.
  • (possibly others)
  • window – for completeness sake, I’ll note that there are certain “global” (i.e. window) functions worth knowing about.

Let’s look at how those classes give our web app a model and view.

First, the model. How is CRUD handled?

Tiddlers are the atomic model unit in TiddlyWiki. You can inspect the contents using tiddler attributes:

  • tiddler.title, tiddler.text, tiddler.fields, tiddler.tags. These are the title (which also acts as ID), content, custom fields map, and tags for a tiddler. To access slices, use getTiddlerSlice(). Note that shadow tiddlers aren’t real instances of “Tiddler”, so use getTiddlerText() to retrieve them and inspect their text. (There are also properties tracking creation, modifier, etc.)
  • tiddler.set()
  • -update several features of a tiddler at once.

So the “U” in CRUD – Updating – is handled primarily by setting its values.

As for creation, reading, and deleting – these go against the collection of tiddlers on the page, viz. the “store” object (an instance of TiddlyWiki). The key functions are:

  • Creation – store.createTiddler(various, args, mostly, optional) – makes a new tiddler. You might then customise it further with tiddler.set(). You might also want to display the tiddler, once created, with story.displayTiddler(title).
  • Reading – store.getTiddler(title) – retrieves a tiddler by title.
  • Updating – see above
  • Deleting – store.deleteTiddler(title) – deletes a tiddler by title.

You persist these changes by calling:

  • saveChanges() – save changes to disk (remember, TiddlyWiki runs on a file:/// URL – using cunning manipulations normally exclusive to file:/// URLs, it modifies itself with the updated data)

Next, the view. This is mainly handled by manipulating the “story” global (as well as customising general structure and look-and-feel using the many appearance-related shadow tiddlers). You will probably want to show and hide tiddlers dynamically:

Some important global functions:

  • wikify() – converts a string of marked-up TiddlyWiki content to HTML. (e.g. wikify(''xyz'') becomes <b>xyz</b>.)
  • createTiddlyElement – just a generic utility function to help create DOM elements (notTiddlyWiki-specific, but appears a lot if you’re creating content via DOM manipulation).

Cookbook

  • Introducing a New Tiddler
  • Using Slices
  • Running Code on Startup
  • Turning Auto-Save on
  • etc etc

Testing

Using JSpec, building a test vertical etc.

Tiddlywiki Plugin Authoring: Detecting onload

Tiddlywiki plugins load on startup, early on in the whole boot sequence. I was wondering how to detect when elements are on the page and got some good advice in #tiddlywiki.

Turns out there’s no hooks, but a kind-of-hook is macros’ init() function. I didn’t want to use this, since I’m not creating a macro, though it would have been possible to create a “fake” macro – ie one that’s not intended to be called, but just does something in its init(). However, I went for the more general solution – hijacking (monkey-patching) restart():

javascript

  1. var origRestart = restart;
  2. window.restart = function() {
  3.   origRestart();
  4.   ... some code which can assume elements are on the page ...
  5. }

mmahemoff: hi, i’m writing a plugin that needs to load as soon as the page is ready…is there an event handler for that?
mmahemoff: specifically, my PageTemplate contains an element and I want my plugin to do document.getElementById(“element”)
Ace_NoOne: mmahemoff: well, there’s a ticket for those kinda hooks – but no implementation so far AFAIK
Ace_NoOne: a hacky way would be to write a macro that’s hidden in (executed by) some PageTemplate element
jayfresh: mmahemoff: there is this:
jayfresh: var startingUp = false;
jayfresh: that’s in the global space and is true during main()
jayfresh: and false the rest of the time
Ace_NoOne: FWIW, http://trac.tiddlywiki.org/ticket/484
Ace_NoOne: has some links
Ace_NoOne: specifically: http://www.tiddlywiki.org/wiki/Dev:Startup_Phases mmahemoff: great tips, thanks 🙂
Ace_NoOne: mmahemoff: macros’ init function run after refreshDisplay
mmahemoff: ok, that ought to do it in my case
mmahemoff: in the more general case, it sounds like hijacking restart() is the typical pattern?
Ace_NoOne: mmahemoff: saqimtiaz1 probably knows best
saqimtiaz1: mmahemoff: standard practice is to hijack restart. Not pretty but the best we have right now

(Snipped some discussion related to a separate thread.)

OpenSocial-Tiddlywiki Integration

Go straight to the demo.

Google Gadget TiddlyWiki plugin - a tiddlywiki macro to embed google/opensocial gadgets

I’m at the Osmosoft hackathon and finally got an opportunity to experiment with OpenSocialTiddlywiki integration. Specifically, how to embed OpenSocial gadgets in Tiddlywiki. I paired (trio’d?) up with Stuart Race and Tom D’Roza on it. We managed to get something useful up in an hour or two, which is testimony to the simplicity of both OpenSocial and Tiddlywiki. It’s only proof-of-concept for now and for some reason doesn’t work in IE.

Technically, it’s iGoogle integration, not OpenSocial. The gadgets are iframes pointing to the iGoogle gadget server (gmodules.com). That’s because right now, it’s not clear there are any OpenSocial containers that will serve out gadgets externally in the same way. Of the various live OpenSocial containers, there is really scant information about how to embed the gadgets in an external page. Suffice to say, the code here is virtually identical to what you’d need to embed an OpenSocial gadget – it’s simply a matter of changing the endpoint URL from whence gadgets are served.

One concern about this is whether or not Google will be cool with us pointing directly to the iframe. Technically, you’re supposed to embed gadgets using a script tag, which will document.writeln() the iframe and its source. I vaguely recall a time last year where we were playing around with direct calls to the gadget, and gmodules saying no. This is possible in theory, if the server was attempting to match each iframe request with a previous corresponding script call. However, I am probably dreaming. It seems to be working fine. In any event, Shindig has no constraint like that, so if we extend this to point to other opensocial servers, or something spun up by an individual, it should work fine.

Usage

With the plugin installed, it’s simple to use:

<<gadget http://abowman.googlepages.com/spider.xml >>

or with options:

<<gadget http://www.btinternet.com/~tdroza/gadgets/twitter/index.xml height:400 width:500 border:0 prefs:"up_max_items=5&up_username=downingstreet&up_feed=http://twitter.com/statuses/user_timeline/" >>

If you’re familiar with opensocial you’ll realise that the height, width, and border are options to the gadget server telling it how to render the gadget and its chrome; whereas the prefs string contains preferences which the gadget itself gets. All this is optional.

Plugin Code

We began with the Shout plugin whose code I’ve already documented. It was only a matter of creating an iframe and pointing it – by setting its src property – to “http://www.gmodules.com/ig/ifr?url=”+params[0]. And that worked fine, and we had gadgets inside tiddlywiki.

The hard part was parameters, took a little investigation to work out how to handle key-value pair parameters. The plugin code below serves as a simple example for how it’s done. It’s very similar to the very cool Ruby/Rails idiom which has also been adopted by Javascript libraries like Prototype (obviously, given its Rails association) and JQuery. i.e. mandatory params followed by a hash of optional params. Here, the URL is mandatory, and its proceeded by a set of key-value pairs.

The plugin code:

javascript

  1. config.macros.gadget = {};
  2. config.macros.gadget .handler = function(place,macroName,params, wikifier, paramString) {
  3. var elem = createTiddlyElement(place,"iframe",null,"greeting","");
  4. var p = paramString.parseParams(null, null, true);
  5.  
  6. elem.src= "http://www.gmodules.com/ig/ifr?url=" + params[0] + "&" + getParam(p,"prefs","");
  7. elem.height = getParam(p,"height","200");
  8. elem.width= getParam(p,"width","300");
  9. elem.style.border= getParam(p,"border","1");
  10.  
  11. place.appendChild(elem);
  12. }

That’s it.

Applications

Why would you want to embed a gadget in a tiddlywiki, aside from proof of concept? Well, tiddlywiki is a platform whose applications are many and varied; OpenSocial integration simply enhances the platform so that developers can select from a rich palette of existing gadgets. For example, if your tiddlywiki involves trip planning, pull in a Google Maps gadget.

It also makes for an extremely easy way to collect up a bunch of related gadgets, and optionally have a conversation around them. For example, you could compile your favourite game and toy gadgets into a game tiddlywiki. Remember tiddlywikis are single files, so this becomes a single file game.html which you can mail around, keep on your hard drive, or post to a server. A related usage would be a single tiddlywiki where all gadgets are the same type, but with different preference strings. For example, 10 weather gadgets with different “location” settings.

Tiddlywiki internals 3 of 3: Key Javascript Classes and Files

(This is part 3 of a 3-part series. Part 1 introduces the internals and highlights some of the key patterns and concepts. Part 2 introduces each Javascript file. Part 3 focuses on the internals of the more important classes and files.)

Concluding this series, below is a list of all core Javascript files, organised into functional groups.

main.js

main() is the function that runs onload.

Key functions:

  • creates a new tiddlywiki data store (new TiddlyWiki()) – this is the collection of tiddlers users are exposed to. The store is populated using TiddlyWiki.prototype.loadFromDiv(), which loads all the tiddlers from the “storeArea” div, which is an invisible block on the page (and rendered back in nice – and visible – manner later on).
  • creates a second TiddlyWiki data store to hold “shadow tiddlers” – these are “meta”/config tiddlers holding data such as CSS styling. Populated from invisible “shadowArea” div (which at compile time is defined in the Shadow/ directory).
  • creates a new “Story div”, a div which will show tiddlers to the user, and themes it according to config.options.txtTheme
  • sets up Popup.onDocumentClick (removes popup menus when user clicks outside of the menu)
  • sets up event propagation – certain tiddlers are notified when certain actions occur. The mappings are defined in refresh.js (e.g. {name: “StyleSheetLayout”, notify: refreshStyles})
  • sets up and renders backstage
  • loads plugins (plugins are evidently supposed to set a global “plugin problem” value if a problem occurs)

General:

  • calls several lifecycle event handlers as it loads – the wiki config can provide hook functions which run upon particular lifetime events
  • benchmarks most of the above (the benchmarking was possibly a quick fix – relies on variables t1,t2…t10 -> this code could be optimised for conciseness using function wrappers, but maybe startup would be too slow that way).
  • After initial setup ensures tiddlywiki data structures and other initialisation/config pieces are in place, it blats and shows the display with restart() and refreshDisplay().

Plugins

Tiddlywiki has a strong plugin architecture. Each plugin is included as a regular (non-shadow) tiddler, one that must be tagged “systemConfig”. (For all intents and purposes, “systemConfig” is synonymous with “plugin”.) There’s an example shipping with the default tiddlywiki instance on tiddlywiki.com (and a more detailed example in the source code – association/plugins/SparklinePlugin/SparklinePlugin.js). (Also of interest, the latest plugin template at the tiddlywiki.org wiki.)

  1. <div title="ExamplePlugin" modifier="JeremyRuston" created="200607271914" modified="200609212329" tags="systemConfig">
  2. <pre>/***
  3. |''Name:''|ExamplePlugin|
  4. |''Description:''|To demonstrate how to write TiddlyWiki plugins|
  5. |''Version:''|2.0.3|
  6. |''Date:''|Sep 22, 2006|
  7. |''Source:''|http://www.tiddlywiki.com/#ExamplePlugin|
  8. |''Author:''|JeremyRuston (jeremy (at) osmosoft (dot) com)|
  9. |''License:''|[[BSD open source license]]|
  10. |''~CoreVersion:''|2.1.0|
  11. |''Browser:''|Firefox 1.0.4+; Firefox 1.5; InternetExplorer 6.0|
  12. ***/
  13.  
  14. //{{{
  15.  
  16. // Uncomment the following line to see how the PluginManager deals with errors in plugins
  17. // deliberateError();
  18.  
  19. // Log a message
  20. pluginInfo.log.push(&quot;This is a test message from &quot; + tiddler.title);
  21.  
  22. //}}}</pre>
  23. </div>

A plugin is essentially just a Javascript block which gets executed on page load. All the biosketch info is optional (although in some cases, it does effect processing, e.g. there is a check against the required TiddlyWiki version). “Just some Javascript” did you say? This post on JQuery plugins by JQuery daddy John Resig is instructive. His point is that a plugin architecture needs explicit points for plugins to hook into – i.e. an API – and the existence of a plugin catalogue. Tiddlywiki doesn’t have a plugin API per se, but is structured with plenty of extension points to naturally hook into. As for the catalogue, there’s also a plugin wiki area, with a grander-scale plugin repo project in progress.

Incidentally, note that you don’t have to register the Javascript block as you might do in some other frameworks (e.g. runOnInit(myPlugin); ). It executes automatically when plugins are loaded.

Okay, so about those plugin extension points. I’m still learning that. In the case of sparklines, the purpose is to create a new macro (e.g. <<sparkline 100 200 300>>), so it defines config.macros.sparkline.handler(place,macroName,params), and its “output” is to populate the place element with sparkline content.

Another popular pasttime for plugin developers is szhushing the global Formatter object to shape how stuff gets rendered. e.g. if your formatter locates the built-in formatter named “heading”, it could easily overwrite its handler method to MAKE ALL THE HEADINGS SHOUT AT UNSUSPECTING READERS.

To install a plugin, users usually use the Import dialog, accessible from Backstage. It’s also possible to manually include plugins via cut-and-paste into Tiddlywiki.

There’s much more to be said about plugins. The bottom line is that Tiddlywiki’s architecture lets you bend the core product into many things. (By “architecture”, I refer to both the plugin mechanism and the flexible nature in which the overall architecture is structured.)

Tiddlers

Tiddlers are the atomic content blocks that make up a Tiddlywiki, typically about a paragraph in length. A Tiddler is simply a block of text, with extra info like a title, a list of tags, and timestamp data. There’s also a fields hash where you could store any arbitrary properties. (This seems suitable for plugins, but the core also makes use of it, and I don’t really get that. Even for plugins, why can’t they just make new fields dynamically?)

Tiddler is a Javascript class, so you get a new instance with new Tiddler(). Internally, it uses a publish-subscribe mechanism, where a changed() method is called after any mutation. This basically ensures the links property is up to date, as links is a redundant (and presumably there for performance) collection of links inside the tiddler.

A Tiddler also has a collection of “slices”, though the collection is managed by TiddlyWiki rather than Tiddler. (This relates to the fact that shadow tiddlers are mere text blocks – using Tiddlywiki to extract slices ensures shadow tiddlers can also be sliced up….and slices are a major feature of most shadow tiddlers, since they are config-related.)

There’s a string->string map from name to slice. This is similar to the fields hash, insofar as it’s a free-form map. In this case, though, it’s something that can easily be changed by the user in real time, as the slice collection is sync’d to the tiddler content. For example: |”slicename:”|”some slice content”|. Slices allow for easily edited meta-data, e.g. a stylesheet tiddler can have a slice called “backgroundColour”. Users then edit the backgroundColor slice content to set the background colour.

A Tiddler also has a set of notification handlers – this is also managed by TiddlyWiki rather than the Tiddlers themselves (again, this ensures the mechanism works for shadow tiddlers). These are listeners/observers that are notified each time tiddler is changed.

A file closely related to Tiddler is TiddlerFields.js. It actually alters the TiddlyWiki definition rather than the Tiddler definition, but in any event it deals with accessing the Tiddler’s fields map.

Shadow Tiddlers

Shadow tiddlers are a particular type of tiddler. There’s no separate “ShadowTiddler” class, but they are held in a separate store and treated in special ways. Indeed, shadow tiddlers aren’t actually of class Tiddler (which is slightly confusing). They are simply a title-text pairing; the data structure is a map from title to text. In contrast, regular Tiddlers are mapped from title to Tiddler.

In particular, TiddlyWiki has a fallback mechanism when asked to return a tiddler – if the tiddler doesn’t exist, it will attempt to revert to a shadow tiddler of the same name. Shadow tiddlers are immutable (unless you hack source code), whereas tiddlers are of course easily edited. You can override shasow tiddlers with regular tiddlers of the same name, but the original shadow still lurks (in a good way) in the background.

To see this, open an editable Tiddlywiki, choose a shadow tiddler from the right sidebar Contents menu (e.g. SiteUrl), edit it, and save it. Then re-open it to verify your changes were affected. Then delete it, and notice that it’s still in the list of shadow tiddlers. When you open it yet again, you’ll see it now contains the original content. (The shadow tiddler itself never changed.)

Shadow tiddlers are used for config stuff like stylesheets. The fail-safe mechanism ensures you can easily “restore factory defaults” at any time.

TiddlyWiki

A Tiddlywiki is essentially a hash of Tiddlers, keyed on their title. More precisely, it’s a wrapper around this hash. Here’s a (slightly refactored) look at the relevant code for managing tiddlers, which looks like any other hash wrapper:

javascript

  1. function TiddlyWiki()
  2. {
  3.   var tiddlers = {}; // Hashmap by name of tiddlers
  4.   ...
  5.   this.clear = function() { tiddlers = {}; };
  6.   this.fetchTiddler = function(title) { return tiddlers[title]; };
  7.   this.deleteTiddler = function(title) { delete tiddlers[title]; };
  8.   this.addTiddler = function(tiddler) { tiddlers[tiddler.title] = tiddler; };
  9. }

There is also a set of similar methods which wrap around these to provide more intelligent behaviour. e.g. createTiddler() wraps addTiddler() to provide “Add or retrieve if exists” functionality. getTiddler() wraps fetchTiddler() to ensure null is returned if no such tiddler exists. removeTiddler() wraps deleteTiddler() to delete only if the tiddler exists, and also notifies the tiddler’s listeners. Most other methods also do “general stuff” with the tiddlers hash. A lot of them also run operations on behalf of Tiddlers themselves (this is mostly so it can endow shadow tiddlers – which are just strings – with certain behaviour, as mentioned in the previous section.)

Story

Story is the sovereign UI element in TiddlyWiki – its the container of all visible Tiddlers which you’ll usually see occupying the main, middle, column. Theoretically, there could be more than one Story instance on the page, but I’m told that there are some hard coding shenanigans that rule it out in the project’s current state. (Specifically, direct references to the “story” instance that main.js creates.) So Story is a singleton class in practice.

One gotcha here with the nomenclature – a “tiddler” inside Story.js is conceptually a DOM element, whereas in most other places its a data structure. Obviously, the tiddler UI element is a rendering of the tiddler data structure. However, the implementation isn’t entirely symmetrical because the data structure has a dedicated class (Tiddler), while the UI element doesn’t; tiddler rendering is handled purely by the Story class. In one case (displayTiddler()), either form is valid as the “tiddler” argument, similar to $() functions that accept either the element or the ID (title = (tiddler instanceof Tiddler) ? tiddler.title : tiddler.)

Story’s key properties are a container ID, which points to the container DOM element, and an idPrefix, the prefix for all tiddler IDs. The container already exists on the page when a Story object is created to manage it.

javascript

  1. function Story(containerId,idPrefix)
  2. {
  3.   this.container = containerId;
  4.   this.idPrefix = idPrefix;
  5.   ...
  6. }

Each tiddler’s ID is simply idPrefix + title. You might expect an array of tiddler DOM elements, but Story doesn’t need it, as it can use the DOM itself to keep track of them; the direct descendents of the Story container are the Tiddler elements. It simply uses DOM traversal techniques to iterate through all such elements, when it needs to. (There’s a generic forEachTiddler function too; I could imagine there might be some value in other collection closure methods.)

Story contains the logic to display a tiddler. displayTiddler() decides if the tiddler is already being shown, and if not, creates a new child element with the tiddler content. It delegates to the animation engine for display.

There is also refreshTiddler() – the logic for rendering the tiddler – which is called from displayTiddler(). For flexibility, tiddlers are rendered using a template, a template which is generally contained in a shadow tiddler. There’s a ViewTemplate shadow tiddler and an EditTemplate shadow tiddler – it depends on whether the tiddler is being edited.

Furthermore, there is the concept of themes, which means you can use different templates. This is handled by switchTheme(). An example of different templates is illustrated here in the TiddlyPedia theme.


And that concludes the three-part series. Thanks again to those who helped me gather this info (see credits in first article). I’ve learned a lot about Tiddlywiki in writing it, but I still have a long way to go. There wil be more.

Tiddlywiki internals 2 of 3: List of Javascript Files

(This is part 2 of a 3-part series. Part 1 introduces the internals and highlights some of the key patterns and concepts. Part 2 introduces each Javascript file. Part 3 focuses on the internals of the more important classes and files.)

Continuing the series, below is a list of all core Javascript files, organised into functional groups.

Initialisation

  • main.js Runs the initialisation sequence.
  • Paramifiers.js Handles URL params.

Generic (Non-Animation)

  • BasicTypes.js Augments built-in Javascript Number and Array.
  • Crypto.js Crypto functions. (Tiddlers can generate fingerprints.)
  • Dates.js Augments built-in Javascript Date class.
  • Dom.js Supports DOM manipulation.
  • FileSystem.js
  • Strings.js Augments built-in Javascript Number and Array.
  • Http.js Supports XmlHttpRequest based remoting.
  • RGB.js CSS colour manipulation.

Generic (Specifically Animation)

See also (2005) TiddlyWiki animation write-up.

  • Animator.js Runs the dynamic flow of stepping through an animation, delegating to specific strategies.
  • Morpher.js Morphing animation strategy. Cool – smoothly animates between two CSS styles.
  • Scroller.js Scroller animation strategy. Scrolls window to show an element. (The way the page smoothly scrolls to show a tiddler when you click its link).
  • Slider.js Slider animation strategy. Slides elements opening and closed (e.g. Closing tiddlers or the Options box on right sidebar.).
  • Zoomer.js Zoomer animation strategy (the way a tiddler “jumps out” from its link).

Tiddlywiki-Specific Utilities

  • FormatterHelpers.js Utilities specifically for Formatters.
  • Refresh.js Mechanism for notifying and updating elements based on changes, e.g. if stylesheet shadow tiddler is updated.
  • Utilities.js Miscellaneous TiddlyWiki-specific utility functions.

Data Structures

  • Tiddler.js Data structure representing a tiddler, i.e. a block of text with a title.
  • TiddlerFields.js Augments TiddlyWiki to manage tiddler fields.
  • TiddlyWiki.js Data structure representing a collection of tiddlers.

Data Import/Export

  • AdaptorBase.js Adaptors convert from various wiki formats (e.g. Mediawiki) to TiddlyWiki. This is the base class for Adaptors.
  • FileAdaptor.js Subclass of AdaptorBase which reads the default/standard Tiddlywiki format.
  • Import.js Macro to import tiddlers from another Tiddlywiki.
  • LoaderSaver.js Converts between HTML and a list of tiddlers. (I think the main purpose is to get a clean HTML list of tiddlers.)
  • Saving.js Saves the Tiddlywiki – main case is serialising everything to DOM elements and saving to local file system.
  • SavingRSS.js Serves Tiddlywiki as RSS format (e.g. TiddlyWiki.com RSS feed) showing time-sorted list of recently updated tiddlers.
  • Sync.js Syncs
  • TW21Loader.js Standard implementation of LoaderBase (defined in LoaderSaver.js).
  • TW21Saver.js Standard implementation of SaverBase (defined in LoaderSaver.js).

Strategies

This is a broad category of options and control-type functions. The control-type functions are here because they are designed using flexible mechanisms which make them easily overrideable by plugin developers.

  • Config.js General Tiddlywiki config – controls capacities, names of shadow tiddlers, which options can be set, other stuff.
  • Commands.js Handlers for menus and toolbar.
  • Macros.js Defines built-in macros.
  • Formatter.js Formatters are strategies for locating regexp patterns in the wiki text (wiki words, image URLs, etc.) and rendering them.
  • Options.js Options are cookie-based preferences. The user can generally set them directly on the Tiddlywiki UI. This is in contrast to Config.js settings, which are fixed unless the uswer cares to dive into the source code.
  • Wikifier.js

UI Elements

  • Backstage.js The backstage space at the top of the page, with access to advanced features and acting as an escape route after over-enthusiastic bouts of customisation.
  • ListView.js A table-like list, e.g. shows all options when you hit Backstage|Tweak.
  • Manager.js Plugin manager (accessible from Backstage|Plugins)
  • Messages.js Simple status notifications.
  • NewTiddler.js Macro for a new tiddler, e.g. when user hits “New Tiddler” menu option, and also new journal.
  • Popup.js Popup menu (e.g. when you click on the name of a tiddler in the list of shadow tiddlers).
  • Search.js Search implementation – allows user to search for a term.
  • Sparkline.js Generates CSS based sparklines graphic.
  • Story.js Manages the container of all visible tiddler UI elements.
  • Tabs.js A UI element for handling tabs.
  • Toolbar.js The toolbar shown in the top of a tiddler (with “close”, “close others” etc controls – or “done”-“cancel”-“delete” if open).
  • Wizard.js Multi-step wizard UI framework.

Miscellaneous

  • Deprecated.js Deprecated functions.
  • Guide.js A short readme file.
  • Lingo.js internationalisation-localisation support – contains string keys and their English values.
  • Upgrade.js Support for upgrading Tiddlywiki vgersion.
  • Version.js Short file with info about this version of Tiddlywiki.

Tiddlywiki internals 1 of 3: Architectural Concepts

(This is part 1 of a 3-part series. Part 1 introduces the internals and highlights some of the key patterns and concepts. Part 2 introduces each Javascript file. Part 3 focuses on the internals of the more important classes and files.)

This is the first in a 3-part series on the internal design of Tiddlywiki. The series is more or less stream of consciousness – I’m a green Tiddlywiki developer, so I’ve been making these notes as I trawl through the source and learn it myself. Thanks to various people at Osmosoft for explaining some of this, and special thanks to Jeremy for some overviews and reviewing the writing here, Saq for a great overview on many feature, and Fred for reviewing the initially published version.

Overview

A Tiddlywiki is a collection of “tiddlers”, small blocks of content typically a paragaph or so in length. At any time, a subset of these tiddlers is displayed in the UI (between zero and the total number of stored tiddlers).

A special property of Tiddlywiki is that the entire application resides in a single HTML/Javascript/CSS file (it’s the quintessential SPA – Single-Page Application). This is why you can save a Tiddlywiki locally and run it off a file:// URL and stick it on your iPod or novelty hamburger USB stick.

In the file, all the tiddlers are stored inside invisible DIVs, which are read on startup into a “TiddlyWiki” data structure. When you invoke the save feature, for example by hitting the “save changes” control, the invisible DIVs are refreshed with latest content from memory, and the entire file is written out to the hard drive.

TiddlyWiki is much more than a specialised wiki – due to its flexible architecture and the possibility of plugins, it is more like a platform. Examples of apps built on Tiddlywiki.

TiddlyWeb, though not discussed specifically here, marks an important step in the future of TiddlyWiki development. It’s a RESTful server of Tiddlers which would allow for great flexibility in the kinds of UIs you end up with, as well as allowing non-UI clients.

Anatomy of a Tiddlywiki

The image below shows an Tiddlywiki in editable mode. As for the UI, you can see it consists of a main area with two sidebars. The main area is a “Story” – a story is a sequence of visible tiddlers.

A lot of this is configurable by changing special tiddlers. In particular, the tiddler called “PageTemplate” provides the overall structure, with references to other tiddlers, and “Stylesheet” the CSS styles.

Object-Oriented Concepts in Tiddlywiki

There are many ways to deal with classes, objects, and prototypes in Javascript – see “Javascript: The Good Parts” by Doug Crockford and “Pro Javascript Design Patterns” by Ross Harmes and Dustin Diaz.

Tiddlywiki’s OO relies on the constructor function pattern, where you create new objects using the new keyword.

javascript

  1. var tiddler = new Tiddler();

In Javascript, new Fn() will magically does a couple of things that let us use the familiar (from C++, Java, etc.) idiom above. It sparks the creation of a blank object, then it conducts a special execution of Fn() in which this is superfrajalistically tied to the new-born object. This leads us to an idiom which is called a “constructor function” because it is a function that is both called and implemented as if it were, for the most part, a constructor in OO languages like C++ and Java. The Tiddler constructor function is defined as follows:

javascript

  1. function Tiddler(title)
  2.   {  
  3.     this.title = title;
  4.     this.text = "";
  5.     ...
  6.     return this;
  7.   }

In addition, the new Tiddler has a number of standard Tiddler methods associated with it, so I can call them in typical OO fashion, e.g. tiddler.isTagged("happy"). The implementations refer to the specific instance using the this keyword. In Javascript, this can easily be achieved via prototypes. Therefore, subsequent to the constructor definition, we encounter in Tiddler.js a menagerie of method definitions like:

javascript

  1. Tiddler.prototype.isTagged = function(tag)
  2. {
  3.   return this.tags.indexOf(tag) != -1;
  4. }

All of the attributes above are public, but Tiddlywiki also uses closures to ensure some attributes are only available externally via declared methods. For example, the tiddlers of a Tiddlywiki is a declared as a local variable, so there’s no direct reference to it outside the methods declared in the same scope.

javascript

  1. function TiddlyWiki()
  2. {
  3.   var tiddlers = {}; // Hashmap by name of tiddlers
  4.   this.tiddlersUpdated = false;
  5.   ...
  6.   this.fetchTiddler = function(title) {
  7.     var t = tiddlers[title];
  8.     return t instanceof Tiddler ? t : null;
  9.   };
  10. }

The above methods will also be available on each instance created with new, just as with those declared using the prototype assignment. They are used in exactly the same way. The only difference is that all these functions are re-created with each new instance, so they will consume more memory. That’s the price we pay for the encapsulation.

You will also find static methods present (i.e. global functions attached to a constructor purely for the sake of namespacing them). For example:

javascript

  1. TiddlyWiki.isStandardField = function(name)
  2. {
  3.   return TiddlyWiki.standardFieldAccess[name] != undefined;
  4. }

Typically, a class will be contained in a single, dedicated, Javascript file (within the source code from which a Tiddlywiki is built). However, the previous example was actually contained in TiddlerFields.js rather than Tiddlywiki.js, so it seems that class definitions may be distributed across multiple files in some limited cases.

And that’s how Tiddlywiki handles basic OO.

You’ll also see some parts of TiddlyWiki enhancing built-in Javascript types by extending their prototype – for example, BasicTypes.js endows all Arrays with a contains() method and Dates.js sticks a getAmPm() method onto each Date that’s created. Number, Array, and Date receive a dozen or so new methods.

Last but not least, there’s also a healthy dose of inheritance in Tiddlywiki. Javascript inheritance is a whole new can of worms. We see an example in AdaptorBase, which serves as the base class for server adaptor subclasses. AdaptorBase looks very normal, like Tiddler above. FileAdaptor, a subclass, looks like this:

javascript

  1. function FileAdaptor() {
  2. }
  3.  
  4. FileAdaptor.prototype = new AdaptorBase();

Basically, Javascript has a concept of prototype chains. The assignment means that any instance of FileAdaptor will now have all methods present in a new instance of AdaptorBase. FileAdaptor goes on to define its own methods, using the standard prototype pattern. If so inclined, it can override AdaptorBase’s methods by defining them on its own prototype method. (This is why we say “new AdaptorBase()” – if we had assigned FileAdaptor.prototype to AdaptorBase.prototype, anything we set on FileAdaptor would also be set on AdaptorBase.)

URL Arguments

Tiddlywiki uses the fragment identifier pattern (described here) to provide flexible loading strategies.

Normally, the “DefaultTiddlers” shadow tiddler is used to specify which tiddlers are shown on startup. However, this can be overridden via URL params. For example, use http://www.tiddlywiki.com/#Examples to load with just the Examples tiddler showing. Or, for multiple tiddlers, just separate with a space (%20 in URL-5peak) http://www.tiddlywiki.com/#Examples%20Plugins. (An interesting possibility would be for Tiddlywiki to keep updating the URL to ensure its sync’d with the state of the app, so you could bookmark it at any time to save that configuration.)

But maybe you don’t want to manually list all the tiddlers – instead, you might want to show all tiddlers matching some criteria. Then you’d want an automated mechanism for auto-selecting those criteria (think iTunes Smart Playlist for dramatic effect.) This would make the URL shorter, easier to understand the true purpose of the configuation, and future-proof it against any changes to the set of tiddlers we’re interested in.

In Tiddlywiki, that mechanism is achieved with a URL “filter” prefix. For example, show all tiddlers with “systemConfig” tag – http://tiddlywiki.com/#filter:[tag[systemConfig]].

Other things you can do – http://tiddlywiki.com/#newTiddler:tiddlername – create a new tiddler, specifying the name

The URL is modelled as a map, i.e. key-value pairs. In the case of http://www.tiddlywiki.com/#Examples%20Plugins, that’s just an alias for the canonical map form, http://www.tiddlywiki.com/#open:Examples%20open:Plugins. All this is managed by the Paramifiers class.

Shout – First TiddlyWiki Plugin

Okay, it’s remarkably lame, but here’s my first TiddlyWiki plugin. It defines a new shout macro – you include <<shout some-message>> in a tiddler and it includes A SHOUTING VERSION OF THE MESSAGE.

I adapted it from the sparklines plugin. The important code is at the end – you create a macro by defining a value config.macros.helloWorld, then you simply implement config.macros.helloWorld.handler. This function receives a place variable, defining where the macro will go, and the parameters that were passed into the macro (params).

javascript

  1. /***
  2. |''Name''|ShoutPlugin|
  3. |''Description''|Make yourself heard - capitalises all text|
  4. |''Version''|1.0.0|
  5. |''Status''|stable|
  6. |''Source''|http://www.example.com/plugins.html#Shout|
  7. |''License''|[[BSD open source license]]|
  8. |''~CoreVersion''|2.4.0|
  9. |''Feedback''|[[TiddlyWiki community|http://groups.google.com/group/TiddlyWiki]] |
  10. |''Keywords''|exclamation|
  11. !Usage
  12. {{{
  13. <<shout message-to-be-shouted-about>>
  14. }}}
  15. Pretty simple really
  16. !!Examples
  17. {{{<<shout these tiddlers are making me thirsty!!!>>}}}
  18. !Code
  19. ***/
  20. //{{{
  21. if(!version.extensions.ShoutPlugin) {
  22. version.extensions.ShoutPlugin = {installed:true};
  23.  
  24. //--
  25. //-- Shout
  26. //--
  27.  
  28. config.macros.shout = {};
  29. config.macros.shout.handler = function(place,macroName,params)
  30. {
  31.  var greeting = createTiddlyElement(place,"span",null,"greeting",params.join(" ").toUpperCase());
  32.   place.appendChild(greeting);
  33. };
  34.  
  35.  
  36. }
  37. //}}}

Injecting HTML into an IFrame

Walking through Tiddlywiki source (write-up to follow), I noticed some interesting code in TiddlyWiki.js, importTiddlyWiki function.

The code takes a string and injects into an IFrame. I had talked to Jon a little while ago about a similar problem and was wondering about it ever since. The technique here looks like this:

It wraps the text with tags to make it an HTML document:

javascript

  1. var content = "<html><body>" + text + "</body></html>";

It then introduces a new iframe to the page.

javascript

  1. var iframe = document.createElement("iframe");
  2.     document.body.appendChild(iframe);

Now comes the tricky part. You would think you could just create a new iframe element and set its innerHTML, but with iframes you must use an internal doc ument property.

javascript

  1. var doc = iframe.document;
  2.     if(iframe.contentDocument)
  3.         doc = iframe.contentDocument; // For NS6
  4.     else if(iframe.contentWindow)
  5.         doc = iframe.contentWindow.document; // For IE5.5 and IE6
  6.     // Put the content in the iframe
  7.     doc.open();
  8.     doc.writeln(content);
  9.     doc.close();

Now the content is in our new iframe. We can then manipulate the content using standard Javascript…but ensuring a call like getElementById is executed against the iframe document, not the global document. (i.e. don’t use the usual document.getElementById()).

javascript

  1. // Load the content into a TiddlyWiki() object
  2.     var storeArea = doc.getElementById("storeArea");
  3. };

With this technique, you can take an arbitrary HTML string and delegate its parsing to the browser’s built-in DOM engine.