Osmosoft Hackathon: WikiData, a Wiki of Companies Data

End of The WikiData Hackday

Osmosoft, Hackathons

At Osmosoft, we have been engaging in a one-day hackathon about every month or so. There are several benefits:

  • It helps us prove our tech in a range of application contexts.
  • It helps improve and demonstrate our capabilities for reuse. Reuse can be as simple as slapping a Javascript file with inline comments on a server somewhere, or exposing a simple web service. With those capabilities in place, it’s possible to build something useful in a day, just as industry hackathons/camps demonstrate.
  • It helps us spread the word about web standards and why everything needs its own URI.
  • It helps us demonstrate and gain experience in agile processes.
  • It gets us Osmosofties working together in the same place at the same time, whereas day-to-day we tend to work on products in small groups or individually.

The WikiData Hackathon

The latest hackathon was last Thursday, July 16, which saw us collaborate with Avox, a company specialising in business entity data. I believe it’s working in a similar space to companies like Dunn and Bradstreet and other information providers, in that they collect, verify, and report information about companies. Avox is keen to open up parts of their database and gain the benefits of community feedback – with users being able to contribute information, leave comments, add new companies, and so on. Of course, Avox is in the business of gathering verified data, so it’s not just a case of making a new wiki and letting it run by itself. There remain challenges about how Avox will merge and verify community input, and how they will make it clear to users what’s verified and what’s not.

Pre-Work

We had a conversation on the mailing list about what we did last time and what could do differently for this hackathon. Like a retrospective, but over email. TiddlyWeb architect Chris Dent set up some instances with sample data taken from the existing product.

Venue and Attendance

The hackathon took place in Osmosoft’s office, centered around our big table in the centre of the room. Seven people from Avox, in a range of technical and non-technical roles, attended for the duration of the event. Osmosoft had five developers working on stories, a developer working on integration and mediation, and another helping with the server and offering second-line support.

Introductions and Overview (about 1 hour)

We went round the table and everyone introduced themselves. Jeremy Ruston explained “what’s in it” for Osmosoft, as outlined above, and Ken Price outlined Avox’s interest in building wikidata. We also looked at the existing product. We then had a discussion which was mostly Osmosoft people asking Avox people questions about the nature of their business, the technologies involves, and their vision for wikidata. Paul began writing down stories on flashcards during this time.

User Stories (about 1 hour)

We finished writing stories – this was somewhat in parallel to the previous activity – and put them all up on the wall, with the magic of Blu-Tac. The stories were in the form “As a (role), I want to (task), so that I can (benefit)”. With about 20 stories, it was useful to organise them, so we grouped them according to the role. The main roles were for Avox staff and general users. There were also some stories involving 3rd party developers and employees of companies listed in the wiki.

Everyone gathered around the stories, we read them out, and we all agreed on priorities. We didn’t worry too much about ordering the stories at the bottom as it was unlikely we’d get to them during the event; if we did, we could prioritise later.

What we ended up with was a solid idea of the steel thread we were building. It would mostly be standard wiki functionality, but applied to the particular data and context of companies info. We had some bonus stories we could implement if we had time, like comments and tagging.

Planning and Infrastructure(about 30 minutes)

Developers put their initials against the first story they’d be working on, and likewise each story needed a customer representative to initial the story, so they would help refine requirements. In the event, we didn’t talk all that much with customers during development; it’s obviously an extremely important thing to do in a real project, but when you’re wanting to get real functionality out in a day, the cost of a conversation is relatively high and the benefit to the task is relatively low. It would have been different in a 2-3 day event, or with certain stories that are highly specific to the domain. The main thing was to checkpoint with customers to check that we’re generally on track.

Around noon, we drew up a timeline and agreed on a hard stop at 7pm. This means we timebox – the time is fixed and the only variable is the number of stories that get implemented in that time. We then got into

Development Sprints and Standup Meetings (about 6.5 hours)

We developed in sprints, with standup meetings in between. It was felt the previous hackathon’s hourly sprints were too frequent, so in this case the plan was every 1.5-2 hours; we decided at the end of each standup when the next would be.

Most of us developed against the user stories. We also had a very useful role fulfilled by Fred, who was designated as a general “helper” and mediator. This was the result of our feeling that things were sometimes disjointed in previous hackathons – not enough focus on integration and infrastructure. I feel that this role was very useful and Fred fulfilled it admirably, although at times he probably felt like he had 400 people talking to him at once! We also had Chris Dent working remotely to ensure the TiddlyWeb server was running and help deploy to the live site.

The tool we developed was a standard Javascript/JQuery web app (ie nothing to do with TiddlyWiki) talking to the TiddlyWeb server.

At the start, we intended to write the web app and talk to the live server. But it soon became apparent that we would step on each others’ toes by doing this, and opted for a more conventional setup where we each have our own server instance. We also had a quick debate about version control – github, private osmosoft repository, or tiddlywiki repository. This is a recurring debate which we ought to decide in advance next time. It partly arose after deciding to run our local copies of the server; it was felt this would be too much data for the tiddlywiki repo, so we used the private osmo svn. As for github, it would be nice to use git, but most of us are more familiar with SVN as we use the tiddlywiki SVN repo day-to-day, so it would cause too much complication to switch to git. Again, it might be a good idea for next time to use github instead, with some pre-work.

Presentation (about 45 minutes)

After the usual last-minute rush, we got a working product deployed. It must be noted that we let the “hard stop” slip by about 45 minutes, to about 7:30. Admittedly a bad practice, but it did yield some major results in this case, as it got us search, edit, and maps all working during that time.

Each of the developers explained what they’d worked on. We then gathered round the screen and walked through the app. Avox gave their thoughts, I recorded the video above while others broke out into informal conversations, and by that point, it was pub o’clock.

We were able to produce the steel thread we’d been planning; the key stories were demonstrated. We also implemented commenting and Google Maps integration. Being based on TiddlyWeb means we had also produced a working API “for free”; it’s just the nature of TiddlyWeb as a RESTful data container. (On a personal note, commenting was the main thing I did – I extracted the comments feature from Scrumptious into a standalone plugin, and integrated it into the emerging WikiData product. I’ll post about that separately.)

Wikidatathon

Wikidatathon

About the Video

I recorded the video above at the end of the event. One of my bugbears about hackathon events is that people spend all day coding and setting up infrastructure, and it inevitably comes down after that, or gradually falls apart as various third-party services close, databases get corrupted, people forget to keep hosting it, etc etc. In other words, you have to assume the online result of any hackathon event is transient. This is unfortunate, because the deliverable should be something that people can use to decide what happens next, whether to fund a project, and so on. While meeting minutes are often a waste of time, the artifacts that emerge in a workshop are critical to followup.

For that reason, I am perfectly passionate about taking a screencast or video to capture what took place in such events. Thanks to Paul and Ken for taking part in it.

Update: Avox CEO Ken Price (who appears in the video), has published a summary of the event.

List of Interesting and Popular MediaWiki Instances

I’m currently at an Osmosoft hackathon event, working on a tiddlywiki vertical called mediawikiunplugged. No time to say more now, but we needed a bunch of mediawiki sites for testing. So in addition to the obWikipedia and obAjaxPatterns, here are more:

And a list of the biggest wikis according to Alexa (retrieved thanks to archive.org).

Tiddlywiki internals 3 of 3: Key Javascript Classes and Files

(This is part 3 of a 3-part series. Part 1 introduces the internals and highlights some of the key patterns and concepts. Part 2 introduces each Javascript file. Part 3 focuses on the internals of the more important classes and files.)

Concluding this series, below is a list of all core Javascript files, organised into functional groups.

main.js

main() is the function that runs onload.

Key functions:

  • creates a new tiddlywiki data store (new TiddlyWiki()) – this is the collection of tiddlers users are exposed to. The store is populated using TiddlyWiki.prototype.loadFromDiv(), which loads all the tiddlers from the “storeArea” div, which is an invisible block on the page (and rendered back in nice – and visible – manner later on).
  • creates a second TiddlyWiki data store to hold “shadow tiddlers” – these are “meta”/config tiddlers holding data such as CSS styling. Populated from invisible “shadowArea” div (which at compile time is defined in the Shadow/ directory).
  • creates a new “Story div”, a div which will show tiddlers to the user, and themes it according to config.options.txtTheme
  • sets up Popup.onDocumentClick (removes popup menus when user clicks outside of the menu)
  • sets up event propagation – certain tiddlers are notified when certain actions occur. The mappings are defined in refresh.js (e.g. {name: “StyleSheetLayout”, notify: refreshStyles})
  • sets up and renders backstage
  • loads plugins (plugins are evidently supposed to set a global “plugin problem” value if a problem occurs)

General:

  • calls several lifecycle event handlers as it loads – the wiki config can provide hook functions which run upon particular lifetime events
  • benchmarks most of the above (the benchmarking was possibly a quick fix – relies on variables t1,t2…t10 -> this code could be optimised for conciseness using function wrappers, but maybe startup would be too slow that way).
  • After initial setup ensures tiddlywiki data structures and other initialisation/config pieces are in place, it blats and shows the display with restart() and refreshDisplay().

Plugins

Tiddlywiki has a strong plugin architecture. Each plugin is included as a regular (non-shadow) tiddler, one that must be tagged “systemConfig”. (For all intents and purposes, “systemConfig” is synonymous with “plugin”.) There’s an example shipping with the default tiddlywiki instance on tiddlywiki.com (and a more detailed example in the source code – association/plugins/SparklinePlugin/SparklinePlugin.js). (Also of interest, the latest plugin template at the tiddlywiki.org wiki.)

  1. <div title="ExamplePlugin" modifier="JeremyRuston" created="200607271914" modified="200609212329" tags="systemConfig">
  2. <pre>/***
  3. |''Name:''|ExamplePlugin|
  4. |''Description:''|To demonstrate how to write TiddlyWiki plugins|
  5. |''Version:''|2.0.3|
  6. |''Date:''|Sep 22, 2006|
  7. |''Source:''|http://www.tiddlywiki.com/#ExamplePlugin|
  8. |''Author:''|JeremyRuston (jeremy (at) osmosoft (dot) com)|
  9. |''License:''|[[BSD open source license]]|
  10. |''~CoreVersion:''|2.1.0|
  11. |''Browser:''|Firefox 1.0.4+; Firefox 1.5; InternetExplorer 6.0|
  12. ***/
  13.  
  14. //{{{
  15.  
  16. // Uncomment the following line to see how the PluginManager deals with errors in plugins
  17. // deliberateError();
  18.  
  19. // Log a message
  20. pluginInfo.log.push(&quot;This is a test message from &quot; + tiddler.title);
  21.  
  22. //}}}</pre>
  23. </div>

A plugin is essentially just a Javascript block which gets executed on page load. All the biosketch info is optional (although in some cases, it does effect processing, e.g. there is a check against the required TiddlyWiki version). “Just some Javascript” did you say? This post on JQuery plugins by JQuery daddy John Resig is instructive. His point is that a plugin architecture needs explicit points for plugins to hook into – i.e. an API – and the existence of a plugin catalogue. Tiddlywiki doesn’t have a plugin API per se, but is structured with plenty of extension points to naturally hook into. As for the catalogue, there’s also a plugin wiki area, with a grander-scale plugin repo project in progress.

Incidentally, note that you don’t have to register the Javascript block as you might do in some other frameworks (e.g. runOnInit(myPlugin); ). It executes automatically when plugins are loaded.

Okay, so about those plugin extension points. I’m still learning that. In the case of sparklines, the purpose is to create a new macro (e.g. <<sparkline 100 200 300>>), so it defines config.macros.sparkline.handler(place,macroName,params), and its “output” is to populate the place element with sparkline content.

Another popular pasttime for plugin developers is szhushing the global Formatter object to shape how stuff gets rendered. e.g. if your formatter locates the built-in formatter named “heading”, it could easily overwrite its handler method to MAKE ALL THE HEADINGS SHOUT AT UNSUSPECTING READERS.

To install a plugin, users usually use the Import dialog, accessible from Backstage. It’s also possible to manually include plugins via cut-and-paste into Tiddlywiki.

There’s much more to be said about plugins. The bottom line is that Tiddlywiki’s architecture lets you bend the core product into many things. (By “architecture”, I refer to both the plugin mechanism and the flexible nature in which the overall architecture is structured.)

Tiddlers

Tiddlers are the atomic content blocks that make up a Tiddlywiki, typically about a paragraph in length. A Tiddler is simply a block of text, with extra info like a title, a list of tags, and timestamp data. There’s also a fields hash where you could store any arbitrary properties. (This seems suitable for plugins, but the core also makes use of it, and I don’t really get that. Even for plugins, why can’t they just make new fields dynamically?)

Tiddler is a Javascript class, so you get a new instance with new Tiddler(). Internally, it uses a publish-subscribe mechanism, where a changed() method is called after any mutation. This basically ensures the links property is up to date, as links is a redundant (and presumably there for performance) collection of links inside the tiddler.

A Tiddler also has a collection of “slices”, though the collection is managed by TiddlyWiki rather than Tiddler. (This relates to the fact that shadow tiddlers are mere text blocks – using Tiddlywiki to extract slices ensures shadow tiddlers can also be sliced up….and slices are a major feature of most shadow tiddlers, since they are config-related.)

There’s a string->string map from name to slice. This is similar to the fields hash, insofar as it’s a free-form map. In this case, though, it’s something that can easily be changed by the user in real time, as the slice collection is sync’d to the tiddler content. For example: |”slicename:”|”some slice content”|. Slices allow for easily edited meta-data, e.g. a stylesheet tiddler can have a slice called “backgroundColour”. Users then edit the backgroundColor slice content to set the background colour.

A Tiddler also has a set of notification handlers – this is also managed by TiddlyWiki rather than the Tiddlers themselves (again, this ensures the mechanism works for shadow tiddlers). These are listeners/observers that are notified each time tiddler is changed.

A file closely related to Tiddler is TiddlerFields.js. It actually alters the TiddlyWiki definition rather than the Tiddler definition, but in any event it deals with accessing the Tiddler’s fields map.

Shadow Tiddlers

Shadow tiddlers are a particular type of tiddler. There’s no separate “ShadowTiddler” class, but they are held in a separate store and treated in special ways. Indeed, shadow tiddlers aren’t actually of class Tiddler (which is slightly confusing). They are simply a title-text pairing; the data structure is a map from title to text. In contrast, regular Tiddlers are mapped from title to Tiddler.

In particular, TiddlyWiki has a fallback mechanism when asked to return a tiddler – if the tiddler doesn’t exist, it will attempt to revert to a shadow tiddler of the same name. Shadow tiddlers are immutable (unless you hack source code), whereas tiddlers are of course easily edited. You can override shasow tiddlers with regular tiddlers of the same name, but the original shadow still lurks (in a good way) in the background.

To see this, open an editable Tiddlywiki, choose a shadow tiddler from the right sidebar Contents menu (e.g. SiteUrl), edit it, and save it. Then re-open it to verify your changes were affected. Then delete it, and notice that it’s still in the list of shadow tiddlers. When you open it yet again, you’ll see it now contains the original content. (The shadow tiddler itself never changed.)

Shadow tiddlers are used for config stuff like stylesheets. The fail-safe mechanism ensures you can easily “restore factory defaults” at any time.

TiddlyWiki

A Tiddlywiki is essentially a hash of Tiddlers, keyed on their title. More precisely, it’s a wrapper around this hash. Here’s a (slightly refactored) look at the relevant code for managing tiddlers, which looks like any other hash wrapper:

javascript
< view plain text >
  1. function TiddlyWiki()
  2. {
  3.   var tiddlers = {}; // Hashmap by name of tiddlers
  4.   ...
  5.   this.clear = function() { tiddlers = {}; };
  6.   this.fetchTiddler = function(title) { return tiddlers[title]; };
  7.   this.deleteTiddler = function(title) { delete tiddlers[title]; };
  8.   this.addTiddler = function(tiddler) { tiddlers[tiddler.title] = tiddler; };
  9. }

There is also a set of similar methods which wrap around these to provide more intelligent behaviour. e.g. createTiddler() wraps addTiddler() to provide “Add or retrieve if exists” functionality. getTiddler() wraps fetchTiddler() to ensure null is returned if no such tiddler exists. removeTiddler() wraps deleteTiddler() to delete only if the tiddler exists, and also notifies the tiddler’s listeners. Most other methods also do “general stuff” with the tiddlers hash. A lot of them also run operations on behalf of Tiddlers themselves (this is mostly so it can endow shadow tiddlers – which are just strings – with certain behaviour, as mentioned in the previous section.)

Story

Story is the sovereign UI element in TiddlyWiki – its the container of all visible Tiddlers which you’ll usually see occupying the main, middle, column. Theoretically, there could be more than one Story instance on the page, but I’m told that there are some hard coding shenanigans that rule it out in the project’s current state. (Specifically, direct references to the “story” instance that main.js creates.) So Story is a singleton class in practice.

One gotcha here with the nomenclature – a “tiddler” inside Story.js is conceptually a DOM element, whereas in most other places its a data structure. Obviously, the tiddler UI element is a rendering of the tiddler data structure. However, the implementation isn’t entirely symmetrical because the data structure has a dedicated class (Tiddler), while the UI element doesn’t; tiddler rendering is handled purely by the Story class. In one case (displayTiddler()), either form is valid as the “tiddler” argument, similar to $() functions that accept either the element or the ID (title = (tiddler instanceof Tiddler) ? tiddler.title : tiddler.)

Story’s key properties are a container ID, which points to the container DOM element, and an idPrefix, the prefix for all tiddler IDs. The container already exists on the page when a Story object is created to manage it.

javascript
< view plain text >
  1. function Story(containerId,idPrefix)
  2. {
  3.   this.container = containerId;
  4.   this.idPrefix = idPrefix;
  5.   ...
  6. }

Each tiddler’s ID is simply idPrefix + title. You might expect an array of tiddler DOM elements, but Story doesn’t need it, as it can use the DOM itself to keep track of them; the direct descendents of the Story container are the Tiddler elements. It simply uses DOM traversal techniques to iterate through all such elements, when it needs to. (There’s a generic forEachTiddler function too; I could imagine there might be some value in other collection closure methods.)

Story contains the logic to display a tiddler. displayTiddler() decides if the tiddler is already being shown, and if not, creates a new child element with the tiddler content. It delegates to the animation engine for display.

There is also refreshTiddler() – the logic for rendering the tiddler – which is called from displayTiddler(). For flexibility, tiddlers are rendered using a template, a template which is generally contained in a shadow tiddler. There’s a ViewTemplate shadow tiddler and an EditTemplate shadow tiddler – it depends on whether the tiddler is being edited.

Furthermore, there is the concept of themes, which means you can use different templates. This is handled by switchTheme(). An example of different templates is illustrated here in the TiddlyPedia theme.


And that concludes the three-part series. Thanks again to those who helped me gather this info (see credits in first article). I’ve learned a lot about Tiddlywiki in writing it, but I still have a long way to go. There wil be more.

Tiddlywiki internals 2 of 3: List of Javascript Files

(This is part 2 of a 3-part series. Part 1 introduces the internals and highlights some of the key patterns and concepts. Part 2 introduces each Javascript file. Part 3 focuses on the internals of the more important classes and files.)

Continuing the series, below is a list of all core Javascript files, organised into functional groups.

Initialisation

  • main.js Runs the initialisation sequence.
  • Paramifiers.js Handles URL params.

Generic (Non-Animation)

  • BasicTypes.js Augments built-in Javascript Number and Array.
  • Crypto.js Crypto functions. (Tiddlers can generate fingerprints.)
  • Dates.js Augments built-in Javascript Date class.
  • Dom.js Supports DOM manipulation.
  • FileSystem.js
  • Strings.js Augments built-in Javascript Number and Array.
  • Http.js Supports XmlHttpRequest based remoting.
  • RGB.js CSS colour manipulation.

Generic (Specifically Animation)

See also (2005) TiddlyWiki animation write-up.

  • Animator.js Runs the dynamic flow of stepping through an animation, delegating to specific strategies.
  • Morpher.js Morphing animation strategy. Cool – smoothly animates between two CSS styles.
  • Scroller.js Scroller animation strategy. Scrolls window to show an element. (The way the page smoothly scrolls to show a tiddler when you click its link).
  • Slider.js Slider animation strategy. Slides elements opening and closed (e.g. Closing tiddlers or the Options box on right sidebar.).
  • Zoomer.js Zoomer animation strategy (the way a tiddler “jumps out” from its link).

Tiddlywiki-Specific Utilities

  • FormatterHelpers.js Utilities specifically for Formatters.
  • Refresh.js Mechanism for notifying and updating elements based on changes, e.g. if stylesheet shadow tiddler is updated.
  • Utilities.js Miscellaneous TiddlyWiki-specific utility functions.

Data Structures

  • Tiddler.js Data structure representing a tiddler, i.e. a block of text with a title.
  • TiddlerFields.js Augments TiddlyWiki to manage tiddler fields.
  • TiddlyWiki.js Data structure representing a collection of tiddlers.

Data Import/Export

  • AdaptorBase.js Adaptors convert from various wiki formats (e.g. Mediawiki) to TiddlyWiki. This is the base class for Adaptors.
  • FileAdaptor.js Subclass of AdaptorBase which reads the default/standard Tiddlywiki format.
  • Import.js Macro to import tiddlers from another Tiddlywiki.
  • LoaderSaver.js Converts between HTML and a list of tiddlers. (I think the main purpose is to get a clean HTML list of tiddlers.)
  • Saving.js Saves the Tiddlywiki – main case is serialising everything to DOM elements and saving to local file system.
  • SavingRSS.js Serves Tiddlywiki as RSS format (e.g. TiddlyWiki.com RSS feed) showing time-sorted list of recently updated tiddlers.
  • Sync.js Syncs
  • TW21Loader.js Standard implementation of LoaderBase (defined in LoaderSaver.js).
  • TW21Saver.js Standard implementation of SaverBase (defined in LoaderSaver.js).

Strategies

This is a broad category of options and control-type functions. The control-type functions are here because they are designed using flexible mechanisms which make them easily overrideable by plugin developers.

  • Config.js General Tiddlywiki config – controls capacities, names of shadow tiddlers, which options can be set, other stuff.
  • Commands.js Handlers for menus and toolbar.
  • Macros.js Defines built-in macros.
  • Formatter.js Formatters are strategies for locating regexp patterns in the wiki text (wiki words, image URLs, etc.) and rendering them.
  • Options.js Options are cookie-based preferences. The user can generally set them directly on the Tiddlywiki UI. This is in contrast to Config.js settings, which are fixed unless the uswer cares to dive into the source code.
  • Wikifier.js

UI Elements

  • Backstage.js The backstage space at the top of the page, with access to advanced features and acting as an escape route after over-enthusiastic bouts of customisation.
  • ListView.js A table-like list, e.g. shows all options when you hit Backstage|Tweak.
  • Manager.js Plugin manager (accessible from Backstage|Plugins)
  • Messages.js Simple status notifications.
  • NewTiddler.js Macro for a new tiddler, e.g. when user hits “New Tiddler” menu option, and also new journal.
  • Popup.js Popup menu (e.g. when you click on the name of a tiddler in the list of shadow tiddlers).
  • Search.js Search implementation – allows user to search for a term.
  • Sparkline.js Generates CSS based sparklines graphic.
  • Story.js Manages the container of all visible tiddler UI elements.
  • Tabs.js A UI element for handling tabs.
  • Toolbar.js The toolbar shown in the top of a tiddler (with “close”, “close others” etc controls – or “done”-”cancel”-”delete” if open).
  • Wizard.js Multi-step wizard UI framework.

Miscellaneous

  • Deprecated.js Deprecated functions.
  • Guide.js A short readme file.
  • Lingo.js internationalisation-localisation support – contains string keys and their English values.
  • Upgrade.js Support for upgrading Tiddlywiki vgersion.
  • Version.js Short file with info about this version of Tiddlywiki.

Tiddlywiki internals 1 of 3: Architectural Concepts

(This is part 1 of a 3-part series. Part 1 introduces the internals and highlights some of the key patterns and concepts. Part 2 introduces each Javascript file. Part 3 focuses on the internals of the more important classes and files.)

This is the first in a 3-part series on the internal design of Tiddlywiki. The series is more or less stream of consciousness – I’m a green Tiddlywiki developer, so I’ve been making these notes as I trawl through the source and learn it myself. Thanks to various people at Osmosoft for explaining some of this, and special thanks to Jeremy for some overviews and reviewing the writing here, Saq for a great overview on many feature, and Fred for reviewing the initially published version.

Overview

A Tiddlywiki is a collection of “tiddlers”, small blocks of content typically a paragaph or so in length. At any time, a subset of these tiddlers is displayed in the UI (between zero and the total number of stored tiddlers).

A special property of Tiddlywiki is that the entire application resides in a single HTML/Javascript/CSS file (it’s the quintessential SPA – Single-Page Application). This is why you can save a Tiddlywiki locally and run it off a file:// URL and stick it on your iPod or novelty hamburger USB stick.

In the file, all the tiddlers are stored inside invisible DIVs, which are read on startup into a “TiddlyWiki” data structure. When you invoke the save feature, for example by hitting the “save changes” control, the invisible DIVs are refreshed with latest content from memory, and the entire file is written out to the hard drive.

TiddlyWiki is much more than a specialised wiki – due to its flexible architecture and the possibility of plugins, it is more like a platform. Examples of apps built on Tiddlywiki.

TiddlyWeb, though not discussed specifically here, marks an important step in the future of TiddlyWiki development. It’s a RESTful server of Tiddlers which would allow for great flexibility in the kinds of UIs you end up with, as well as allowing non-UI clients.

Anatomy of a Tiddlywiki

The image below shows an Tiddlywiki in editable mode. As for the UI, you can see it consists of a main area with two sidebars. The main area is a “Story” – a story is a sequence of visible tiddlers.

A lot of this is configurable by changing special tiddlers. In particular, the tiddler called “PageTemplate” provides the overall structure, with references to other tiddlers, and “Stylesheet” the CSS styles.

Object-Oriented Concepts in Tiddlywiki

There are many ways to deal with classes, objects, and prototypes in Javascript – see “Javascript: The Good Parts” by Doug Crockford and “Pro Javascript Design Patterns” by Ross Harmes and Dustin Diaz.

Tiddlywiki’s OO relies on the constructor function pattern, where you create new objects using the new keyword.

javascript
< view plain text >
  1. var tiddler = new Tiddler();

In Javascript, new Fn() will magically does a couple of things that let us use the familiar (from C++, Java, etc.) idiom above. It sparks the creation of a blank object, then it conducts a special execution of Fn() in which this is superfrajalistically tied to the new-born object. This leads us to an idiom which is called a “constructor function” because it is a function that is both called and implemented as if it were, for the most part, a constructor in OO languages like C++ and Java. The Tiddler constructor function is defined as follows:

javascript
< view plain text >
  1. function Tiddler(title)
  2.   {  
  3.     this.title = title;
  4.     this.text = "";
  5.     ...
  6.     return this;
  7.   }

In addition, the new Tiddler has a number of standard Tiddler methods associated with it, so I can call them in typical OO fashion, e.g. tiddler.isTagged("happy"). The implementations refer to the specific instance using the this keyword. In Javascript, this can easily be achieved via prototypes. Therefore, subsequent to the constructor definition, we encounter in Tiddler.js a menagerie of method definitions like:

javascript
< view plain text >
  1. Tiddler.prototype.isTagged = function(tag)
  2. {
  3.   return this.tags.indexOf(tag) != -1;
  4. }

All of the attributes above are public, but Tiddlywiki also uses closures to ensure some attributes are only available externally via declared methods. For example, the tiddlers of a Tiddlywiki is a declared as a local variable, so there’s no direct reference to it outside the methods declared in the same scope.

javascript
< view plain text >
  1. function TiddlyWiki()
  2. {
  3.   var tiddlers = {}; // Hashmap by name of tiddlers
  4.   this.tiddlersUpdated = false;
  5.   ...
  6.   this.fetchTiddler = function(title) {
  7.     var t = tiddlers[title];
  8.     return t instanceof Tiddler ? t : null;
  9.   };
  10. }

The above methods will also be available on each instance created with new, just as with those declared using the prototype assignment. They are used in exactly the same way. The only difference is that all these functions are re-created with each new instance, so they will consume more memory. That’s the price we pay for the encapsulation.

You will also find static methods present (i.e. global functions attached to a constructor purely for the sake of namespacing them). For example:

javascript
< view plain text >
  1. TiddlyWiki.isStandardField = function(name)
  2. {
  3.   return TiddlyWiki.standardFieldAccess[name] != undefined;
  4. }

Typically, a class will be contained in a single, dedicated, Javascript file (within the source code from which a Tiddlywiki is built). However, the previous example was actually contained in TiddlerFields.js rather than Tiddlywiki.js, so it seems that class definitions may be distributed across multiple files in some limited cases.

And that’s how Tiddlywiki handles basic OO.

You’ll also see some parts of TiddlyWiki enhancing built-in Javascript types by extending their prototype – for example, BasicTypes.js endows all Arrays with a contains() method and Dates.js sticks a getAmPm() method onto each Date that’s created. Number, Array, and Date receive a dozen or so new methods.

Last but not least, there’s also a healthy dose of inheritance in Tiddlywiki. Javascript inheritance is a whole new can of worms. We see an example in AdaptorBase, which serves as the base class for server adaptor subclasses. AdaptorBase looks very normal, like Tiddler above. FileAdaptor, a subclass, looks like this:

javascript
< view plain text >
  1. function FileAdaptor() {
  2. }
  3.  
  4. FileAdaptor.prototype = new AdaptorBase();

Basically, Javascript has a concept of prototype chains. The assignment means that any instance of FileAdaptor will now have all methods present in a new instance of AdaptorBase. FileAdaptor goes on to define its own methods, using the standard prototype pattern. If so inclined, it can override AdaptorBase’s methods by defining them on its own prototype method. (This is why we say “new AdaptorBase()” – if we had assigned FileAdaptor.prototype to AdaptorBase.prototype, anything we set on FileAdaptor would also be set on AdaptorBase.)

URL Arguments

Tiddlywiki uses the fragment identifier pattern (described here) to provide flexible loading strategies.

Normally, the “DefaultTiddlers” shadow tiddler is used to specify which tiddlers are shown on startup. However, this can be overridden via URL params. For example, use http://www.tiddlywiki.com/#Examples to load with just the Examples tiddler showing. Or, for multiple tiddlers, just separate with a space (%20 in URL-5peak) http://www.tiddlywiki.com/#Examples%20Plugins. (An interesting possibility would be for Tiddlywiki to keep updating the URL to ensure its sync’d with the state of the app, so you could bookmark it at any time to save that configuration.)

But maybe you don’t want to manually list all the tiddlers – instead, you might want to show all tiddlers matching some criteria. Then you’d want an automated mechanism for auto-selecting those criteria (think iTunes Smart Playlist for dramatic effect.) This would make the URL shorter, easier to understand the true purpose of the configuation, and future-proof it against any changes to the set of tiddlers we’re interested in.

In Tiddlywiki, that mechanism is achieved with a URL “filter” prefix. For example, show all tiddlers with “systemConfig” tag – http://tiddlywiki.com/#filter:[tag[systemConfig]].

Other things you can do - http://tiddlywiki.com/#newTiddler:tiddlername – create a new tiddler, specifying the name

The URL is modelled as a map, i.e. key-value pairs. In the case of http://www.tiddlywiki.com/#Examples%20Plugins, that’s just an alias for the canonical map form, http://www.tiddlywiki.com/#open:Examples%20open:Plugins. All this is managed by the Paramifiers class.

WIKI sells, Wiki doesn’t

‘Wiki’ derived its name from the Hawaiian for “quick”. But acronyms sell better in the corporate world, as anyone who’s ever won a business case for working with POJOs will be acutely aware, or anyone who’s found AJAX 0wns Ajax. JAVA beats Java when you’re selling a $2K seminar and you wouldn’t be the first to ask, what does COMET stand for anyway?

Therefore, wiki adoption (or, world domination) will be faster if we can make wiki become WIKI. According to YouKnowWhoPedia, there is already one backronym for wiki: “What I Know Is”. Yeah, it’s not much; I know “I-K-I” isn’t the easiest set of letters to work with, but still, we can do better!!!

How’s about …

  • Whoa! It Keeps Improving!
  • Webified Information, Knowledge, and Ideas
  • Wikis Improve Knowledge Iteratively
  • Work It, Keep It
  • Wiki Isn’t Knowledge Immutable
  • Wiki Is Knot Icky
  • Wikis Iterate Knowledge Indefinitely
  • Ward Is Knowingly Ignorant (intended as a compliment!)
  • Ward’s Intelligent Knowledge Igloo
  • Wait, I Know It!
  • Words Idle? Keep Iterating!
  • Want Improvements Kept Indefinitely?
  • Wanted: Invaluable Knowledge on the Internet

Documentation As Conversation

I’m busy preparing a list of desirables for Web 2.0 APIs. One of them is good documentation, and I came up with this term – “Documentation As Conversation” – to articulate much of what is needed in modern software docs – documentation which belongs to the community of users/end-developers as it does to the people who created the product. The counter-pattern is “Documentation As 1995esque blob of HTML that you apparently haven’t updated since 2004″.

Examples of “Documentation As Conversation”:

  • The official PHP docs allow for comments on every single page. Each page represents a function (strlen() etc), meaning you get a great conversation about the intricacies of every single function, and there *are* plenty of intricacies.
  • The offiicial Scriptaculous wiki is a great description of the ins and outs of the API. Likewise Rails Wiki.
  • Javalobby had an effort to effectively wikify the official Java documentation – it still exists, but only for third-party libraries. See, Sun didn’t let it go ahead with the official API and did nothing with the idea themselves. Come on Sun, it’s not too late to help your users grok the doc!
  • Through his aptly named Loud Thinking blog as well as the official Rails blog, Rails creator DHH (+others in the latter case) offers an ongoing insight into the evolution of Rails that is at once colourful and extremely useful to the community. Unlike a number of clueless “official” blogs, comments are wide open.
  • Jon Udell uploads a video explaining how his lawnmower is operated. Manufacturers should be building a gallery of videos, images, and docs submitted by users to themselves or other sites.
  • The oldest online form of Documentation As Conversation – mailing lists, usenet, forums. Developers listen and take art in the conversation.
  • Those who practice “Documentation As Conversation” not only write themselves, but they shepherd the community and keep their ear to the ground. That is, they curate wikis; they respond to blog articles by commenting or blogging back; and they make themselves available for inteviews and appearances.

Documentation As Conversation is the way software should be documented in the world of Web 2.0.

(“Documentation As Conversation” is a play on “Markets As Conversation”, AKA the Cluetrain Manifesto.)

The Wiki Twitch

I can feel a case of the Wiki Twitch coming on …

Victims of the Wiki Twitch have a perfectionist tendency which causes them to optimise content they come across, for the benefit of others and for reasons of “enlightened selfishness” – the motivation to improve what they will likely read again in the future. Many of these Wiki Twitchers have spent hundreds of hours in front of a computer screen, browsing encyclopaedias, reading community-created theories, and some are mad enough to write entire books inside a wiki. The twitch may begin when using these systems, but after some time, Wiki Twitchers fail to differentiate between websites that are wikis and those that are static, leading to a general desire to edit all content, even that which is not open for editing.

When content is hosted on editable wikis, the Wiki Twitch can actually promote a general feeling of well-being. But outside that realm, on the Static Web, it can lead to ultimately unsuccessful gestures – “Wiki Twitches” – towards a non-existent Edit button. Over 99% of the web remains ineditable at this time, leading to frequent delusions of editability.

The situation may be different in the future. Some younger Wiki Twitchers will never know what it’s like to pick up a static encyclopaedia and experience the feeling that it’s immutable. They are increasingly growing up to know a world where everything they can see can be changed by consensus.

A further trend is revealed by the increased interest in Second Life and other Massive Multi Player Online Role Playing Games (MMORPGs). Its only a matter of time before the “everything is editable” delusion of the Wiki Twitch extends to real-world objects.

A significant sub-population of Wiki Twitchers are programmers who have succumbed to the related phenomenon known as Test Infection. The refactoring bug is the cause of this infection, and, though primarily involved in a symbiotic relationship with software code, it has a natural affinity with wiki content.

Individuals suffering Wiki Twitch are advised to spend more time inside the Editable Web. If pain persists, install an appropriate annotation tool, such as Diigo or WizLite.

Got Captcha? Antispam on AjaxPatterns

I’ve been blabbing on about how I’m going to open up the AjaxPatterns wiki for as long as it’s been online (about a year), blah blah, talk is cheap. Anyway, it’s a few steps closer now. The main issue has been protection against spam – some entrepeneurial folks behind numerous proxies have discovered there’s a good niche market among Ajax developers for fake watches and cheap pharmaceuticals, and AjaxPatterns is just the thing for their cunning Long Tail marketing strategy. I wish these measures weren’t necessary, and they certainly won’t be foolproof, but hopefully they’ll let us grow a bunch of useful Ajax content without too much interruption. After The Ajax Experience, I realised how much more there is left to document and how much people want to hear and say.

So these are a few things you’ll see at AjaxPatterns.

  • Captcha – just implemented. Let me know if you have any problems. Unfortunately, it does go against accessibility, but contributors who have difficulty with it could always mail me contributions. Hopefully, mediawiki will incorporate captcha at some point, the kind of project where the resources for a more accessible solution would make sense. Mail me if you want more info on the implementation.
  • Links to book version Implemented, but not live. Each pattern page will link to the corresponding book version (well, a close-to-complete draft). Even if a spammer messes with the pattern description for a short time, there will be a permanent link to the corresponding description in the book.
  • Word/URL BlackList Based on Spam Blacklist. Sorry, no examples involving discount Rolexes.
  • Backup strategy Another measure is frequent snapshots of the whole thing (that’s always been there) .

Wikipedia as a Honeypot

How long until wikipedia becomes a honeypot?

“Who wants to be a millionaire” contestant is struggling to answer the question, “What year did the Fonz jump the shark?”, and calls out to Lifeline Buddy. Back in 2005, Lifeline Buddy would have googled for the answer. But this is 2007, and “wiki” is now a household name (the media refers to “wiki” and “wikipedia” interchangeably). Lifeline Buddy bangs out “fonz jump shark” into wikipedia’s search field and quickly finds the right page, reporting confidently the year was 1975. Only, it’s wrong; Fonzarelli, of course, jumped the shark in 1977. The producers had entered the fraudulent details at precisely the moment the lifeline was consulted. Contestant takes his $1000 consellation and exits, muttering something about the Britannica under his breath.

Producers wouldn’t stoop so low? If the BBC can do it, draw your own conclusion. The Register, in any event, would have a field day with their latest whipping boy.

Instead of restorting to restricting edits, wikipedia first needs to try out a “heat map” view to help people decide how stable the information is. Not as gaudy as the Ajax Patterns authoring heatmap (using a more subtle theme now), but some way for people to know what’s new and what’s old. Again, this comes back to the idea of separating out wiki content from presentation, ideally using some kind of web service. A wiki needs more than one view, even without any Maps/Flickr/Delicious mashup. For example, you could have three standard views:

  • Pure wiki reading, just like wikipedia today.
  • Stability view. e.g. Most content in white as now, but with a few shades of grey to distinguish how old each phrase is (darker grey = past minute, medium grey = past hour, light grey = past day; so a phrase “graduates” from grey to white as it matures).
  • Inspection mode. Full-on data mining interface, using Ajax (of course) to explore history, drill down to author info, etc.