The TiddlyWeb Request-Response Lifecycle

We had a session a little while ago where Chris Dent walked through the lifecycle of a Tiddlyweb call, from incoming request to outgoing response.

Each step is a WSGI middleware object. The exact sequence is of course configurable, by manipulating config[‘server_response_filters’].

Some Terminology

First, some relevant terminology.

Serializer – converts from a tiddler to some format (externalisation), some format to a tiddler (internalisation), or both directions.

Challenger/Extractor – runs through extractors until it finds a valid user. If none found, user is GUEST. User is just a string identifier, doesn’t include a realm in the default cases. Note that TiddlyWeb ships with the basic cookie form challenger and also the open id challenger. If you were to create a basic cookie user called “http://mahemoff.myopenid.com”, you would be causing trouble – two different identities with the same ID, so the rest of TiddlyWeb sees them as the same thing. It’s up to the site operator to configure things right, add the right constraints etc.

Validator – checks content

Filters – modifies the list of tiddlers. select – take a subset of tiddlers, sort – order the tiddlers, limit – limit the number of tiddlers. can also add other keywords via plugin.

“environ” is a “semi-global” – ie it’s a request-scoped variable, ie it’s available to to every stage of the request.

“Incoming” Filters

IN: (each of these is WSGI middleware)

Environator. Looks at WSGI environment. This includes all the CGI goodies – path, user agent, etc; and also anything else we want to control, and we make use of the extra stuff. (~empty right now)

Configurator. Reads config. Sets up environ[‘tiddlyweb.config’] – simply translates the global config (which was read at server start time) into the request. (For a development mode, it would be possible to make a Configurator that reads the config from file each time.)

Query. Deals with incoming query. Can get original query string as environ[‘QUERY_STRING’], but this stage parses it to make it higher level. environ[‘tiddlyweb.query’][‘fat’][0]. If request method is POST and content type is CGI form, it will parse and place into envron[‘tiddlyweb.query’] (and if the POSTed message also contained query params, they’ll be merged there).

StoreSet. Sets environ[‘tiddlyweb.store’] to the actual store object (not just a string like ‘sql’ or ‘text’). (Runs something like environ[‘tiddlyweb.store’] = Store(‘text’….). (where Configurator has setup this kind of store.)

UserExtract. Determines who current user is. environ[‘tiddlyweb.usersign’] = [ ‘username’: ‘..’, ‘roles’: []]. Checks for user via cookie and MAC check. (Doesn’t actually do authentication/login, the challenger is separate.)

Header. Is only here to satisfy “HEAD” requests – calls self as if it was GET and throws out the body.

Negotiate. Determines serialiser based on query string and accept header.

Selector. (unlike all the others, this is not TiddlyWeb, but a separate Middleware package.) Dispatches handlers based on URL pattern, just like Rails routes. urls.map is the default mapping and you would typically extend it (as opposed to replacing it) using something like config.selector.add(‘/bookmarks/{bookmark_id}.{format}’, GET=bookmark_view).

“Outgoing” Filters

HTMLPresenter – top and tail the output. This is only entered iff the content type is HTML.

PermissionsExceptor

HTTPExceptor

EncodeUTF8

SimpleLog

Scrumptious Update: Open ID Support, UI Enhancements

I’ve released a new version of Scrumptious. Main change is it now supports Open ID. You can click a “login” link to comment by Open ID. It’s optional by default, though a Scrumptious site operator could easily make it mandatory for read, write, or both by just changing a couple of words in the “policy” config files. Similar to other CMSs like WordPress, non logged in users can indicate their name and URL when they submit a comment.

There are also UI enhancements – the design is cleaner and looks closer to the original original TiddlyWiki comments plugin. Interestingly, I retained almost identical markup, so I was able to cut-and-paste the original CSS for the comments plugin and it mostly worked. I also now include the default TiddlyWiki stylesheets as well. It’s not just look-and-feel which is closer to the original plugin, but the content – you now have info like modifier, modified date, and a permalink available.

I also added something I always wanted to add on the original plugin, which is some animation, e.g. when you add a new reply, the page scrolls to that reply, and a yellow fade effect highlights it. This is a genuinely useful feature as I was finding it difficult to see which reply I’d just added, when there are a lot of comments around.

I’ve also begun work on a Comments Report showing recent comments. Obvious related enhancement is to take the TiddlyWeb Atom plugin and make a comments feed.

Right now, all this is only tested on Firefox (the original was tested on all browsers, at least the full website view); my next priority is to work on browser compatibility, and after that, extract a modular JQuery comments plugin.

Implementation Notes

Regarding the implementation, TiddlyWeb ships with Open ID by default (Open ID is one of two default challengers, the other being the usual simple user-pass key pair config). The most challenging thing here was getting the UI right for both anonymous users and logged in users, as well as handling a redirect in the popup after a successful login; but at the back end, Open ID “just works”.

In summary, I added Open ID support as follows:

  • Add a “login” link to the TiddlyWeb OpenID challenger UI, using a “target” attribute so the challenger opens in a popup.
  • The challenger URL in that link also contains a redirect param, which I redirected to a new static page. This static page shows the user their login ID (by inspecting the “tiddlyweb_user” cookie value), calls a callback “onLogin” method on the original page, and closes itself.
  • The “onLogin” callback updates the login display to show the logged in user and a logout link; the logout link simply runs some Javascript to remove the cookie. The callback also updates any forms that are open prompting for bio info; it hides this info and in its place, shows the current user ID (read-only).

Thanks to Ben and Chris for pointing me in the right direction on TiddlyWeb’s Open ID support.

PS I discovered late in the day (literally) that TiddlyWeb lets the client specify whatever modifier they want

When Resource IDs are URLs

Being a web app about websites and other resources, Scrumptious has a resource which is basically a URL, called “Pages”. A Scrumptious resource modelling google.com/about looks like:

http://scrumptious.tv/comments/bags/ pages/tiddlers/http%3A%2F%2Fgoogle.com%2Fabout

See? The resource ID is http://google.com/about, encoded. (encodeURIComponent("http://google.com/about")

This was working fine on my dev machine, running “twanager server comments.boz 8080” (comments.boz being an alias for my local machine). But on the server, and run through apache and TiddlyWeb’s apache.py, it failed:

The fix was twofold – both of the following were required:

  • AllowEncodedSlashes On in apache config. This option ensures encoded slashes (%2F) are passed through to the end-app.
  • PathInfoHack plugin As with any TiddlyWeb plugin, I downloaded it and added it to ‘system_plugins’ list in tiddlywebconfig.py. (Thanks Chris for the pointer.)

And now we can happily talk resources with URLs as IDs to the server.

TiddlyDocs, TiddlyCMS, and Permissioning Models

We’re designing a setup for TiddlyDocs (and potentially other “TiddlyCMS”s) where there will be an instance for each group of contributors. This is one TiddlyWeb design pattern, where you say “we all trust each other” (although there are still audit logs!) and so everyone can add, delete, or modify all the content freely. That is, there is only one content bag accessible to all. Likewise, anyone can append comments. There is still an admin role for setting up config stuffs, which only the admin can change; otherwise, any user could easily perform a phisihing attack, for example. Users also have private bags which they can do what they like with – they could use it for drafts, or for overriding shadow tiddlers setup by the admin in config.

A rough sketch looks like this:

(Some things need fixing, but the overall idea is conveyed.)

I discuss more about this architectural pattern with Jon in the following video:

TiddlyDocs Architecture from Michael Mahemoff on Vimeo.

Now that model alone is good enough for basic cases, where you have a group of people who work together and just want to set up an instance. TiddlyWeb handles that just fine, even using the basic cookie authentication technique. In more complex situations (read: “enterprises”), you have a many-to-many relationship between users and instances. I was talking with TiddlyWeb’s architect, Chris Dent, about possibilities, and we decided on one possible model:

A central user-to-pool mapping database, which is shared by three web apps: (a) all the TiddlyDocs’ servers, specifically their Challenger module; (b) an admin app for managing the mappings (typical Django/Rails app with forms generated from data model); (c) a user-facing app (possibly a gadget in our envisioned OpenSocial scenario) showing the user (already authenticated via single-sign-on) a dropdown of instances they can launch.

Scrumptious: Conversations About Websites (and other resources)

Scrumptious is a web framework I’ve begun working on at Osmosoft. It’s a web app and web service for sharing bookmarks and comments about websites, and pretty much anything else with a unique URL. Things it is related to: Delicious (bookmarking), Digg (threaded comments), JS-Kit and Disqus (embedded comments with common identity across multiple sites).

Scrumptious is open source, under the BSD license (meaning you can do just about anything you like with it). There are already many open-source clones of this sort of thing, so why make a new one? There are a few reasons:

  • Adherence to RESTful principles – Scrumptious is backed by TiddlyWeb, a RESTful data service.
  • A non-TiddlyWiki TiddlyWeb client – So far, people who have used TiddlyWeb as a server have used TiddlyWiki as a client. They do play nicely together, but TiddlyWeb is a powerful RESTful framework on its own. Part of the motivation for Scrumptious was to port the TiddlyWiki nest comments plugin to a generic JQuery plugin that could be used on any web page. (Comments are indeed implemented as a plugin right now, but more work needs to be done to extract it into something truly modular and reusable; for example, the plugin right now assumes comments are about a web page; also, they are tied to TiddlyWeb. Nevertheless, the app still does achieve the main purpose of demonstrating TiddlyWeb is a fine data service for generic web apps.)
  • Demonstrating the power of URLs – In evangelising web standards, a very practical piece of advice is simply to associate a unique URL with each distinct resource. That’s REST 101, but it’s something lacking in many web apps. With a tool like Scrumptious, you get a comment system “for free” as long as each resource in your system has a unique URL. We’ll be developing a similar framework for URL Trails in the future, and the same principle applies: use unique URLs, and people can put your stuff in trails “for free”.
  • Flexible security model – Again, TiddlyWeb offers flexible permissioning, so you can use the app in different ways. e.g. a private conversation between colleagues; a public conversation (as with the demo), a publicly readable conversation where only certain individuals can contribute, etc. Likewise, TiddlyWeb offers flexible authentication, so you could hook into an organisation’s LDAP system, use open ID, simple user-password pairs, or any other form of authentication you wish to use.

Scrumptious is still at an early stage. Future work includes:

  • Bookmarking – for now, there is only commenting rather than social bookmarking per se.
  • Nested comments UI needs work to give it the same kind of UI as in the TiddlyWiki comments UI. e.g. show info like creator and creation data, and use suitable rendering and indenting.
  • Graceful degradation – It would be possible to provide a basic system not requiring Javascript. This would be probably be done using TiddlyWeb plugins, though I’d also be interested in running the JQuery-powered web app on the server, using server-side Javascript.
  • TiddlyWiki comments plugin harmonisation – As TiddlyWiki now ships with JQuery, it would be ideal if there was a single code base for the comments plugin, running in and out of TiddlyWiki. Indeed, I hope TiddlyWiki moves towards a general microkernel architecture, in which all plugins are useful outside TiddlyWiki. This is certainly becoming the case, with generic JQuery plugins being extracted for core activities like file saving, CSS applying, and wikifying.
  • Browser extensions – instead of a bookmarklet, use a browser extension to show, for each page, if comments are available, as well as bookmarking info. (Similar to the StumbleUpon or Delicious Firefox extensions.) A good opportunity to get my hands dirty with JetPack.
  • Login and identity management – while TiddlyWeb already provides the security and permissioning model, work is required to handle this at the UI level. For example, let anonymous users enter their email address and homepage, and/or register.
  • User admin – for situations where users must be authenticated, some form of user management would be handy. Again, TiddlyWeb provides a good model for this – a “bag” of tiddlers has its permission specified in JSON, which may include a list of users for each access type, so you simply need to PUT and GET this if you are sufficiently permissioned (ie an admin). What’s missing right now is a UI (in TiddlyWiki too, let alone standalone web apps).

Scrumptious Screencast:

<

p>Scrumptious from Michael Mahemoff on Vimeo.

easy_install: Prerequisite for TiddlyWeb

Done a few TiddlyWeb installs now – here’s a tip. It’s easiest done with easy_install. This is a tool that installs or upgrades everything “in a click”, except that installing easy_install isn’t really easy; the Python community could learn a lot from the Rails community about how to make downloading and installing easy, for those who aren’t intimately familiar with the underlying technology. (Summary: I google for the tool, I click on the first result, and I see a big fat download button). If you try to look up easy_install, also called “easyinstall”, you’ll find it’s actually included with “setuptools”, and you’ll find a website that makes you go around in circles if you click the links you think you’re meant to click. What you might not find is the package you need as the downloads are buried down the bottom of

For a Mac at least, get easy_install with the following steps:

As We May Think: URL Trails

As We May Think, Vannevar Bush’s legendary paper presaging hypertext, wikis, and the web, penned in 1945. Many features of his ficticious “memex” device have come to fruition on the web, but one thing I always find lacking is his concept of trails:

When the user is building a trail, he names it, inserts the name in his code book, and taps it out on his keyboard. Before him are the two items to be joined, projected onto adjacent viewing positions. At the bottom of each there are a number of blank code spaces, and a pointer is set to indicate one of these on each item. The user taps a single key, and the items are permanently joined. In each code space appears the code word. Out of view, but also in the code space, is inserted a set of dots for photocell viewing; and on each item these dots by their positions designate the index number of the other item.

Thereafter, at any time, when one of these items is in view, the other can be instantly recalled merely by tapping a button below the corresponding code space. Moreover, when numerous items have been thus joined together to form a trail, they can be reviewed in turn, rapidly or slowly, by deflecting a lever like that used for turning the pages of a book. It is exactly as though the physical items had been gathered together from widely separated sources and bound together to form a new book. It is more than this, for any item can be joined into numerous trails.

The owner of the memex, let us say, is interested in the origin and properties of the bow and arrow. Specifically he is studying why the short Turkish bow was apparently superior to the English long bow in the skirmishes of the Crusades. He has dozens of possibly pertinent books and articles in his memex. First he runs through an encyclopedia, finds an interesting but sketchy article, leaves it projected. Next, in a history, he finds another pertinent item, and ties the two together. Thus he goes, building a trail of many items. Occasionally he inserts a comment of his own, either linking it into the main trail or joining it by a side trail to a particular item. When it becomes evident that the elastic properties of available materials had a great deal to do with the bow, he branches off on a side trail which takes him through textbooks on elasticity and tables of physical constants. He inserts a page of longhand analysis of his own. Thus he builds a trail of his interest through the maze of materials available to him.

And his trails do not fade. Several years later, his talk with a friend turns to the queer ways in which a people resist innovations, even of vital interest. He has an example, in the fact that the outraged Europeans still failed to adopt the Turkish bow. In fact he has a trail on it. A touch brings up the code book. Tapping a few keys projects the head of the trail. A lever runs through it at will, stopping at interesting items, going off on side excursions. It is an interesting trail, pertinent to the discussion. So he sets a reproducer in action, photographs the whole trail out, and passes it to his friend for insertion in his own memex, there to be linked into the more general trail.

Now at one level, any web page is a trail. Thanks to the magic of hypertext, any website is effectively a view onto the rest of the web, with the author being able to list links and comment on them. However, I don’t find much in the way of explicit support for this concept of trails.

Basically, I see a trail as a linear sequence through a bunch of resources, with the potential to annotate at each step. On the web, a resource is represented as a URL, so assuming everyone plays ball, building a trail is as simple as building a list of URLs, with some form of annotation for each. That’s the nice thing about URLs – it doesn’t matter if it’s a written article, a photo, or a video; they are all URLs. Furthermore, it doesn’t even have to be served via http; it could be a file on your hard drive (file:), an email adress (mail:), or an IRC channel (irc:). They can all live together in harmony in the same trail.

The closest popular thing I’ve seen to this is Amazon’s ListMania. My favourite ones are those beginning with “So you …” as in So you want to be a geologist or So you like to read about the Trojan War. These capture the spirit of Vannevar B’s tome; as a budding geologist, I might find a book or two on geology and see that list linked from the side, where I can get the (hopefully) expert opinion on the trail I can follow towards mastery.

There is also some interest around outlining, which is basically the trail concept when taken in a web concept, although you don’t see it being a big part of the web explicitly.

I was chatting to Jon last night and he refers to these as tour guides, and pointed me to The Nethernet, an unusual “game” that appears on websites you surf, based on Firefox plugins. It includes a concept of a “quest” where you can jump between websites, following a trail laid out by someone else.

All of this has been converging lately with several activities happening at Osmosoft. As a project in my spare time, I’ve been working on a tool to caption images. A trail of captioned images would be good as a slideshow. As a project in his spare time, Simon has been working on an excellent video player. The player runs through a playlist, which can be populated in various ways. Each of the videos is effectively a URL, so the playlist is effectively a trail. And in TiddlyDocs, one of our main projects at Osmosoft, a document is a hierarchy of sections, each being a tiddler sitting on a unique (TiddlyWeb-backed) URL. So the document’s table of contents is also a trail. Jeremy realised the implications of this early on, and we’ve begun talking with him about a richer format for the document spec, so it can include annotations and transitions between items. Note that it’s also a hierarchy rather than a list, but that’s fine because (a) a list is a subset of a hierarchy, so the scheme we arrive at will be able to degenerate into a list representaiton; (b) a hierarchy can still be traversed in a deterministic, linear, fashion – as you do when you read a book which is composed of sections, subsections, and so on.

So where we’re at now is deciding on the format for a trail like this. It will probably be JSON-based and be inspired by OPML at some level. Being a file format which will be baked into people’s content, it needs a little more upfront thinking than a coding exercise.

List of Interesting and Popular MediaWiki Instances

I’m currently at an Osmosoft hackathon event, working on a tiddlywiki vertical called mediawikiunplugged. No time to say more now, but we needed a bunch of mediawiki sites for testing. So in addition to the obWikipedia and obAjaxPatterns, here are more:

And a list of the biggest wikis according to Alexa (retrieved thanks to archive.org).

TiddlyGuv

I’ve just presented TiddlyGuv at the FOSSBazaar workshop, the event I wrote about yesterday. It’s a good time to introduce TiddlyGuv – I’ve mentioned it in some talks and tweeted about it on occasion, but this is its maiden appearance on Software As She’s Developed.

TiddlyGuv (that’s a code name we’ll use for a while) is a web app to support open source software governance. In promoting and supporting open source at Osmosoft, one of the activities is a governance process. This means establishing policies for open source usage and auditing projects against it. TiddlyGuv is the tool we’re developing to support this process – it will support community discussions, policy publication, project tracking, workflow, and alerts about potential problems.

A few things about TiddlyGuv:

  • TiddlyGuv is in progress, it’s not complete and still needs work around a clean installation package. (We’re currently re-working how all this is done with TiddlyWeb verticals; once that’s done, there will be a package so people can try it out.) In any event, the code in some form is online in Trac.
  • TiddlyGuv is a web app. No surprise there :).
  • TiddlyGuv motivated the comments plugin, which has now been used in various other projects, including the live TiddlyWeb documentation wiki.
  • TiddlyGuv is built on the TiddlyWiki-TiddlyWeb stack, like TiddlyDocs. We see a general trend of CMS apps emerging on this stack, which has been called TiddlyCMS. (TiddlyCMS is an architectural pattern.) The aim is that TiddlyCMS apps share a large suite of common components such as the comments plugin.
  • TiddlyGuv is open source. The license hasn’t been finalised, but will likely be BSD, same as TiddlyWiki. We hope other organisations use TiddlyGuv in ther own organisations; TiddlyWiki and TiddlyWeb offer highly flexible architectures which lend themselves to adaptability.

Functionally, the first thing we’re aiming for is a tool with document wiki, for general discussion about open source usage, and a license portal, showing a list of licenses and information about each one. The comments plugin ensures each document and each license can be commented on too. All of this is there today, albeit in an untested state. Later on, we will add info about projects and further still, a workflow tool – for example, “Once the project form has been submitted, it must be reviewed by two named reviewers. They will each receive an email notification when a project is ready for review.” We would want the system to follow rules like that, and we would want to allow site administrators to maintain such rules. The TiddlyWiki model is to encapsulate rules like that inside tiddlers. I foresee some kind of Javascript-powered Domain-Specific Language (DSL).

The FOSSBazaar slide pack:

The feedback from the workshop was positive; there’s interest in the project and there was also interest from one vendor of a commercial product about ways to introduce the wiki or commenting aspects into an existing product. A good example of this sort of thing is Amazon – the way it has a structured product page, which incorporates a forum and a wiki. It raises some interesting architectural issues about how a TiddlyWiki/TiddlyWeb app could be embedded onto a page as a widget-style component, with identity and permissioning handled too. I also had an extended discussion with Mark Donohoe about how TiddlyGuv could talk to FOSSology. Of which, more in the next post.