TiddlyWiki Screencast: Forum in 15 Minutes

TiddlyWiki Screencast: Multi-User Forum in 15 Minutes from Michael Mahemoff on Vimeo.

This screencast is how I finished up the 12Days project, as it captures a lot about what I have come to appreciate in TiddlyWiki. It shows how you can do some simple hacking in the browser, in a single-page web app, reusing and configuring existing plugins from the community. And you can then “turn the handle” to deploy it into a multi-user environment, through the magic of TiddlyWeb. The deployment process in the video uses Hoster, which provides lower-level functionality, but it will be even simpler when TiddlySpace is mature.

This is a screencast showing the evolution of a tiddlywiki, starting from a freshly-minted TiddlyWiki downloaded from TiddlyWiki.com, and transforming it into a viable multi-user forum. The steps shown in this screencast:
  • Customising the forum’s look-and-feel by updating shadow tiddlers (SiteTitle, SiteSubtitle, ColorPalette, StyleSheet, DefaultTiddlers, MainMenu).
  • Using tags, the list macro, and the newTiddler macro to show and create new topics.
  • Reusing components with backstage (CommentsPlugin, SinglePageMode, TaggedTemplateTweak)
  • Building a custom macro (taggedCount)
  • Transforming into a multi-user forum (Hoster)
  • The Tiddly* Universe Leading Up to TiddlySpace

    There have been a dizzying array of Tiddly* products spun off recently. I’m talking merely about the core infrastructure stuff rather than applications like TiddlyPocketBook. We’re starting to get some convergence within the team, as we’re working on a high-level product called TiddlySpace. It’s basically the manifestation of earlier discussion too.the ideas we were discussing earlier. Similar to WordPress MU or Wikia, you’ll be able to spin off new TiddlyWikis at the click of a button. The server itself is open source, so you can host it in your own enterprise or on your own server. And because TiddlyWiki is nowadays an application framework, not just a wiki thing, spinning off new instances means composing new applications.

    Pretty much all the applications we’re working on right now are intended to run on this infrastructure. Many will run in standalone, file-based TiddlyWiki, as well; i.e. the whole server-sidedness and user authorisation is a completely separate deployment concern. Which is an awesome way to develop. I recently realised we’re about the only people in the world who primarily develop file:// based web apps! That they will magically work as multi-user, server-hosted, apps too is certainly a boon for the Tiddly* model.

    Anyway, what are all these Tiddly* products Osmosoft’s been working on, and how do they lead up to TiddlySpace? Below is the universe of products we’ve been working on – if all this seems horrendously complicated at first glance, note that most users will only ever see one thing: either TiddlyWiki or TiddlySpace, and even that may be invisible to them if they are using certain high-level applications. Here, I’m just describing the various components we’re working on to pave that path. It’ll all be nice for users. Trust.

    TiddlyWiki – This is the original client UI. It’s a web app fitting inside a single HTML file – HTML, CSS and Javascript all together. It can run on a file:/// URI, which means you can store TiddlyWikis on your local drive, a USB drive, or a share drive … or a http:// URI. In the former case, thanks to some very clever and still-obscure browser hacks, you can make edit the TiddlyWiki and save changes. TiddlyWiki is, at its heart, a personal scrapbook, where you can easily gather and manage free-form bits of content (“tiddlers”). However, some of these bits can include CSS or Javascript, meaning that you can actually start building powerful applications with it and using the underlying tiddlers as a kind of scaffolding. These applicatons can be useful for individuals, but we’re still limited to a single user working on their own hard drive. Hence the need for server-side products (see below).

    TiddlyWiki 5 – This is a major in-progress upgrade of TiddlyWiki, a complete rewrite, the biggest overhaul in its 5 year existence. There are some much-requested features: follows graceful degradation principle so that the TiddlyWiki5 presents nicely to search engine bots and users with Javascript turned off; includes rich text editing control; SVG graphics; embedded images (with the IE6-compatible MHTML hack); better readiness for server-side integration (e.g. the possibility to track revisions). And yes, all of this in a single HTML file.

    TiddlyWeb (low-level) – This is new RESTful server-side component we’ve been building at Osmosoft for persistence, where each Tiddler has its own URI; it’s similar to CouchDB and other NOSQL frameworks; the main difference is it provides extreme flexibility on all manner of things like authentication and storage and presentation formats, and will mostly be invisible to users as we’re building higher-level abstractions on top of it. Also, the underlying data model is that of tiddlers, which means a title, text, list of tags, creation/modification info, and key-value fields (many NOSQL frameworks will have similar document models, but none exactly matching the Tiddler data structure that has always been present in the TiddlyWiki client UI). For most people, TiddlyWeb is best considered as a powerful component further down the stack than what you actually interact with. You’d only need to know about it in detail if you were trying to do powerful admin things, or change the code, on something higher-level like TiddlySpace; or if you wanted to build something equivalent to TiddlySpace; or you were trying to use it in a kind of NOSQL server capacity.

    TiddlyWebWiki (low-level) – This is a set of plugins for TiddlyWiki that help it talk to TiddlyWeb. It “hijacks” (intercepts) calls like store.saveTiddler() so that instead of saving to the local drive, it will upload to the TiddlyWeb server.

    TiddlyHoster and TiddlyConsole These products will probably converge. TiddlyHoster is a first-cut at the TiddlySpace concept, an inspiring product mostly built by @cdent and cobbling together various plugins. You can try it out now at http://hoster.peermore.com. It’s a fairly direct exposure of the TiddlyWeb bags and recipes model. TiddlyConsole (formerly TiddlyRecon) is similarly a bag and recipe and user mangement tool, with more emphasis on admin usage than that from end-users. We hope the essence of these tools can be incorporporated into TiddlySpace to support power users.

    TiddlySpace – This is a higher-level web app, building on TiddlyWeb, to make it easy for folks to spin off new multi-user (or single-user) TiddlyWiki instances. This is really the most important piece, and the one everything’s been leading up to, when it comes to Osmosoft’s mission to get all manner of web apps running inside the enterprise, and will hopefully be just as useful for other enterprises as well as ad hoc groups working on the broader internet. With TiddlySpace, you have an application running at http://TiddlyPocketBook.TiddlySpace.com for example. You click a “clone” button, type in “CameraGuide”, and suddenly you have a clone of that web app running at http://CameraGuide.TiddlySpace.com. The neat thing is that the new space has copies of all the pocketbook tiddlers, which you can happily hack. The application tiddlers are “copied by reference” from a master application space, say http://TiddlyPocketBookApp.TiddlySpace.com, so you will inherit any future changes made to the application. All this sounds complicated, but will be seamless to the user, and is also easy for us to implement while building TiddlySpace, thanks to TiddlyWebs’ flexible bags and recipes model for containment of tiddlers. (See my earlier discussion too.) Pretty much all of TiddlySpace is driven by client-side code TiddlyWiki plugins, connecting to the TiddlyWeb server.

    We’ve spent a while talking about the design of TiddlySpace and tomorrow we have a hackday with the goal of getting a v0.1 running.

    Multi-User TiddlyWiki

    TiddlyWiki MU is what I’m calling – in the absence of an official name – an effort within Osmosoft to pull together a bunch of work into something that will be very useful in the enterprise and beyond. You could also call it “tiddlywiki as a service” or “multi-room tiddlywiki”. Similar to Wikia or WordPress MU, you can spin off a new multi-user instance with a “single click”. This is “MU” in the sense that there’s already a TiddlyWeb-backed TiddlyWiki product (TiddlyWebWiki). The value-add of TiddlyWiki MU (or whatever it ends up being called) is that you can make a new instance of such a thing without being a system administrator and without going through the effort of building it on the server. This is exactly like the way WordPress MU lets you spin off a new blog without having to set up a new instance of WordPress on the server.

    In addition to the ease of spinning off new instances, there’s an important side benefit from this architectural pattern: synergy. From a user’s perspective, they only have one URL to remember/bookmark/share/link; once at that URL, the system can helpfully guide them through the different instances they have access to. Also, it would be possible to make content that’s used across different rooms. For example, give each user a private bag of tiddlers and they could use it to set their global preferences (with the right UI).

    The whole thing is an open-source server, so an enterprise can just download it, deploy it, and let a thousand flowers bloom as users spin off new instances and do what they will. The really important thing here is that tiddlywiki is not just a wiki, but a framework for web apps. Room admins can easily perform customisations like changing the stylesheet; or go so far as coding up Javascript plugins to radically alter look-and-feel. More importantly, they can update the room’s recipe to include a bags of tiddlers sitting elsewhere on the system. That bag might be the TiddlyDocs system for instance, if they are interested in collaborative document publication, or it might be TiddlyGuv if they want an open-source governance system. They can then fine-tune those systems according to their own needs.

    Here is a rich user story (a rich user story is like a regular user story, but it deliberately includes detail that a “pure agile” developer might object to as “getting ahead of yourself” , You-Ain’t-Gonna-Need-It, or unverifiable; while those objections are valid reasons not to use the story as a direct input to coding or task breakdown, and while rich visions are inevitably too premature to turn out into accurate forecasts, they do help to prove you’re heading in the right direction, to motivate user-centric developers like myself, and to illustrate what you’re doing to people outside the project) …

    (cue dream sequence)


    Jimmy is a moderately experienced TiddlyWiki user and wants to make a room for the enterprise’s music club to collaborate on compositions. So he logs into TiddlyWiki MU, hits “New Room”, and creates a room called “musicians”. He then sees the new room – a vanilla TiddlyWiki with a list of users down the side: currently, just Jimmy, with an Admin icon beside his name. He ignores the “Invite Users” button for now so he can concentrate on setting up the room. The groups wants to put out compositions as collective anthologies, so he decides TiddlyDocs will form the basis of this room. Thus, he clicks on “Manage Room” and a lightbox appears. It shows him the technical detail of the room on one side of the dialog – the bags that make up the room’s recipe. And on the other side, he can search and browse for new bags. He navigates to “tiddlydocs” bag and hits the “Add” button beside it. It now appears on the top of the room recipe.

    Dismissing the lightbox and reloading, the “musicians” room is now a vanilla TiddlyDocs. He clicks on the backstage button on the top of the page and edits a few tiddlers – SiteTitle, SiteSubTitle, and ColorPalette, to give the room its own identity. He also needs a way for the participants to enter musical compositions. Luckily, he already has a standalone TiddlyWiki for music composition, with a Music plugin in it. From backstage, he pulls up the Import dialog and uploads the Music plugin from there. For each imported tiddler, there’s a dropdown showing the bags he can import it into (“musicians-config”, “musicians-comments’ etc). He indicates it should go in the musicians-config bag. On reloading the page, he finds something is broken – he can’t add musical notes from the TiddlyDocs editor. It’s time to call a friend …

    He clicks “Invite Users” and up pops a modal dialog where he enters the email address of Dwight Doomore, a TiddlyWiki expert he’s fortunate to know. He checks the “make this user an admin” box and in the optional message area, explains the problem he’s having. Dwight clicks on a link in the subsequent email and he’s up and running inside the slightly broken “musicians” room. Donning his cyber-shades with Matrix-like precision, he proceeds to create a new plugin tiddler and monkey patches the Music plugin functionality so it plays nicely with TiddlyDocs. Then he tests it by writing up the first new composition, pulls up the user admin panel to remove himself from the room, and replies to Jimmy that it’s all taken care of.

    Pleased with the result, Jimmy writes a few compositions, re-arranges them using TiddlyDocs’ tree control, and adds a little text. Instead of using “Invite Users”, he just mails his colleagues the URL of the “musicians” room. They each visit the URL, indicate they wish to join, and a few days later, Jimmy jumps into the user admin panel, where he can accept their requests to join. Now they can all work together on different documents within the one room, and TiddlyDocs provides enough functionality to control how the final copies will be published.

    Six months later, the first “enterprisey musicians” composition book is printed and bound.


    A single TiddlyWiki MU server has the capacity to facilitate a thousand stories like this, each of them with its own unique characters and quirks. There are plugins for voting, blogging, commenting, structured writing, quizzes, graphics, social bookmarking … it will be fun to see how users deal with all that. Most likely, it will follow the usual pyramid structure:

    • Many users will just create a vanilla room – plain old (multi-user) tiddlywiki. Of these, some will just make it a private scratchpad for themselves and never invite anyone at all.
    • Some users will make the room become a vanilla edition of a specialised application for their group to work on. e.g. collaborate on publication-ready docs with TiddlyDocs; collect and annotate websites with Scrumptious; brainstorm and vote on innovations with New Ideas.
    • A smaller number of users will customise the config tiddlers; pull in extra plugins; build new room-specific plugins; or attempt to combine multiple bags.
    • An even smaller number of users will have to deploy a standalone customised server, perhaps deploying a customised edition of TiddlyWiki MU with certain server-side plugins and configurations. This is the kind of thing you might do if using TiddlyWiki MU for a hardcore, full-fledged, 110% uptime, supported to the teeth, enterprise web app. You’d perhaps run a pilot on the regular TiddlyWiki MU server and then perform a (relatively effortless) migration onto its own dedicated server. Or a power user might just hand-configure a TiddlyWeb instance from parts. (This is where we get into “TiddlyWeb as a generic RESTful server” territory.)

    The first case is already well-supported by other wiki platforms, so it’s nice and all being open-source and TiddlyWiki-based, but not really the killer app. The second is really the sweet spot when it comes to optimising user experience in the short term. We’ve spent a lot of time at Osmosoft thinking about how to deploy “TiddlyDocs” as a standalone package. That’s still interesting, but I think it’s a lot more interesting to ask how to deploy TiddlyWiki MU as a standalone package, and then let users go into web app and make a new “TiddlyDocs” instance with one click. The third case is supported by the general TiddlyWeb model and doesn’t need any special optimisation at this stage.

    [There’s a thread about some of this stuff on the mailing list, so I’ve switched off comments to keep the conversation in one place.]

    TiddlyDocs User Authentication – Generic Design and Custom Features

    We are working on a design that will keep TiddlyDocs as generic as possible. The core TiddlyDocs product will have a concept of “rooms”. This is the same kind of room you see in online forums or chat sites, where everyone inside the room contributes and sees content inside the room they’re in, and can’t see or do anything about rooms they’re not in.

    Our design works like this. Each room is a recipe of several room-specific bags, along with a global config bag. Each room is associated with a unique user role.

    Note the role is used inside the respective bags’ policy files, and they each vary somewhat. e.g. the comments bag says readable by everyone in ‘role’, appendable (“create tiddler” permission) by everyone in ‘role’, editable by “#(role)_admin”. (ie there is a second role called “role_admin”.); whereas the content bags says readable and writeable and editable by everyone in ‘role’; and the config bag says readable by everyone in ‘role’, appendable by ‘#{role}_admin”.

    For now, there’s a “makeroom” script that generates the recipe, the bags and the policies. (I’d ideally like to see the recipe be “magic” in some respects, using a wildcard mechanism).

    That’s the basic design. We also have some unique challenges in this particular installation – (a) users are authenticated using a custom, enterprise-specific, single-sign on solution, where the incoming header for each request identifies user’s ID and (effectively) company; (b) we have certain rooms whose composition is manually configured and are a subset of the company’s membership, and we also have a “everyone” room which is automatically composed of everyone in the company (e.g. we might have a “megacorp-everyone” room along with the manually configured “megacorp-accounting” room, and “megacorp-marketing” room); (c) to let users allocate specific TiddlyDocs to users, we need a dropdown containing real names of all users.

    BTW I’m simplifying some of the finer details here to make the discussion more pertinent to the general topic of TiddlyWeb/TiddlyDocs, rather than our specific problem. All of the following will be developed as proprietary plugins for our own installation. It’s a somewhat complicated set of requirements because it’s a real-world enterprise problem, where we have specialised requirements, combined with integration issues where we can control some things, but can’t control others. (e.g. We can’t change the underlying user store, where we would ideally store which rooms a user is in, instead of holding secondary user records inside TiddlyDocs.)

    We will be building a TiddlyWeb plugin that sets the “extractor” filter. This is one of the several filters the request is subject to, on the way into TiddlyWeb. It is usually paired with the challenger filter – where challenger might present a username/password form and set a cookie, and extractor will read the cookie and set usersign (aka user ID) and any other info like users’ roles. In this case, though, we already have the header set, so we don’t need a challenger at all. We simply need an extractor that sets the usersign from the incoming header, and also notes the user’s company. Verification will then check that (a) if the user is attempting to access the “everyone” company room, they are a member of the company; (b) if the user is attempting to access a “manual” company room, they are a member of the company and they have the role associated with the company. We use a naming convention to determine which company a manual room belongs to.

    On (c), we need a tiddler in the room “config” bag containing a list of user names. So in addition to the custom extractor, we will also need a periodic script to poll the user store associated with the SSO system. This will give us all the info about users in each company, so we can build up a mapping from company to list of users, where each user record has an ID and name. company->list[{user ID, user name]}. The script will then (a) rewrite the list of user names for the “everyone room” based on the company list, by effectively “copying” the results for that company (b) rewrite the list of user names for each manual room, by walking through each user, checking their roles, and accumulating their user name into the room list for each room they’re in (c) do some cleanup – removing TiddlyWeb records for any users not in the list.

    There’s nothing very elegant about these custom implementations, and I’ve primarily written these details for the benefit of myself and colleagues who are implementing these custom features. The more interesting, general, point here is about open-source and plugins. TiddlyWeb’s architecture allows us deliver a generic solution for TiddlyDocs rooms, which is applicable to any organisation, and we can then superimpose a custom feature using the plugin mechanism. Furthermore, the plugin mechanism is a great way to decouple concerns – plugins are moduels on steroids. The core TiddlyDocs developer can concentrate on building an open-source, functional, multi-room TiddlyDocs edition, using the default authentication mechanism. Meanwhile, other developers can work in parallel on custom plugins.

    (This post is a refinement of this post; we have been doing further thinking as we get closer to the final product and we learn more about our external environment.)

    TiddlyWeb Now Reloads Code

    I’ve begun working on the server-side of TiddlyWeb a bit, mostly to make stuff degrade gracefully as I begin to build a Trails facility into Scrumptious. In TiddlyWeb, an app-specific plugin is how you build an app on the server-side (or you can use a completely separate web framework like Django or Rails, making server-to-server RESTful calls, but that’s another story).

    I asked about reloading code on the group this week. I want to change my Python code and hit reload in the browser and see the changes. To me, this this kind of “hot deploy” during development is a must-have for any web framework. There’s an argument that testing would render it unnecessary, but (a) it’s difficult to write tests when you’re learning and exploring an environment; (b) kool-aid aside, some things can’t be tested or are Test-Driven Design is simply not the most productive way to develop it (it comes down to forces – a cost-benefit analysis); (c) functional and integration tests, working across the whole server, and reloading code on each request is an easy way to make those happen.

    Anyway, Chris has kindly produced the necessary pieces.

    Here’s how I got it working. Here’s what I did to get it working on Mac, which is slightly different from Linux setup:

    • Download reloader.py and wserver.py stick them in top-level tiddlyweb instance directory.
    • Introduce the following to your tiddlywebconfig.py: 'twanager_plugins': ['wserver']. Note – do not try to use reloader as a plugin; it just needs to be in your directory as wserver uses it.
    • Run the server using twanager wserver.

    If you have browser caching on, you might need to hit shift-reload at times. I don’t have caching on and it works fine with just reload. This is a big help, thanks Chris.

    TiddlyWeb-JQuery Comments Plugin

    TiddlyWeb-JQuery Comments Plugin – Screencast @ Vimeo

    I’ve extracted from Scrumptious a nested comments plugin you can integrate into any POJA (plain ol’ Javascript app). You can find it here in the repo: TiddlyWeb-JQuery Comments Plugin.

    As the README explains, usage is a one-liner once you’ve set it up. Just do $(selector).comments(“topic”). The topic is an identifier for the set of comments; so that when the UI loads, it pulls down all comments with that ID in its “topic” field. Internally, topic acts as the root of the comments tree…but you don’t really need to know that.

    This plugin is a JQuery equivalent of the TiddlyWiki comments plugin, minus some of the more exotic options, and the ability to delete. Eventually, I hope to introduce those things, but only as the need arises – so if you’re using this, please let me know how I can improve it.

    Osmosoft Hackathon: WikiData, a Wiki of Companies Data

    End of The WikiData Hackday

    Osmosoft, Hackathons

    At Osmosoft, we have been engaging in a one-day hackathon about every month or so. There are several benefits:

    • It helps us prove our tech in a range of application contexts.
    • It helps improve and demonstrate our capabilities for reuse. Reuse can be as simple as slapping a Javascript file with inline comments on a server somewhere, or exposing a simple web service. With those capabilities in place, it’s possible to build something useful in a day, just as industry hackathons/camps demonstrate.
    • It helps us spread the word about web standards and why everything needs its own URI.
    • It helps us demonstrate and gain experience in agile processes.
    • It gets us Osmosofties working together in the same place at the same time, whereas day-to-day we tend to work on products in small groups or individually.

    The WikiData Hackathon

    The latest hackathon was last Thursday, July 16, which saw us collaborate with Avox, a company specialising in business entity data. I believe it’s working in a similar space to companies like Dunn and Bradstreet and other information providers, in that they collect, verify, and report information about companies. Avox is keen to open up parts of their database and gain the benefits of community feedback – with users being able to contribute information, leave comments, add new companies, and so on. Of course, Avox is in the business of gathering verified data, so it’s not just a case of making a new wiki and letting it run by itself. There remain challenges about how Avox will merge and verify community input, and how they will make it clear to users what’s verified and what’s not.

    Pre-Work

    We had a conversation on the mailing list about what we did last time and what could do differently for this hackathon. Like a retrospective, but over email. TiddlyWeb architect Chris Dent set up some instances with sample data taken from the existing product.

    Venue and Attendance

    The hackathon took place in Osmosoft’s office, centered around our big table in the centre of the room. Seven people from Avox, in a range of technical and non-technical roles, attended for the duration of the event. Osmosoft had five developers working on stories, a developer working on integration and mediation, and another helping with the server and offering second-line support.

    Introductions and Overview (about 1 hour)

    We went round the table and everyone introduced themselves. Jeremy Ruston explained “what’s in it” for Osmosoft, as outlined above, and Ken Price outlined Avox’s interest in building wikidata. We also looked at the existing product. We then had a discussion which was mostly Osmosoft people asking Avox people questions about the nature of their business, the technologies involves, and their vision for wikidata. Paul began writing down stories on flashcards during this time.

    User Stories (about 1 hour)

    We finished writing stories – this was somewhat in parallel to the previous activity – and put them all up on the wall, with the magic of Blu-Tac. The stories were in the form “As a (role), I want to (task), so that I can (benefit)”. With about 20 stories, it was useful to organise them, so we grouped them according to the role. The main roles were for Avox staff and general users. There were also some stories involving 3rd party developers and employees of companies listed in the wiki.

    Everyone gathered around the stories, we read them out, and we all agreed on priorities. We didn’t worry too much about ordering the stories at the bottom as it was unlikely we’d get to them during the event; if we did, we could prioritise later.

    What we ended up with was a solid idea of the steel thread we were building. It would mostly be standard wiki functionality, but applied to the particular data and context of companies info. We had some bonus stories we could implement if we had time, like comments and tagging.

    Planning and Infrastructure(about 30 minutes)

    Developers put their initials against the first story they’d be working on, and likewise each story needed a customer representative to initial the story, so they would help refine requirements. In the event, we didn’t talk all that much with customers during development; it’s obviously an extremely important thing to do in a real project, but when you’re wanting to get real functionality out in a day, the cost of a conversation is relatively high and the benefit to the task is relatively low. It would have been different in a 2-3 day event, or with certain stories that are highly specific to the domain. The main thing was to checkpoint with customers to check that we’re generally on track.

    Around noon, we drew up a timeline and agreed on a hard stop at 7pm. This means we timebox – the time is fixed and the only variable is the number of stories that get implemented in that time. We then got into

    Development Sprints and Standup Meetings (about 6.5 hours)

    We developed in sprints, with standup meetings in between. It was felt the previous hackathon’s hourly sprints were too frequent, so in this case the plan was every 1.5-2 hours; we decided at the end of each standup when the next would be.

    Most of us developed against the user stories. We also had a very useful role fulfilled by Fred, who was designated as a general “helper” and mediator. This was the result of our feeling that things were sometimes disjointed in previous hackathons – not enough focus on integration and infrastructure. I feel that this role was very useful and Fred fulfilled it admirably, although at times he probably felt like he had 400 people talking to him at once! We also had Chris Dent working remotely to ensure the TiddlyWeb server was running and help deploy to the live site.

    The tool we developed was a standard Javascript/JQuery web app (ie nothing to do with TiddlyWiki) talking to the TiddlyWeb server.

    At the start, we intended to write the web app and talk to the live server. But it soon became apparent that we would step on each others’ toes by doing this, and opted for a more conventional setup where we each have our own server instance. We also had a quick debate about version control – github, private osmosoft repository, or tiddlywiki repository. This is a recurring debate which we ought to decide in advance next time. It partly arose after deciding to run our local copies of the server; it was felt this would be too much data for the tiddlywiki repo, so we used the private osmo svn. As for github, it would be nice to use git, but most of us are more familiar with SVN as we use the tiddlywiki SVN repo day-to-day, so it would cause too much complication to switch to git. Again, it might be a good idea for next time to use github instead, with some pre-work.

    Presentation (about 45 minutes)

    After the usual last-minute rush, we got a working product deployed. It must be noted that we let the “hard stop” slip by about 45 minutes, to about 7:30. Admittedly a bad practice, but it did yield some major results in this case, as it got us search, edit, and maps all working during that time.

    Each of the developers explained what they’d worked on. We then gathered round the screen and walked through the app. Avox gave their thoughts, I recorded the video above while others broke out into informal conversations, and by that point, it was pub o’clock.

    We were able to produce the steel thread we’d been planning; the key stories were demonstrated. We also implemented commenting and Google Maps integration. Being based on TiddlyWeb means we had also produced a working API “for free”; it’s just the nature of TiddlyWeb as a RESTful data container. (On a personal note, commenting was the main thing I did – I extracted the comments feature from Scrumptious into a standalone plugin, and integrated it into the emerging WikiData product. I’ll post about that separately.)

    Wikidatathon

    Wikidatathon

    About the Video

    I recorded the video above at the end of the event. One of my bugbears about hackathon events is that people spend all day coding and setting up infrastructure, and it inevitably comes down after that, or gradually falls apart as various third-party services close, databases get corrupted, people forget to keep hosting it, etc etc. In other words, you have to assume the online result of any hackathon event is transient. This is unfortunate, because the deliverable should be something that people can use to decide what happens next, whether to fund a project, and so on. While meeting minutes are often a waste of time, the artifacts that emerge in a workshop are critical to followup.

    For that reason, I am perfectly passionate about taking a screencast or video to capture what took place in such events. Thanks to Paul and Ken for taking part in it.

    Update: Avox CEO Ken Price (who appears in the video), has published a summary of the event.

    The TiddlyWeb Request-Response Lifecycle

    We had a session a little while ago where Chris Dent walked through the lifecycle of a Tiddlyweb call, from incoming request to outgoing response.

    Each step is a WSGI middleware object. The exact sequence is of course configurable, by manipulating config['server_response_filters'].

    Some Terminology

    First, some relevant terminology.

    Serializer – converts from a tiddler to some format (externalisation), some format to a tiddler (internalisation), or both directions.

    Challenger/Extractor – runs through extractors until it finds a valid user. If none found, user is GUEST. User is just a string identifier, doesn’t include a realm in the default cases. Note that TiddlyWeb ships with the basic cookie form challenger and also the open id challenger. If you were to create a basic cookie user called “http://mahemoff.myopenid.com”, you would be causing trouble – two different identities with the same ID, so the rest of TiddlyWeb sees them as the same thing. It’s up to the site operator to configure things right, add the right constraints etc.

    Validator – checks content

    Filters – modifies the list of tiddlers. select – take a subset of tiddlers, sort – order the tiddlers, limit – limit the number of tiddlers. can also add other keywords via plugin.

    “environ” is a “semi-global” – ie it’s a request-scoped variable, ie it’s available to to every stage of the request.

    “Incoming” Filters

    IN: (each of these is WSGI middleware)

    Environator. Looks at WSGI environment. This includes all the CGI goodies – path, user agent, etc; and also anything else we want to control, and we make use of the extra stuff. (~empty right now)

    Configurator. Reads config. Sets up environ['tiddlyweb.config'] – simply translates the global config (which was read at server start time) into the request. (For a development mode, it would be possible to make a Configurator that reads the config from file each time.)

    Query. Deals with incoming query. Can get original query string as environ['QUERY_STRING'], but this stage parses it to make it higher level. environ['tiddlyweb.query']['fat'][0]. If request method is POST and content type is CGI form, it will parse and place into envron['tiddlyweb.query'] (and if the POSTed message also contained query params, they’ll be merged there).

    StoreSet. Sets environ['tiddlyweb.store'] to the actual store object (not just a string like ‘sql’ or ‘text’). (Runs something like environ['tiddlyweb.store'] = Store(‘text’….). (where Configurator has setup this kind of store.)

    UserExtract. Determines who current user is. environ['tiddlyweb.usersign'] = [ ‘username’: ‘..’, ‘roles’: []]. Checks for user via cookie and MAC check. (Doesn’t actually do authentication/login, the challenger is separate.)

    Header. Is only here to satisfy “HEAD” requests – calls self as if it was GET and throws out the body.

    Negotiate. Determines serialiser based on query string and accept header.

    Selector. (unlike all the others, this is not TiddlyWeb, but a separate Middleware package.) Dispatches handlers based on URL pattern, just like Rails routes. urls.map is the default mapping and you would typically extend it (as opposed to replacing it) using something like config.selector.add(‘/bookmarks/{bookmark_id}.{format}’, GET=bookmark_view).

    “Outgoing” Filters

    HTMLPresenter – top and tail the output. This is only entered iff the content type is HTML.

    PermissionsExceptor

    HTTPExceptor

    EncodeUTF8

    SimpleLog

    Scrumptious Update: Open ID Support, UI Enhancements

    I’ve released a new version of Scrumptious. Main change is it now supports Open ID. You can click a “login” link to comment by Open ID. It’s optional by default, though a Scrumptious site operator could easily make it mandatory for read, write, or both by just changing a couple of words in the “policy” config files. Similar to other CMSs like WordPress, non logged in users can indicate their name and URL when they submit a comment.

    There are also UI enhancements – the design is cleaner and looks closer to the original original TiddlyWiki comments plugin. Interestingly, I retained almost identical markup, so I was able to cut-and-paste the original CSS for the comments plugin and it mostly worked. I also now include the default TiddlyWiki stylesheets as well. It’s not just look-and-feel which is closer to the original plugin, but the content – you now have info like modifier, modified date, and a permalink available.

    I also added something I always wanted to add on the original plugin, which is some animation, e.g. when you add a new reply, the page scrolls to that reply, and a yellow fade effect highlights it. This is a genuinely useful feature as I was finding it difficult to see which reply I’d just added, when there are a lot of comments around.

    I’ve also begun work on a Comments Report showing recent comments. Obvious related enhancement is to take the TiddlyWeb Atom plugin and make a comments feed.

    Right now, all this is only tested on Firefox (the original was tested on all browsers, at least the full website view); my next priority is to work on browser compatibility, and after that, extract a modular JQuery comments plugin.

    Implementation Notes

    Regarding the implementation, TiddlyWeb ships with Open ID by default (Open ID is one of two default challengers, the other being the usual simple user-pass key pair config). The most challenging thing here was getting the UI right for both anonymous users and logged in users, as well as handling a redirect in the popup after a successful login; but at the back end, Open ID “just works”.

    In summary, I added Open ID support as follows:

    • Add a “login” link to the TiddlyWeb OpenID challenger UI, using a “target” attribute so the challenger opens in a popup.
    • The challenger URL in that link also contains a redirect param, which I redirected to a new static page. This static page shows the user their login ID (by inspecting the “tiddlyweb_user” cookie value), calls a callback “onLogin” method on the original page, and closes itself.
    • The “onLogin” callback updates the login display to show the logged in user and a logout link; the logout link simply runs some Javascript to remove the cookie. The callback also updates any forms that are open prompting for bio info; it hides this info and in its place, shows the current user ID (read-only).

    Thanks to Ben and Chris for pointing me in the right direction on TiddlyWeb’s Open ID support.

    PS I discovered late in the day (literally) that TiddlyWeb lets the client specify whatever modifier they want