TiddlyWiki Screencast: Forum in 15 Minutes

TiddlyWiki Screencast: Multi-User Forum in 15 Minutes from Michael Mahemoff on Vimeo.

This screencast is how I finished up the 12Days project, as it captures a lot about what I have come to appreciate in TiddlyWiki. It shows how you can do some simple hacking in the browser, in a single-page web app, reusing and configuring existing plugins from the community. And you can then “turn the handle” to deploy it into a multi-user environment, through the magic of TiddlyWeb. The deployment process in the video uses Hoster, which provides lower-level functionality, but it will be even simpler when TiddlySpace is mature.

This is a screencast showing the evolution of a tiddlywiki, starting from a freshly-minted TiddlyWiki downloaded from TiddlyWiki.com, and transforming it into a viable multi-user forum. The steps shown in this screencast:
  • Customising the forum’s look-and-feel by updating shadow tiddlers (SiteTitle, SiteSubtitle, ColorPalette, StyleSheet, DefaultTiddlers, MainMenu).
  • Using tags, the list macro, and the newTiddler macro to show and create new topics.
  • Reusing components with backstage (CommentsPlugin, SinglePageMode, TaggedTemplateTweak)
  • Building a custom macro (taggedCount)
  • Transforming into a multi-user forum (Hoster)
  • TWelve Days of TiddlyWiki, TWelve Days of Osmosoft

    I’m going to be moving on from Osmosoft soon. It’s been 18-odd months since joining and 3 years at BT. I’ve had a wonderful time here, worked alongside so much dedicated talent and energetic passion, and learned a ton about working according to the principles of open source, as well as new ways of working and thinking about things in a large enterprise. What I’m doing next is the subject of a subsequent post … this one’s about Osmosoft and TiddlyWiki and a new site I’ve set up for the occasion: 12 Days of TiddlyWiki.

    I have to echo much of Jon Lister’s sentiments. I, too, am extremely grateful to Jeremy and JP for being brave enough to set up Osmosoft. It’s an organisation within BT that works predominately on the public internet and produces open source for everyone to use, but not in the mould of a “pure research outpost” or an “isolated skunkworks initative”. Osmosoft directly supports BT through the applications and components it produces, its involvement in open source policy, and its general involvement in BT’s software community, among other things. We just operate very differently to a conventional IT department. Oh, and we also make world class stock photos …

    To celebrate my time here and to gently encourage myself to get a few things wrapped up, I decided to make a little microsite: 12 Days of TiddlyWiki. It will feature a new product for each of my final 12days at Osmosoft. And of course, it’s dogfooding: it’s powered by TiddlyWiki and if you peek under the RESTful hood, you’ll see it’s just another view into the main Osmosoft server, and could easily be mashed up however you please.

    There’s one more benefit of Osmosoft working in an open source way to mention: Leaving is not leaving. And here I’ll echo another friend, Dion, who mentioned a similar effect when he moved on from the Bespin project … the code’s out there and I’m still able to remain part of the TiddlyWiki community. I had no idea how powerful TiddlyWiki was when I first joined. It’s a story that still needs to be told in full, but the bottom line for me is it scratches plenty of itches, and on that basis, I’ll still be using it and contributing back.

    The Tiddly* Universe Leading Up to TiddlySpace

    There have been a dizzying array of Tiddly* products spun off recently. I’m talking merely about the core infrastructure stuff rather than applications like TiddlyPocketBook. We’re starting to get some convergence within the team, as we’re working on a high-level product called TiddlySpace. It’s basically the manifestation of earlier discussion too.the ideas we were discussing earlier. Similar to WordPress MU or Wikia, you’ll be able to spin off new TiddlyWikis at the click of a button. The server itself is open source, so you can host it in your own enterprise or on your own server. And because TiddlyWiki is nowadays an application framework, not just a wiki thing, spinning off new instances means composing new applications.

    Pretty much all the applications we’re working on right now are intended to run on this infrastructure. Many will run in standalone, file-based TiddlyWiki, as well; i.e. the whole server-sidedness and user authorisation is a completely separate deployment concern. Which is an awesome way to develop. I recently realised we’re about the only people in the world who primarily develop file:// based web apps! That they will magically work as multi-user, server-hosted, apps too is certainly a boon for the Tiddly* model.

    Anyway, what are all these Tiddly* products Osmosoft’s been working on, and how do they lead up to TiddlySpace? Below is the universe of products we’ve been working on – if all this seems horrendously complicated at first glance, note that most users will only ever see one thing: either TiddlyWiki or TiddlySpace, and even that may be invisible to them if they are using certain high-level applications. Here, I’m just describing the various components we’re working on to pave that path. It’ll all be nice for users. Trust.

    TiddlyWiki – This is the original client UI. It’s a web app fitting inside a single HTML file – HTML, CSS and Javascript all together. It can run on a file:/// URI, which means you can store TiddlyWikis on your local drive, a USB drive, or a share drive … or a http:// URI. In the former case, thanks to some very clever and still-obscure browser hacks, you can make edit the TiddlyWiki and save changes. TiddlyWiki is, at its heart, a personal scrapbook, where you can easily gather and manage free-form bits of content (“tiddlers”). However, some of these bits can include CSS or Javascript, meaning that you can actually start building powerful applications with it and using the underlying tiddlers as a kind of scaffolding. These applicatons can be useful for individuals, but we’re still limited to a single user working on their own hard drive. Hence the need for server-side products (see below).

    TiddlyWiki 5 – This is a major in-progress upgrade of TiddlyWiki, a complete rewrite, the biggest overhaul in its 5 year existence. There are some much-requested features: follows graceful degradation principle so that the TiddlyWiki5 presents nicely to search engine bots and users with Javascript turned off; includes rich text editing control; SVG graphics; embedded images (with the IE6-compatible MHTML hack); better readiness for server-side integration (e.g. the possibility to track revisions). And yes, all of this in a single HTML file.

    TiddlyWeb (low-level) – This is new RESTful server-side component we’ve been building at Osmosoft for persistence, where each Tiddler has its own URI; it’s similar to CouchDB and other NOSQL frameworks; the main difference is it provides extreme flexibility on all manner of things like authentication and storage and presentation formats, and will mostly be invisible to users as we’re building higher-level abstractions on top of it. Also, the underlying data model is that of tiddlers, which means a title, text, list of tags, creation/modification info, and key-value fields (many NOSQL frameworks will have similar document models, but none exactly matching the Tiddler data structure that has always been present in the TiddlyWiki client UI). For most people, TiddlyWeb is best considered as a powerful component further down the stack than what you actually interact with. You’d only need to know about it in detail if you were trying to do powerful admin things, or change the code, on something higher-level like TiddlySpace; or if you wanted to build something equivalent to TiddlySpace; or you were trying to use it in a kind of NOSQL server capacity.

    TiddlyWebWiki (low-level) – This is a set of plugins for TiddlyWiki that help it talk to TiddlyWeb. It “hijacks” (intercepts) calls like store.saveTiddler() so that instead of saving to the local drive, it will upload to the TiddlyWeb server.

    TiddlyHoster and TiddlyConsole These products will probably converge. TiddlyHoster is a first-cut at the TiddlySpace concept, an inspiring product mostly built by @cdent and cobbling together various plugins. You can try it out now at http://hoster.peermore.com. It’s a fairly direct exposure of the TiddlyWeb bags and recipes model. TiddlyConsole (formerly TiddlyRecon) is similarly a bag and recipe and user mangement tool, with more emphasis on admin usage than that from end-users. We hope the essence of these tools can be incorporporated into TiddlySpace to support power users.

    TiddlySpace – This is a higher-level web app, building on TiddlyWeb, to make it easy for folks to spin off new multi-user (or single-user) TiddlyWiki instances. This is really the most important piece, and the one everything’s been leading up to, when it comes to Osmosoft’s mission to get all manner of web apps running inside the enterprise, and will hopefully be just as useful for other enterprises as well as ad hoc groups working on the broader internet. With TiddlySpace, you have an application running at http://TiddlyPocketBook.TiddlySpace.com for example. You click a “clone” button, type in “CameraGuide”, and suddenly you have a clone of that web app running at http://CameraGuide.TiddlySpace.com. The neat thing is that the new space has copies of all the pocketbook tiddlers, which you can happily hack. The application tiddlers are “copied by reference” from a master application space, say http://TiddlyPocketBookApp.TiddlySpace.com, so you will inherit any future changes made to the application. All this sounds complicated, but will be seamless to the user, and is also easy for us to implement while building TiddlySpace, thanks to TiddlyWebs’ flexible bags and recipes model for containment of tiddlers. (See my earlier discussion too.) Pretty much all of TiddlySpace is driven by client-side code TiddlyWiki plugins, connecting to the TiddlyWeb server.

    We’ve spent a while talking about the design of TiddlySpace and tomorrow we have a hackday with the goal of getting a v0.1 running.

    Multi-User TiddlyWiki

    TiddlyWiki MU is what I’m calling – in the absence of an official name – an effort within Osmosoft to pull together a bunch of work into something that will be very useful in the enterprise and beyond. You could also call it “tiddlywiki as a service” or “multi-room tiddlywiki”. Similar to Wikia or WordPress MU, you can spin off a new multi-user instance with a “single click”. This is “MU” in the sense that there’s already a TiddlyWeb-backed TiddlyWiki product (TiddlyWebWiki). The value-add of TiddlyWiki MU (or whatever it ends up being called) is that you can make a new instance of such a thing without being a system administrator and without going through the effort of building it on the server. This is exactly like the way WordPress MU lets you spin off a new blog without having to set up a new instance of WordPress on the server.

    In addition to the ease of spinning off new instances, there’s an important side benefit from this architectural pattern: synergy. From a user’s perspective, they only have one URL to remember/bookmark/share/link; once at that URL, the system can helpfully guide them through the different instances they have access to. Also, it would be possible to make content that’s used across different rooms. For example, give each user a private bag of tiddlers and they could use it to set their global preferences (with the right UI).

    The whole thing is an open-source server, so an enterprise can just download it, deploy it, and let a thousand flowers bloom as users spin off new instances and do what they will. The really important thing here is that tiddlywiki is not just a wiki, but a framework for web apps. Room admins can easily perform customisations like changing the stylesheet; or go so far as coding up Javascript plugins to radically alter look-and-feel. More importantly, they can update the room’s recipe to include a bags of tiddlers sitting elsewhere on the system. That bag might be the TiddlyDocs system for instance, if they are interested in collaborative document publication, or it might be TiddlyGuv if they want an open-source governance system. They can then fine-tune those systems according to their own needs.

    Here is a rich user story (a rich user story is like a regular user story, but it deliberately includes detail that a “pure agile” developer might object to as “getting ahead of yourself” , You-Ain’t-Gonna-Need-It, or unverifiable; while those objections are valid reasons not to use the story as a direct input to coding or task breakdown, and while rich visions are inevitably too premature to turn out into accurate forecasts, they do help to prove you’re heading in the right direction, to motivate user-centric developers like myself, and to illustrate what you’re doing to people outside the project) …

    (cue dream sequence)


    Jimmy is a moderately experienced TiddlyWiki user and wants to make a room for the enterprise’s music club to collaborate on compositions. So he logs into TiddlyWiki MU, hits “New Room”, and creates a room called “musicians”. He then sees the new room – a vanilla TiddlyWiki with a list of users down the side: currently, just Jimmy, with an Admin icon beside his name. He ignores the “Invite Users” button for now so he can concentrate on setting up the room. The groups wants to put out compositions as collective anthologies, so he decides TiddlyDocs will form the basis of this room. Thus, he clicks on “Manage Room” and a lightbox appears. It shows him the technical detail of the room on one side of the dialog – the bags that make up the room’s recipe. And on the other side, he can search and browse for new bags. He navigates to “tiddlydocs” bag and hits the “Add” button beside it. It now appears on the top of the room recipe.

    Dismissing the lightbox and reloading, the “musicians” room is now a vanilla TiddlyDocs. He clicks on the backstage button on the top of the page and edits a few tiddlers – SiteTitle, SiteSubTitle, and ColorPalette, to give the room its own identity. He also needs a way for the participants to enter musical compositions. Luckily, he already has a standalone TiddlyWiki for music composition, with a Music plugin in it. From backstage, he pulls up the Import dialog and uploads the Music plugin from there. For each imported tiddler, there’s a dropdown showing the bags he can import it into (“musicians-config”, “musicians-comments’ etc). He indicates it should go in the musicians-config bag. On reloading the page, he finds something is broken – he can’t add musical notes from the TiddlyDocs editor. It’s time to call a friend …

    He clicks “Invite Users” and up pops a modal dialog where he enters the email address of Dwight Doomore, a TiddlyWiki expert he’s fortunate to know. He checks the “make this user an admin” box and in the optional message area, explains the problem he’s having. Dwight clicks on a link in the subsequent email and he’s up and running inside the slightly broken “musicians” room. Donning his cyber-shades with Matrix-like precision, he proceeds to create a new plugin tiddler and monkey patches the Music plugin functionality so it plays nicely with TiddlyDocs. Then he tests it by writing up the first new composition, pulls up the user admin panel to remove himself from the room, and replies to Jimmy that it’s all taken care of.

    Pleased with the result, Jimmy writes a few compositions, re-arranges them using TiddlyDocs’ tree control, and adds a little text. Instead of using “Invite Users”, he just mails his colleagues the URL of the “musicians” room. They each visit the URL, indicate they wish to join, and a few days later, Jimmy jumps into the user admin panel, where he can accept their requests to join. Now they can all work together on different documents within the one room, and TiddlyDocs provides enough functionality to control how the final copies will be published.

    Six months later, the first “enterprisey musicians” composition book is printed and bound.


    A single TiddlyWiki MU server has the capacity to facilitate a thousand stories like this, each of them with its own unique characters and quirks. There are plugins for voting, blogging, commenting, structured writing, quizzes, graphics, social bookmarking … it will be fun to see how users deal with all that. Most likely, it will follow the usual pyramid structure:

    • Many users will just create a vanilla room – plain old (multi-user) tiddlywiki. Of these, some will just make it a private scratchpad for themselves and never invite anyone at all.
    • Some users will make the room become a vanilla edition of a specialised application for their group to work on. e.g. collaborate on publication-ready docs with TiddlyDocs; collect and annotate websites with Scrumptious; brainstorm and vote on innovations with New Ideas.
    • A smaller number of users will customise the config tiddlers; pull in extra plugins; build new room-specific plugins; or attempt to combine multiple bags.
    • An even smaller number of users will have to deploy a standalone customised server, perhaps deploying a customised edition of TiddlyWiki MU with certain server-side plugins and configurations. This is the kind of thing you might do if using TiddlyWiki MU for a hardcore, full-fledged, 110% uptime, supported to the teeth, enterprise web app. You’d perhaps run a pilot on the regular TiddlyWiki MU server and then perform a (relatively effortless) migration onto its own dedicated server. Or a power user might just hand-configure a TiddlyWeb instance from parts. (This is where we get into “TiddlyWeb as a generic RESTful server” territory.)

    The first case is already well-supported by other wiki platforms, so it’s nice and all being open-source and TiddlyWiki-based, but not really the killer app. The second is really the sweet spot when it comes to optimising user experience in the short term. We’ve spent a lot of time at Osmosoft thinking about how to deploy “TiddlyDocs” as a standalone package. That’s still interesting, but I think it’s a lot more interesting to ask how to deploy TiddlyWiki MU as a standalone package, and then let users go into web app and make a new “TiddlyDocs” instance with one click. The third case is supported by the general TiddlyWeb model and doesn’t need any special optimisation at this stage.

    [There's a thread about some of this stuff on the mailing list, so I've switched off comments to keep the conversation in one place.]

    InfoBoxPlugin: A TiddlyWiki Plugin for InfoBoxen

    InfoBoxPlugin - Lists tiddlers in a table (by mahemoff)

    G’Day, here’s a new tiddlywiki plugin I’ve been working on: InfoBoxPlugin. It’s based on the equally-monikered infoBox in MediaWiki/Wikipedia, which you’ll see in any article that is marked “current event” or “controversial”, for example, on the big W. I find infoBoxes elegant, as they are unobtrusive enough to let you get on reading the article, and are easily ignored in the same way as web ads, but when you do focus on them, they are clear in meaning and support pattern recognition, with each type of infoBox having its own look and distinct icon.

    The infoBox macro makes infoBoxen that look as shown in the diagram above. In the simplest case, you type this into a tiddler:

    <<infoBox>>This is good stuff – pay attention mkay.>>

    Pretty straightforward. One funniness here is the bracket asymmetry. We’re trying to do something similar to XML tags, what with their start tag and attribs, followed by body, followed by closing tag, so you’d expect to see:

    <<infoBox>>This is good stuff – pay attention mkay.<</infoBox>>

    or sumptink like dat. But one of my lessons was that tiddlywiki has a somewhat unusual convention, enshrined only in the gradient macro to my knowledge, for the former syntax. The only way to do something else would be to refactor or replicate or hijack the core code, and I don’t fancy it, and in any event it would go against the standard already set by gradient. In any event, I was pleased enough when Jeremy showed me the example of the gradient tag, which shows that this kind of “macro body” is even possible, so I settled with that.

    Okay, so that’s nice, you can do a simple message, but where it gets more to the point is where you build up a family of infoBox types for your TiddlyWiki, each having a separate definition tiddler. For example, you will often want a “warning” infoBox to appear in various places. So you make a “warningInfoBox” tiddler, with the following text:

    |background|#fdd|
    |borderColor|#f66|
    |headingColor|#800|
    |heading|Danger, Will Robinson!!!|
    |messageFontStyle|normal|
    |messageColor|#900|
    |message|Please follow these instructions carefully.|
    |iconURL|icons##stop|
    |iconWidth|40|
    

    And then in a tiddler, you just write <<infoBox warning>>>> The association between “warning” and “warningInfoBox” is an enforced convention; my macro just appends “InfoBox” to the type you declare. The double-double closing bracket is a consequence of what I said above, combined with the fact that the definition has a pre-defined “message”. This is often going to be what we want, but not always. In fact, all of the fields above are optional, so you could leave out “message” and then the warning macro would have to specify it (if you wanted a message, that is). i.e. <<infoBox warning>>This Is Serious Mum.>> (… because the undeniably-talented, always-controversial, Aussie band This Is Serious Mum – aka TISM – popped into my head as an instructive example for this warning.)

    Some other things to say

    • This plugin is part of a greater effort to pull out the goodies from TiddlyGuv into reusable modules. TiddlyGuv was the first thing I worked on upon joining Osmosoft, and when I looked it at with fresh eyes recently, I realised how much of what I built is just generic TiddlyWiki functionality. I now have a better understanding of what makes an independent TiddlyWiki plugin, and indeed I have a much greater respect for the whole modularity concept in TiddlyWiki, because it really satisfies all the traditional software engineering principles of encapsulation, orthogonality, etc etc in a neat bite-size way, which I will have to explain more about elsewhere. infoBox is the first of several plugins that need to be exorcised from the TiddlyGuv base.
    • The “macro body” thing is actually quite easy – you just call wikifier.subWikify(domEl, ">>"); where domEl is some DOM element. This will keep reading from the end of the macro definition to the next occurrence of >>, and fill domEl with the results of wikifying that content. It would be nice if there was a function to let you just get the string, instead of sticking it in the DOM, but you could still achieve that yourself by shoving it into a hidden element and capturing its innerHTML. The only complication for me was a common one these days of jumping between JQuery-land and traditional DOM-land, since TiddlyWiki now ships with JQuery, but much of the infrastructure has not (yet) been retrofitted to talk JQuery. Once I did a JQuery-to-DOM conversion, it worked fine.
    • Another interesting detour that happened here concerned cross-referencing tiddlers. Something you will want often want to do with infoBoxes is use icons. In a single-file TiddlyWiki, often intended to be offline, the icons would need to be data: URIs. This applies to other plugins too, so I dedcided to look into it a bit. I wanted a way for the “iconURL” slice in the InfoBox definition to either be a regular URL, or a reference to another tiddler containing the data: URI, so as to isolate it away. This led me to learn more about transclusion. With thanks to @FND, and after some experimentation, I found the easiest way was to support linking to sections of another tiddler. See how it works in the demo tiddlywiki, where an “icons” tiddler contains the data:URI icons.

    Wow, I didn’t expect to say so much about what is not a major plugin. It comes at a time when I’ve been making some realisations about tiddlywiki, hence the verbeage.

    SimpleMessagePlugin: Unobtrusive TiddlyWiki Status Message

    To simplify TiddlyGuv message rendering, I made “SimpleMessagePlugin”. It removes the message box 1 second after a message was shown (using displayMessage). In the event another message appears in that time, it appends the message (as it normally does) and extends the message box’s lifetime by a second. In other words, it always closes a second after the last message was shown. The algorithm is a pretty similar throttling deal as Ajaxagram.

    Demo here.

    Latest version from SimpleMessagePlugin.

    FossBazaar at Open World Forum (OWF), Paris, Oct-2009

    Quick notes from today’s FOSSBazaar event at Open World Forum in Paris. Usual caveats on typos etc as I was writing these notes live.

    Martin Michlmayr overviews FOSSBazaar

    IMG_0069 (by mahemoff)

    Procurement – might like to avoid, but they have a really important role to play. With open source, it’s easy to ignore procurement – why go through procurement when you can download it! Great, that saved me several weeks of work, I can go home now :). But then you ship the code a few months later with GPL.

    Make FOSS “business as usual” – how acquired, chosen, used, supported, updated, project tracked, licensed, mature.

    FOSSBazaar to help people manage these issues. Good to have members outside of the US (Europe and maybe some now joining from Asia) as issues are different; this is the first FOSSBazaar meeting here.

    FOSSBazaar for experienced users – working together to define standards and best practices

    FOSSBazaar for inexperienced users – learning about open source issues, reducing the fear

    Discuss, resolve, and document the “hard” issues related to adopting FOSS in the enterprise.

    Question – why is HP spending money supporting the community? Martin explains HP has some experience dealing with compliance issues. Want to share experiences, e.g. we explained how we explored Palamida; other companies might do the same with Black Duck. It’s been a really good experience, talking to people who we might otherwise have not talked to.

    Open Source Compliance – View of Validos. Martin von Willebreand (lawyer basedin Finland).

    IMG_0070 (by mahemoff)

    Vaidos has DB of 130+ packages validated.

    “Shall we use this package or not?” Validos adds legal validation

    Shows DB report showing licenses being used in a package.

    Talking about typical guidelines, e.g. distributing the licenses with the code. Putting license on your website isn’t really redistributing it.

    Showing how they prepared a document for republication. It’s big, it’s scary, and it’s very likely necessary for legal purposes. [Kind of sad that open source gets this complex - makes me feel like just releasing all my code as public domain; losing attribution would be a shame, but the real downside would be the risk of liability if there's no warranty. It also makes me think that licenses should be embracing the web more, instead of forcing all this stuff to be repackaged - cut-and-pasting a URL would be a good start to what is a long path.]

    Martin comments FOSSBazaar has been thinking about standardising some of this license declaration stuff; and there was a parallel effort recently which he’s hoping to work with.

    IP Tracking Methodology at INRIA

    IMG_0074 (by mahemoff)

    On the forge: 1600 projects; 400 open source.

    Done various tracking and would be good to compare methodologies, work together on tools.

    Discussion session – Working on a Creative Commons contract to establish relationship with contractors, ensuring they use open source the right way.

    TiddlyGuv Tool (the bit with me in it)

    Steve Barnes motivated the tool from his perspective as a governance administrator. I then gave a demo and explanation.

    IMG_0081 (by mahemoff)

    Feedback:

    • Possible to get metadata e.g. to get XML feed. [Yes. Showed the bags/tiddlers model in the browser and how you can easily get a list of JSON or txt. Made the mistake of not demoing xml, which would have taken all of 5 seconds. Take off the pro-json blinkers next time!)

    • Can I see which packages (components) are using a paricular license?

    • What does a software policy look like? Do we need them at all. [Steve explained we probably wouldn't want to have a blanket policy like "don't use GPL", though others could; the tool lets you declare whatever policy you like.]

    • It should be more about training than official policies. [I explained TiddlyGuv is intended to be a tool for developers and governance authorities to work together, share information, leave comments, etc.; not a rigid policing-the-masses weapon]

    FOSSology

    IMG_0089 (by mahemoff)

    Disclaimer that you need more than just tools and need other tools too.

    Looks at every single file in a package – fuzzy match against a library of >400 known license.

    Walks through demo. Can try online.

    Many new features – email notifications, new licenses, tutorial section, cleanups.

    Plans – new license analyser (based on phrases); concept of license categorisastion (e.g. “good license”/”commercial license”)

    Question regarding international labour organisation – what happens if a child developed code and contributed it? Do they have the legal authority to donate the code etc. Is there a child labour question? Martin notes international treaties as a similar issue – notes you’re not allowed to distribute code from one country to another; how do you accept contributions across that boundary. It’s a potential FUD issue people can use against open source in government.

    Community Management and Project Governance: A checklist with an attitude Charles-H Schultz

    IMG_0094 (by mahemoff)

    Good “HowTo”/tutorial presentation of governance on a particular projct.

    Not just about making code available. but make it easy to find – e.g. when it’s on sourceforge, but not linked easily from project page.

    You never manage your community; so “community management” doesn’t exist. - fair, transparent rules and governance - soft power - trust employees like you trust other contributors

    TiddlyDocs User Authentication – Generic Design and Custom Features

    We are working on a design that will keep TiddlyDocs as generic as possible. The core TiddlyDocs product will have a concept of “rooms”. This is the same kind of room you see in online forums or chat sites, where everyone inside the room contributes and sees content inside the room they’re in, and can’t see or do anything about rooms they’re not in.

    Our design works like this. Each room is a recipe of several room-specific bags, along with a global config bag. Each room is associated with a unique user role.

    Note the role is used inside the respective bags’ policy files, and they each vary somewhat. e.g. the comments bag says readable by everyone in ‘role’, appendable (“create tiddler” permission) by everyone in ‘role’, editable by “#(role)_admin”. (ie there is a second role called “role_admin”.); whereas the content bags says readable and writeable and editable by everyone in ‘role’; and the config bag says readable by everyone in ‘role’, appendable by ‘#{role}_admin”.

    For now, there’s a “makeroom” script that generates the recipe, the bags and the policies. (I’d ideally like to see the recipe be “magic” in some respects, using a wildcard mechanism).

    That’s the basic design. We also have some unique challenges in this particular installation – (a) users are authenticated using a custom, enterprise-specific, single-sign on solution, where the incoming header for each request identifies user’s ID and (effectively) company; (b) we have certain rooms whose composition is manually configured and are a subset of the company’s membership, and we also have a “everyone” room which is automatically composed of everyone in the company (e.g. we might have a “megacorp-everyone” room along with the manually configured “megacorp-accounting” room, and “megacorp-marketing” room); (c) to let users allocate specific TiddlyDocs to users, we need a dropdown containing real names of all users.

    BTW I’m simplifying some of the finer details here to make the discussion more pertinent to the general topic of TiddlyWeb/TiddlyDocs, rather than our specific problem. All of the following will be developed as proprietary plugins for our own installation. It’s a somewhat complicated set of requirements because it’s a real-world enterprise problem, where we have specialised requirements, combined with integration issues where we can control some things, but can’t control others. (e.g. We can’t change the underlying user store, where we would ideally store which rooms a user is in, instead of holding secondary user records inside TiddlyDocs.)

    We will be building a TiddlyWeb plugin that sets the “extractor” filter. This is one of the several filters the request is subject to, on the way into TiddlyWeb. It is usually paired with the challenger filter – where challenger might present a username/password form and set a cookie, and extractor will read the cookie and set usersign (aka user ID) and any other info like users’ roles. In this case, though, we already have the header set, so we don’t need a challenger at all. We simply need an extractor that sets the usersign from the incoming header, and also notes the user’s company. Verification will then check that (a) if the user is attempting to access the “everyone” company room, they are a member of the company; (b) if the user is attempting to access a “manual” company room, they are a member of the company and they have the role associated with the company. We use a naming convention to determine which company a manual room belongs to.

    On (c), we need a tiddler in the room “config” bag containing a list of user names. So in addition to the custom extractor, we will also need a periodic script to poll the user store associated with the SSO system. This will give us all the info about users in each company, so we can build up a mapping from company to list of users, where each user record has an ID and name. company->list[{user ID, user name]}. The script will then (a) rewrite the list of user names for the “everyone room” based on the company list, by effectively “copying” the results for that company (b) rewrite the list of user names for each manual room, by walking through each user, checking their roles, and accumulating their user name into the room list for each room they’re in (c) do some cleanup – removing TiddlyWeb records for any users not in the list.

    There’s nothing very elegant about these custom implementations, and I’ve primarily written these details for the benefit of myself and colleagues who are implementing these custom features. The more interesting, general, point here is about open-source and plugins. TiddlyWeb’s architecture allows us deliver a generic solution for TiddlyDocs rooms, which is applicable to any organisation, and we can then superimpose a custom feature using the plugin mechanism. Furthermore, the plugin mechanism is a great way to decouple concerns – plugins are moduels on steroids. The core TiddlyDocs developer can concentrate on building an open-source, functional, multi-room TiddlyDocs edition, using the default authentication mechanism. Meanwhile, other developers can work in parallel on custom plugins.

    (This post is a refinement of this post; we have been doing further thinking as we get closer to the final product and we learn more about our external environment.)

    TiddlyWeb-JQuery Comments Plugin

    TiddlyWeb-JQuery Comments Plugin – Screencast @ Vimeo

    I’ve extracted from Scrumptious a nested comments plugin you can integrate into any POJA (plain ol’ Javascript app). You can find it here in the repo: TiddlyWeb-JQuery Comments Plugin.

    As the README explains, usage is a one-liner once you’ve set it up. Just do $(selector).comments(“topic”). The topic is an identifier for the set of comments; so that when the UI loads, it pulls down all comments with that ID in its “topic” field. Internally, topic acts as the root of the comments tree…but you don’t really need to know that.

    This plugin is a JQuery equivalent of the TiddlyWiki comments plugin, minus some of the more exotic options, and the ability to delete. Eventually, I hope to introduce those things, but only as the need arises – so if you’re using this, please let me know how I can improve it.

    Osmosoft Hackathon: WikiData, a Wiki of Companies Data

    End of The WikiData Hackday

    Osmosoft, Hackathons

    At Osmosoft, we have been engaging in a one-day hackathon about every month or so. There are several benefits:

    • It helps us prove our tech in a range of application contexts.
    • It helps improve and demonstrate our capabilities for reuse. Reuse can be as simple as slapping a Javascript file with inline comments on a server somewhere, or exposing a simple web service. With those capabilities in place, it’s possible to build something useful in a day, just as industry hackathons/camps demonstrate.
    • It helps us spread the word about web standards and why everything needs its own URI.
    • It helps us demonstrate and gain experience in agile processes.
    • It gets us Osmosofties working together in the same place at the same time, whereas day-to-day we tend to work on products in small groups or individually.

    The WikiData Hackathon

    The latest hackathon was last Thursday, July 16, which saw us collaborate with Avox, a company specialising in business entity data. I believe it’s working in a similar space to companies like Dunn and Bradstreet and other information providers, in that they collect, verify, and report information about companies. Avox is keen to open up parts of their database and gain the benefits of community feedback – with users being able to contribute information, leave comments, add new companies, and so on. Of course, Avox is in the business of gathering verified data, so it’s not just a case of making a new wiki and letting it run by itself. There remain challenges about how Avox will merge and verify community input, and how they will make it clear to users what’s verified and what’s not.

    Pre-Work

    We had a conversation on the mailing list about what we did last time and what could do differently for this hackathon. Like a retrospective, but over email. TiddlyWeb architect Chris Dent set up some instances with sample data taken from the existing product.

    Venue and Attendance

    The hackathon took place in Osmosoft’s office, centered around our big table in the centre of the room. Seven people from Avox, in a range of technical and non-technical roles, attended for the duration of the event. Osmosoft had five developers working on stories, a developer working on integration and mediation, and another helping with the server and offering second-line support.

    Introductions and Overview (about 1 hour)

    We went round the table and everyone introduced themselves. Jeremy Ruston explained “what’s in it” for Osmosoft, as outlined above, and Ken Price outlined Avox’s interest in building wikidata. We also looked at the existing product. We then had a discussion which was mostly Osmosoft people asking Avox people questions about the nature of their business, the technologies involves, and their vision for wikidata. Paul began writing down stories on flashcards during this time.

    User Stories (about 1 hour)

    We finished writing stories – this was somewhat in parallel to the previous activity – and put them all up on the wall, with the magic of Blu-Tac. The stories were in the form “As a (role), I want to (task), so that I can (benefit)”. With about 20 stories, it was useful to organise them, so we grouped them according to the role. The main roles were for Avox staff and general users. There were also some stories involving 3rd party developers and employees of companies listed in the wiki.

    Everyone gathered around the stories, we read them out, and we all agreed on priorities. We didn’t worry too much about ordering the stories at the bottom as it was unlikely we’d get to them during the event; if we did, we could prioritise later.

    What we ended up with was a solid idea of the steel thread we were building. It would mostly be standard wiki functionality, but applied to the particular data and context of companies info. We had some bonus stories we could implement if we had time, like comments and tagging.

    Planning and Infrastructure(about 30 minutes)

    Developers put their initials against the first story they’d be working on, and likewise each story needed a customer representative to initial the story, so they would help refine requirements. In the event, we didn’t talk all that much with customers during development; it’s obviously an extremely important thing to do in a real project, but when you’re wanting to get real functionality out in a day, the cost of a conversation is relatively high and the benefit to the task is relatively low. It would have been different in a 2-3 day event, or with certain stories that are highly specific to the domain. The main thing was to checkpoint with customers to check that we’re generally on track.

    Around noon, we drew up a timeline and agreed on a hard stop at 7pm. This means we timebox – the time is fixed and the only variable is the number of stories that get implemented in that time. We then got into

    Development Sprints and Standup Meetings (about 6.5 hours)

    We developed in sprints, with standup meetings in between. It was felt the previous hackathon’s hourly sprints were too frequent, so in this case the plan was every 1.5-2 hours; we decided at the end of each standup when the next would be.

    Most of us developed against the user stories. We also had a very useful role fulfilled by Fred, who was designated as a general “helper” and mediator. This was the result of our feeling that things were sometimes disjointed in previous hackathons – not enough focus on integration and infrastructure. I feel that this role was very useful and Fred fulfilled it admirably, although at times he probably felt like he had 400 people talking to him at once! We also had Chris Dent working remotely to ensure the TiddlyWeb server was running and help deploy to the live site.

    The tool we developed was a standard Javascript/JQuery web app (ie nothing to do with TiddlyWiki) talking to the TiddlyWeb server.

    At the start, we intended to write the web app and talk to the live server. But it soon became apparent that we would step on each others’ toes by doing this, and opted for a more conventional setup where we each have our own server instance. We also had a quick debate about version control – github, private osmosoft repository, or tiddlywiki repository. This is a recurring debate which we ought to decide in advance next time. It partly arose after deciding to run our local copies of the server; it was felt this would be too much data for the tiddlywiki repo, so we used the private osmo svn. As for github, it would be nice to use git, but most of us are more familiar with SVN as we use the tiddlywiki SVN repo day-to-day, so it would cause too much complication to switch to git. Again, it might be a good idea for next time to use github instead, with some pre-work.

    Presentation (about 45 minutes)

    After the usual last-minute rush, we got a working product deployed. It must be noted that we let the “hard stop” slip by about 45 minutes, to about 7:30. Admittedly a bad practice, but it did yield some major results in this case, as it got us search, edit, and maps all working during that time.

    Each of the developers explained what they’d worked on. We then gathered round the screen and walked through the app. Avox gave their thoughts, I recorded the video above while others broke out into informal conversations, and by that point, it was pub o’clock.

    We were able to produce the steel thread we’d been planning; the key stories were demonstrated. We also implemented commenting and Google Maps integration. Being based on TiddlyWeb means we had also produced a working API “for free”; it’s just the nature of TiddlyWeb as a RESTful data container. (On a personal note, commenting was the main thing I did – I extracted the comments feature from Scrumptious into a standalone plugin, and integrated it into the emerging WikiData product. I’ll post about that separately.)

    Wikidatathon

    Wikidatathon

    About the Video

    I recorded the video above at the end of the event. One of my bugbears about hackathon events is that people spend all day coding and setting up infrastructure, and it inevitably comes down after that, or gradually falls apart as various third-party services close, databases get corrupted, people forget to keep hosting it, etc etc. In other words, you have to assume the online result of any hackathon event is transient. This is unfortunate, because the deliverable should be something that people can use to decide what happens next, whether to fund a project, and so on. While meeting minutes are often a waste of time, the artifacts that emerge in a workshop are critical to followup.

    For that reason, I am perfectly passionate about taking a screencast or video to capture what took place in such events. Thanks to Paul and Ken for taking part in it.

    Update: Avox CEO Ken Price (who appears in the video), has published a summary of the event.