WebWait Two Point Oh

I’m pleased to announce a major upgrade to WebWait, the first big upgrade since I launched the site 2.5 years ago. It’s based on watching how people are using it, talking about it, and sending me direct feedback about it. Thanks to all who have provided feedback.

The new features are:

  • Multiple sites You can keep typing in new URLs and hitting “Add”. This will put them in a queue of sites to eventually be benchmarked. (I decided against a big textarea where you could type in all URLs at once; this would be confusing to the majority of WebWait users, who are casual visitors wanting to check out the time for their own site.)
  • Stats Average, median, and standard deviation for each website, in a summary table. Indeed, the summary table is the cultural centre for all these new features – when you queue up a site, it’s immediately added to the table.
  • Export The export feature seems to be popular in List Of Tweets. I took the same UI concept and applied it here, so you can export data in plain text, HTML, and CSV format. The CSV is especially interesting as one of the people who’ve given me feedback is a statistician who’s planning to run analysis on some data.
  • Call Data View times for each individual call.
  • Expand/Collapse Call Data You can choose whether to show call data or not. Collapsing call data gives you a neat overview of stats for each domain.
  • Delete You can delete individual call data, or delete an entire website record. This is useful for outlier data; for example, you can see how fast the site loads from your browser cache, by turning browser cache on and deleting just the first call in the sequence.
  • Stats Average, median, and standard deviation for each website.
  • Browser compatibility. WebWait didn’t work upon launch in Safari, but it does now work in recent versions of Safari (I think a change in 3.0 made it work again :)), and works fine in Chrome too. This is important for users testing how their website loads in different browsers.

Thanks for reading. As always, let us know what you think of the site and anything else you’d like to see.

Table with expanded results:

Table with collapsed results:

Export results data (text):

Export results data (HTML):

Export results data (CSV):

TiddlyWeb Now Reloads Code

I’ve begun working on the server-side of TiddlyWeb a bit, mostly to make stuff degrade gracefully as I begin to build a Trails facility into Scrumptious. In TiddlyWeb, an app-specific plugin is how you build an app on the server-side (or you can use a completely separate web framework like Django or Rails, making server-to-server RESTful calls, but that’s another story).

I asked about reloading code on the group this week. I want to change my Python code and hit reload in the browser and see the changes. To me, this this kind of “hot deploy” during development is a must-have for any web framework. There’s an argument that testing would render it unnecessary, but (a) it’s difficult to write tests when you’re learning and exploring an environment; (b) kool-aid aside, some things can’t be tested or are Test-Driven Design is simply not the most productive way to develop it (it comes down to forces – a cost-benefit analysis); (c) functional and integration tests, working across the whole server, and reloading code on each request is an easy way to make those happen.

Anyway, Chris has kindly produced the necessary pieces.

Here’s how I got it working. Here’s what I did to get it working on Mac, which is slightly different from Linux setup:

  • Download reloader.py and wserver.py stick them in top-level tiddlyweb instance directory.
  • Introduce the following to your tiddlywebconfig.py: 'twanager_plugins': ['wserver']. Note – do not try to use reloader as a plugin; it just needs to be in your directory as wserver uses it.
  • Run the server using twanager wserver.

If you have browser caching on, you might need to hit shift-reload at times. I don’t have caching on and it works fine with just reload. This is a big help, thanks Chris.

TiddlyWeb-JQuery Comments Plugin

TiddlyWeb-JQuery Comments Plugin – Screencast @ Vimeo

I’ve extracted from Scrumptious a nested comments plugin you can integrate into any POJA (plain ol’ Javascript app). You can find it here in the repo: TiddlyWeb-JQuery Comments Plugin.

As the README explains, usage is a one-liner once you’ve set it up. Just do $(selector).comments(“topic”). The topic is an identifier for the set of comments; so that when the UI loads, it pulls down all comments with that ID in its “topic” field. Internally, topic acts as the root of the comments tree…but you don’t really need to know that.

This plugin is a JQuery equivalent of the TiddlyWiki comments plugin, minus some of the more exotic options, and the ability to delete. Eventually, I hope to introduce those things, but only as the need arises – so if you’re using this, please let me know how I can improve it.

Osmosoft Hackathon: WikiData, a Wiki of Companies Data

End of The WikiData Hackday

Osmosoft, Hackathons

At Osmosoft, we have been engaging in a one-day hackathon about every month or so. There are several benefits:

  • It helps us prove our tech in a range of application contexts.
  • It helps improve and demonstrate our capabilities for reuse. Reuse can be as simple as slapping a Javascript file with inline comments on a server somewhere, or exposing a simple web service. With those capabilities in place, it’s possible to build something useful in a day, just as industry hackathons/camps demonstrate.
  • It helps us spread the word about web standards and why everything needs its own URI.
  • It helps us demonstrate and gain experience in agile processes.
  • It gets us Osmosofties working together in the same place at the same time, whereas day-to-day we tend to work on products in small groups or individually.

The WikiData Hackathon

The latest hackathon was last Thursday, July 16, which saw us collaborate with Avox, a company specialising in business entity data. I believe it’s working in a similar space to companies like Dunn and Bradstreet and other information providers, in that they collect, verify, and report information about companies. Avox is keen to open up parts of their database and gain the benefits of community feedback – with users being able to contribute information, leave comments, add new companies, and so on. Of course, Avox is in the business of gathering verified data, so it’s not just a case of making a new wiki and letting it run by itself. There remain challenges about how Avox will merge and verify community input, and how they will make it clear to users what’s verified and what’s not.

Pre-Work

We had a conversation on the mailing list about what we did last time and what could do differently for this hackathon. Like a retrospective, but over email. TiddlyWeb architect Chris Dent set up some instances with sample data taken from the existing product.

Venue and Attendance

The hackathon took place in Osmosoft’s office, centered around our big table in the centre of the room. Seven people from Avox, in a range of technical and non-technical roles, attended for the duration of the event. Osmosoft had five developers working on stories, a developer working on integration and mediation, and another helping with the server and offering second-line support.

Introductions and Overview (about 1 hour)

We went round the table and everyone introduced themselves. Jeremy Ruston explained “what’s in it” for Osmosoft, as outlined above, and Ken Price outlined Avox’s interest in building wikidata. We also looked at the existing product. We then had a discussion which was mostly Osmosoft people asking Avox people questions about the nature of their business, the technologies involves, and their vision for wikidata. Paul began writing down stories on flashcards during this time.

User Stories (about 1 hour)

We finished writing stories – this was somewhat in parallel to the previous activity – and put them all up on the wall, with the magic of Blu-Tac. The stories were in the form “As a (role), I want to (task), so that I can (benefit)”. With about 20 stories, it was useful to organise them, so we grouped them according to the role. The main roles were for Avox staff and general users. There were also some stories involving 3rd party developers and employees of companies listed in the wiki.

Everyone gathered around the stories, we read them out, and we all agreed on priorities. We didn’t worry too much about ordering the stories at the bottom as it was unlikely we’d get to them during the event; if we did, we could prioritise later.

What we ended up with was a solid idea of the steel thread we were building. It would mostly be standard wiki functionality, but applied to the particular data and context of companies info. We had some bonus stories we could implement if we had time, like comments and tagging.

Planning and Infrastructure(about 30 minutes)

Developers put their initials against the first story they’d be working on, and likewise each story needed a customer representative to initial the story, so they would help refine requirements. In the event, we didn’t talk all that much with customers during development; it’s obviously an extremely important thing to do in a real project, but when you’re wanting to get real functionality out in a day, the cost of a conversation is relatively high and the benefit to the task is relatively low. It would have been different in a 2-3 day event, or with certain stories that are highly specific to the domain. The main thing was to checkpoint with customers to check that we’re generally on track.

Around noon, we drew up a timeline and agreed on a hard stop at 7pm. This means we timebox – the time is fixed and the only variable is the number of stories that get implemented in that time. We then got into

Development Sprints and Standup Meetings (about 6.5 hours)

We developed in sprints, with standup meetings in between. It was felt the previous hackathon’s hourly sprints were too frequent, so in this case the plan was every 1.5-2 hours; we decided at the end of each standup when the next would be.

Most of us developed against the user stories. We also had a very useful role fulfilled by Fred, who was designated as a general “helper” and mediator. This was the result of our feeling that things were sometimes disjointed in previous hackathons – not enough focus on integration and infrastructure. I feel that this role was very useful and Fred fulfilled it admirably, although at times he probably felt like he had 400 people talking to him at once! We also had Chris Dent working remotely to ensure the TiddlyWeb server was running and help deploy to the live site.

The tool we developed was a standard Javascript/JQuery web app (ie nothing to do with TiddlyWiki) talking to the TiddlyWeb server.

At the start, we intended to write the web app and talk to the live server. But it soon became apparent that we would step on each others’ toes by doing this, and opted for a more conventional setup where we each have our own server instance. We also had a quick debate about version control – github, private osmosoft repository, or tiddlywiki repository. This is a recurring debate which we ought to decide in advance next time. It partly arose after deciding to run our local copies of the server; it was felt this would be too much data for the tiddlywiki repo, so we used the private osmo svn. As for github, it would be nice to use git, but most of us are more familiar with SVN as we use the tiddlywiki SVN repo day-to-day, so it would cause too much complication to switch to git. Again, it might be a good idea for next time to use github instead, with some pre-work.

Presentation (about 45 minutes)

After the usual last-minute rush, we got a working product deployed. It must be noted that we let the “hard stop” slip by about 45 minutes, to about 7:30. Admittedly a bad practice, but it did yield some major results in this case, as it got us search, edit, and maps all working during that time.

Each of the developers explained what they’d worked on. We then gathered round the screen and walked through the app. Avox gave their thoughts, I recorded the video above while others broke out into informal conversations, and by that point, it was pub o’clock.

We were able to produce the steel thread we’d been planning; the key stories were demonstrated. We also implemented commenting and Google Maps integration. Being based on TiddlyWeb means we had also produced a working API “for free”; it’s just the nature of TiddlyWeb as a RESTful data container. (On a personal note, commenting was the main thing I did – I extracted the comments feature from Scrumptious into a standalone plugin, and integrated it into the emerging WikiData product. I’ll post about that separately.)

Wikidatathon

Wikidatathon

About the Video

I recorded the video above at the end of the event. One of my bugbears about hackathon events is that people spend all day coding and setting up infrastructure, and it inevitably comes down after that, or gradually falls apart as various third-party services close, databases get corrupted, people forget to keep hosting it, etc etc. In other words, you have to assume the online result of any hackathon event is transient. This is unfortunate, because the deliverable should be something that people can use to decide what happens next, whether to fund a project, and so on. While meeting minutes are often a waste of time, the artifacts that emerge in a workshop are critical to followup.

For that reason, I am perfectly passionate about taking a screencast or video to capture what took place in such events. Thanks to Paul and Ken for taking part in it.

Update: Avox CEO Ken Price (who appears in the video), has published a summary of the event.

The TiddlyWeb Request-Response Lifecycle

We had a session a little while ago where Chris Dent walked through the lifecycle of a Tiddlyweb call, from incoming request to outgoing response.

Each step is a WSGI middleware object. The exact sequence is of course configurable, by manipulating config['server_response_filters'].

Some Terminology

First, some relevant terminology.

Serializer – converts from a tiddler to some format (externalisation), some format to a tiddler (internalisation), or both directions.

Challenger/Extractor – runs through extractors until it finds a valid user. If none found, user is GUEST. User is just a string identifier, doesn’t include a realm in the default cases. Note that TiddlyWeb ships with the basic cookie form challenger and also the open id challenger. If you were to create a basic cookie user called “http://mahemoff.myopenid.com”, you would be causing trouble – two different identities with the same ID, so the rest of TiddlyWeb sees them as the same thing. It’s up to the site operator to configure things right, add the right constraints etc.

Validator – checks content

Filters – modifies the list of tiddlers. select – take a subset of tiddlers, sort – order the tiddlers, limit – limit the number of tiddlers. can also add other keywords via plugin.

“environ” is a “semi-global” – ie it’s a request-scoped variable, ie it’s available to to every stage of the request.

“Incoming” Filters

IN: (each of these is WSGI middleware)

Environator. Looks at WSGI environment. This includes all the CGI goodies – path, user agent, etc; and also anything else we want to control, and we make use of the extra stuff. (~empty right now)

Configurator. Reads config. Sets up environ['tiddlyweb.config'] – simply translates the global config (which was read at server start time) into the request. (For a development mode, it would be possible to make a Configurator that reads the config from file each time.)

Query. Deals with incoming query. Can get original query string as environ['QUERY_STRING'], but this stage parses it to make it higher level. environ['tiddlyweb.query']['fat'][0]. If request method is POST and content type is CGI form, it will parse and place into envron['tiddlyweb.query'] (and if the POSTed message also contained query params, they’ll be merged there).

StoreSet. Sets environ['tiddlyweb.store'] to the actual store object (not just a string like ‘sql’ or ‘text’). (Runs something like environ['tiddlyweb.store'] = Store(‘text’….). (where Configurator has setup this kind of store.)

UserExtract. Determines who current user is. environ['tiddlyweb.usersign'] = [ 'username': '..', 'roles': []]. Checks for user via cookie and MAC check. (Doesn’t actually do authentication/login, the challenger is separate.)

Header. Is only here to satisfy “HEAD” requests – calls self as if it was GET and throws out the body.

Negotiate. Determines serialiser based on query string and accept header.

Selector. (unlike all the others, this is not TiddlyWeb, but a separate Middleware package.) Dispatches handlers based on URL pattern, just like Rails routes. urls.map is the default mapping and you would typically extend it (as opposed to replacing it) using something like config.selector.add(‘/bookmarks/{bookmark_id}.{format}’, GET=bookmark_view).

“Outgoing” Filters

HTMLPresenter – top and tail the output. This is only entered iff the content type is HTML.

PermissionsExceptor

HTTPExceptor

EncodeUTF8

SimpleLog

Instalicious: Push Delicious “ToRead” Items Into Instapaper

Update: Moved instalicious to GitHub. The main change I made since posting this was to stop items reappearing in Instapaper once you’ve deleted them; the sync process was causing them to be pushed from Delicious again. The script now changes the “toread” tag to “instaliciousd” (configurable name), so “toread” items will only be pushed once.

Instapaper lets me read stuff later – I hit a bookmarklet when I’m reading it and I can see it later on my browser or in iphone, or print articles in a newspaper format. The problem is, I’d rather add that stuff to my Delicious stream so I have a permanent record, others can see it, and it plays nicely with in mashups.

So I decided I want a way to bookmark things in Delicious as normal, but still have them appear in Instapaper. Here’s the script repo:

Instalicious repository at GitHub

Instalicious is a little Python script that plucks out items tagged “toread” – or some other configurable tag – and pushes them to Instapaper. Fortunately, both services have easy APIs (Instapaper API; Delicious API), and in the case of Delicious, there was even a pleasant Python binding to the API. Also, the Instapaper API happily doesn’t create duplicates when you send it something you’ve already sent…so Instalicious doesn’t have to remember what it sent. It just dumbly polls Delicious and sends it over to Instapaper.

Now you can have your cake and eat it :).

List Of Tweets

ListOfTweets.com is a little twitter mashup I made to organise tweets. The tool lets you build up a list of tweets from search results, which you can convert to HTML or text. The list doesn’t empty unless you clear it, so you can keep running searches and adding to the list. You can easily rearrange tweets with drag-and-drop.

By the way, List Of Tweets is another example of a URL Trail. Thanks to the simplicity of the Twitter API and the elegance of JQuery, it was possible to build a raw version in a couple of hours. t is approaching zero. Only after tweeting about it and discovering people found it useful did I decide to flesh it out some more.

A future upgrade might involve saving the list at a unique URL. It would be nice if any of the pastebins provided some support for this via a JSON API (admittedly with relatively short URLs – only up to 2000 characters could be reliably posted). But I will probably build my own, or just start a new frame and post into a pastebin.

I originally wanted this as I’ve been thinking about podcasting again and wanted to pull out some highlights from my recent tweets, to act as a schedule, and something I could publish as a playlist at the end. However, a colleague suggested a far more useful scenario, which is to find tweets matching a keyword, hence I added the search keyword parameter, which I now use much more often than the tweeter parameter. Here’s an example of someone using it in their blog to archive tweets about a hashtag.

I’ve already let people know about the tool via Twitter, but it feels incomplete until I blog about it. Done!

Making a Bookmarklet: A Challenge on All Fronts

Scrumptious, a social bookmarking/comments app I began building recently, has a bookmarklet.

The bookmarklet is added from the demo homepage.

It’s not the first time I’ve made a bookmarklet for a website, the first time being WebWait.

I found the Scrumptious edition more challenging though, for several reasons. This brief article outlines the challenges. Some I’ve solved to an extent, but all need further work.

Launch UI Challenge

The most obvious challenge is, how do you explain to users what they can do with a bookmarklet. Most users aren’t geeks and I suspect most people still haven’t heard of bookmarklets. Or what are sometimes called “favelets”, since IE users have favourites, not bookmarks, which further highlights the problem. Here, I’ve explained with some text, but it ideally needs a diagram and perhaps a link to a separate page with a more detailed explanation, alongside a video.

Overlay UI Challenge

The next obvious challenge is UI design of the comments overlay. We’re overlaying comments onto a page. One of the issues was using a z-index that’s higher than what’s on the site, hence my article on max z-index. Other questions too. How much transparency, if any? What kind of positioning and scrolling, and how wide should it be?

Coding Challenge

Bookmarklets need to be fairly small. IE6 limits you to just 508 characters, so if you are doing anything remotely interactive, you’ll need to make the bookmarklet pull in a script from elsewhere, using On-Demand Javascript. This is a good practice anyway (as bookmarklet guru Alex Kirk explained to me a while ago) because it means you can always update the app. The downside is you still have to be quite concerned about size, as you want the bookmarklet to come into effect straight away. For this reason, I wrote the Scrumptious bookmarklet without using JQuery, which meant going back to Javascript 101, and lots of ugly low-level DOM manipulation therein.

Deployment Challenge

While there are various websites that let you cut-and-paste Javascript code to build a bookmarklet, I wanted to automate the process. Continuous integration, Automated Deployment, and all that. Luckily, John Gruber has a bookmarklet builder perl script. I yoinked and tweaked it so that I could keep the Scrumptious bookmarklet in its own form.

Another issue is the URL of the Javascript which the bookmarklet requests. Since Scrumptious can run on different servers, this URL will change depending on where the server is stored. Right now, there’s no server-side processing – all server-side content is static HTML. So I wrote a shell script to set the bookmarklet which any site operator could use.

Modularity Challenge

A bookmarklet is not the only place Scrumptious injects itself into the page. There’s also a Greasemonkey script under construction, and I’ve experimented with JetPack. I’ll also be building support for a site toolbar, so that an intranet operator could let you click on a “comments” link and inject Scrumptious comments on the page (so that users don’t have to bother installing the bookmarklet, at least to comment on intranet sites). The bookmarklet script has to be flexible enough to cover these subtly different scenarios. For example, the Greasemonkey script wants the comments overlay to be closed initially with a small button to launch it; the bookmarklet wants it to be open initially; and the Jetpack and toolbar want it to be off altogether until its explicitly launched.