Digg API – Can’t Bust the Cache

It’s often a requirement for an Ajax app to “bust the cache”, i.e. call a service and ensure its response comes direct and not from a cache. For all the talk of fancy header techniques, the easiest way to do it is by appending an arbitrary parameter, typically a random number and/or a timestamp i.e. call service.com/?random=39583483. I use this pattern not just as an Ajax programmer but also as a user; when I suspect my wiki page or blog is cached, I just add an arbitrary parameter and usually get the most recent result.

With Digg API, that’s impossible. I just tried it and discovered you get an unrecognised argument error if you pass in a parameter they weren’t expecting. This is well-intentioned as it will give early failure advice to callers who mistype or misinterpret parameter names. But unfortunately, it has the major downside of breaking this convention and thus breaking compatibility with a large number of toolkits that exploit this pattern.

I discovered this as I was writing a Google Gadget to show Digg stories and when you fetch content with the Google API _IG_FetchContent("http://digg.com/....", callbackFunc, { refreshInterval: 60 }) – the refresh option apparently leads to Google tacking on an arbitrary parameter called “cache” (it’s impossible to be sure as it’s proxied via Google, but the argument to that proxy is the Digg URL with “cache=” on the end). Net effect: I can’t update Digg stories any more than once per hour. The only way I could do it would be to write my own proxy and cache. A lot more work than the one-liner _IG_FetchContent call! Yeah, like that’s gonna happen.

For the Digg API, perhaps they need to introduce an optional “strict” parameter like a compiler offers to give extra debug info. If it’s off, then let anything through.

New Website: Web-O-Random

I made a new Ajax toy. Web-O-Random is a random website finder with some Ajaxy goodness:

  • Sites load into a carousel (slider) component. Thanks to Bill Scott and Yahoo! for providing the original component and Sebastien Gruhier for the prototype port which is used here.
  • Pages load into an IFrame (as with WebWait).
  • User can re-randomize, which leads to a JSON payload of new random sites.

If you hunt for it, you’ll find there’s a way to search too, but I had to de-emphasise it as mysql full-text search takes 1-2 minutes to crunch through 4M (indexed) rows (each representing a website). If I have time, I may set up a reverse index table to cache results.

Initial reactions:

  • “…?” This is the typical reaction. It may be interpreted as a polite version of Cartman’s “Dude, this is pretty **** right here?” as in “Who would ever use this?”. Sometimes people forget that not everything has to be as popular as Google…apps like Weborandom and Webwait can be coded in a few days and hosting costs are pretty trivial nowadays. As development time trends downwards to zero, it’s the long tail of web apps! See What Happens When t Approaches Zero?
  • “So I get a new site by clicking Randomize?” (You’re meant to use the carousel first. I added some initial animation to the carousel, which looks cool and shows people there’s more than meets the eye. I also made the carousel default to start in the middle instead of on the left.)
  • “Are those your ads?” (He thought the carousel was adsense. I refactored by showing a boldfaced website title instead of a blue link).
  • “How do I click the arrow key? I can’t see the cursor.” (Confusing to have keyboard bindings in the absence of a textarea/input control. Changed the instruction text.)

When Tags Fail

Here’s a view of upcoming events in London at this time.

Three closely related topics with a number of web-savvy taggers (if attendees of events like this can’t tag, who can?). And yet, zero tags in common.

It would be very compelling to subscribe to events in your area and under a certain topic, so they automatically appear in ICal, GCal, etc. But the topic would have to be general enough to provide more than one event a year, and chances are there won’t be many “microformats” gigs in your local area on a regular basis. i.e. what’s obviously required is a tag like “geek” or “web” that’s general enough to be useful for the use case of a calendar subscriber.

Perhaps reflecting the tendency for people to think in concrete terms, the taggers here opt for very precise, specific, terms. The conventional wisdom on tagging is that people will come up with their own conventions and systems should stay out of it, but isn’t that a rather defeatist stance on our ability to create remotely intelligent software in 2006? Rather than force users to go against their nature and think abstractly, maybe the system could go all fuzzy and do some useful cluster analysis. I’d rather subscribe to a slightly inaccurate “web-ish” cluster than 20 different feeds covering every concrete concept related to the web. Granted, it’s also more work to develop, but it’s something that could easily be encapsulated in a library.

Ajax/Javascript: 8 Ways to Create Graphics on the Fly

<a href='http://digg.com/programming/Ajax_Javascript_8_Ways_to_Create_Graphics_on_the_Fly'>Ajax/Javascript: 8 Ways to Create Graphics on the Fly &#8211; digg this</a>

The ability to create rich graphics on the fly is one of the critical gaps in Ajax. There are indeed techniques to do it, albeit far from perfect, and some are do-able today if you take a pragmatic view of things and keep graceful degradation in mind.

Here I list all the techniques I can think of to create graphics and images dynamically when you’re creating an Ajax/DHTML/Javascript web app, with varying degrees of portability and usability. Starting with the obvious and ending with the obscure. Please submit any other techniques you’re aware of!!!

  1. Use SVG. Current versions of Firefox, Opera, and Safari (nightly builds) support SVG natively, but with IE and older versions, users need to install a plugin. SVG is an old W3C standard that creates images based on XML. Being a vector format, it stores the image at a high level (curves, lines, etc), so it scales better than a plain bitmap. Here’s an SVG circle (adapted from W3CSchools:

    1. <svg width="100%" height="100%" version="1.1"
    2. xmlns="http://www.w3.org/2000/svg">
    3. <circle cx="25" cy="25" r="30" stroke="black"
    4. stroke-width="2" fill="green"/>
    5. </svg>

    You could build up the XML string in the code or pull it down from the server with a remote request. However, you don’t actually have to specify the XML as a literal string message; you can create a blank SVG document object model (DOM) and manipulate it to build up an image. In this example, we create a circle of radius 25 (adapted from this tutorial):

    javascript
    < view plain text >
    1. var svgDocument = evt.target.ownerDocument;
    2. var shape = svgDocument.createElementNS(svgns, "circle");
    3. shape.setAttributeNS(null, "cx", 25);
    4. shape.setAttributeNS(null, "cy", 25);
    5. shape.setAttributeNS(null, "r",  20);
    6. shape.setAttributeNS(null, "fill", "green");
  2. Use Canvas. Canvas was introduced in Safari and now in Firefox and Opera too. No sign of life in IE, but this there’s a fantastic adaptor hack (via Ajaxian) that emulates Canvas using IE’s native VML support (more on VML below), and it’s now been rolled into an open-source project, ExplorerCanvas by some Googlers.

    Where SVG is about things, Canvas is about actions. SVG documents represent images. Canvas tags include code to build up an image, a bit like moving a turtle in the Logo language. So you set a colour, draw something, change the fill style, draw something else, etc. You manipulate a Canvas tag like this:

    javascript
    < view plain text >
    1. var canvas = document.getElementById('tutorial');
    2.   if (canvas.getContext){
    3.     var ctx = canvas.getContext('2d');
    4.     ctx.fillRect(25,25,100,100);
    5.     ctx.clearRect(45,45,60,60);
    6.     ctx.strokeRect(50,50,50,50);
    7.   }

    (BTW Canvas uses the dreaded term, “Context”, to refer to what should really be called “paintbrush”, “pen”, or “turtle” … more concrete terms than “Context” which imply some sort of metaphor. But wouldn’t an imperfect metaphor be easier to grok than the generic “Context”? Alas, it’s a common idiom in graphics programming and will be around for a while.)

  3. Load dynamic images from the server. Once you have an image tag (either in the initial HTML or created on the fly with document.createElement(“img”), you can set its source to any URL, even external to your domain (though cross-domain “hot-linking” should generally be done only with permission). This is standard DHTML/Ajax stuff and works with any browser.
    javascript
    < view plain text >
    1. button = document.createElement("img");
    2. button.src = "http://example.com/livechart.gif?interval=15"
    On the server, the image need not be a static image file. It can be a server-side script that happens to output a binary image with the appropriate header, and furthermore, the script can accept CGI parameters, so a unique image can be generated on the fly. Of course, performance will probably suffer if you rely on the server to create images on the fly, and you can only generate them once a second or so. The various Ajax image manipulation tools do something like this.
  4. Use Vector Markup Language (VML). VML is effectively the MS equivalent of SVG, and as such works in IE, and only in IE. As with SVG, you use XML to specify an image.
    1. <v:oval style="position:absolute; left:0; top:0;
    2.               width:100pt; height:50pt"
    3.               fillcolor="red">
    4. </v:oval>
    Fortunately, the Canvas-on-VML hack mentioned earlier means you can take advantage of VML without committing to IE-specific code, though it’s (obviously) not a perfect emulation by any means.
  5. Introduce a Richer Plugin, most likely Flash, to beef up the browser’s multimedia capabilities.
  6. Rely on plain old CSS and the DOM. You can do some impressive-looking things with just CSS and the DOM, e.g. the CSS graphs library. And of course it’s easy enough to make it fully portable too.
  7. Create an image and set its src to a data: resource. Firefox and Opera only. From Dive Into GM:
    javascript
    < view plain text >
    1. var logo = document.createElement('img');
    2. logo.src = 'data:image/gif;base64,R0lGODlhDQAOAJEAANno6wBmZgAAAAAAACH5BAAAAAAA'+    'LAAAAAANAA4AQAIjjI8Iyw3GhACSQecutsFV3nzgNi7SVEbo06lZa66LRib2UQAAOw%3D%3D';
    3. document.body.insertBefore(logo, document.body.firstChild);
  8. Embed an XBM file. Yes, some browsers can display XBM images. Works on IE and Firefox, but not Safari or Opera. Unfortunately, XBM has the rather major constraint that it’s black-and-white. However, it does have the virtue of being a plain-text format which, like the data: protocol, you can assign an image source to.

    If you get your image data in that format in a string (complete with the n after each of the #define lines), then you can make any image’s SRC attribute be:
    “javascript:’”+xbmimagestring+”‘”

    XBM’s not used often, but there are some nice examples: Wolfenstein 5K, most notably, as well as this bitmap editor and a sparklines library.

(Off-topic: I’ve just updated my blog page, I prefer the 2-column sidebar because: (a) there are now 20 monthly archives links; (b) I wanted to add a ton of chicklets; (c) I wanted to add more bio info. BTW If you have a blog, here’s a quick exercise: Place yourself in the mind of a new visitor for twenty seconds, and decide if this person could work out who you are, what you do, and how to contact you. Around 90% of blogs fail this test on the grounds it’s impossible or way too hard.)

Ajax Functionality and Usability Patterns – Podcast 4 of 4: Functionality Patterns

This is the fourth and final podcast in the series on Ajax functionality and usability patterns (Book: Part 4, pp 327-530). This 54-minute podcast covers seven patterns of Ajax Architecture (Book: Chapter 17, pp 473-530):

Dedicated to the Nitobians, whose last podcast inspired me to crank another one out again. Recent events suggest it may cost me $5000 to appear on their podcast again, and as Andre points out in this podcast, the same applies for them appearing on my podcast. Thus, my simple proposal would be:

  1. Each of us appear on the others’ podcast, at $5000 each. Actually, let’s make that $50k each.
  2. Cancel the debt
  3. Now each of us can claim our podcast attracts guests who pay $50k to appear. Enough to cover headsets ($20), bandwidth ($10/month with Libsyn), and assorted beverages (name your price).
  4. Profit!!!

Soon I’ll be publishing the final podcast in the overall series, which has already been recorded, and then I’ll be taking it in a more general direction akin to the topics on this blog – talking about agile, programming (Java/Rails/etc), usability, Web2.0, as well as Ajax and the coming revolution of real-time webapps. If you have a skype account and you’d like to join me sometime, drop us an email ([email protected]). Also feel free to suggest any topics that would be good to cover.

“Ajax Design Patterns” – Book of the Month

Ajax Design Patterns is Book of the Month in this month’s .Net mag (p.23, Issue 155, October, 2006). Incidentally, the mag is about the ‘Net, not specifically MS .Net (which it pre-dates).

<

p>The review says:

So AJAX might be the hottest thing in programming since, er, ordinary Javascript, but it’s no good just learning how to implement it – you need design inspiration too. Ajax Design Patterns fits the literary void that exists in AJAX design by using real examples of best practice to enhance your apps.

 

<

p>I’m glad they emphasise use of real examples, because we can debate ad infinitum about whether everything in the book is a pattern or not, but the more important thing is that the examples are real, concrete, and as accessible as typing a URL into your browser.

Thankfully, Ajax Design Patterns is one of the most organised books on any programming subject. It’s a massive book, but you won’t get lost as the chapters are sensibly divided up and the sound layout means there’s nothing whatsoever to fear.

<

p>I’ve had a lot to say about presentation of patterns in the past The fairly unusual presentation of the patterns is the reason it’s not an O’Reilly animal book, and it’s good to see it helped.

Odeo: Engineering Against Customer Loyalty

GigaOM discusses “How Odeo Screwed Up”. Odeo is a service I want to like. I promoted it to others when it came out and I frequently use it as an example of the Richer Plugin pattern as it uses an effective combination of Flash and Ajax.

However, I had to stop using Odeo six months ago, due to an astounding oversight in their central architecture for doing what they are meant to do best: manage podcasts. The problem is simply that Odeo places all podcasts into a single RSS feed. You can probably imagine the consequences.

The feed inevitably grows and grows, and suddenly you have 5000+ multimedia items for your podcatcher to fetch and sync. ITunes simply gives up. Juice/IPodder hangs for about 30 minutes and might then start updating if the stars align with the moon. Under some circumstances, like after a (likely) crash leading to a corrupt record of what you’re downloading, Juice will start downloading all 5000 podcasts.

So the architecture is kind of flawed from the get-go. A single ever-growing feed. What tools does Odeo offer to tame it? Well, you can manually chop the beanstalk down one podcast at a time. This used to be rather difficult because of overzealous use of a Yellow Fade Effect, which meant you could only delete an item every two seconds or so. Now, Odeo offers a checkbox-driven interface, but you must still manually click on each checkbox, you only have 25 checkboxes per page, and there’s no keyboard shortcuts possible in Firefox AFAICT. So still unpractical. And the beanstalk keeps growing as you slowly chop it down at the other end.

What’s blatantly obvious is that Odeo needs an auto-delete feature, e.g. delete feeds older than 1/3/7/30 days or keep a maximum of 10/100/1000 podcasts in your feed. It’s such an obvious thing, it’s almost breathtaking that it doesn’t exist. I keep double-checking as I write this, but the fact is that I’ve previously mailed support about it and there’s no such feature. I don’t really understand what’s going on, as it’s hardly a niche request; it’s something that affects every users.

Odeo will work fine for new subscribers, but as soon as you’ve been subscribed for a few months, it’s impossible to use. I mean impossible! As I say, I like the website a lot and I wish there was a way to get it working. But it’s simply not possible. Way to encourage loyalty! I can understand when this happens with a small startup, but Odeo is high-profile, VC-funded, and continues to roll out completely orthogonal products like Twitter, Hellodeo, and podcast recording. Meanwhile, their core feature has remained impossible to use for twelve months!

The GigaOM article points to an interview which illuminates how these problems have come about. (Note: updated the list numbering from the original article.)

Williams went through a tidy list of the top five Odeo screw-ups: 1. “Trying to build too much” – Odeo set out to be a podcasting company with no focus beyond that. 2. “Not building for people like ourselves” – For example, Williams doesn’t podcast himself, and he says as a result the company’s web-based recording tools were too simplistic.

The first point highlights that Odeo might be better off looking at why subscribers like me have stopped using it. I realise they are probably building services like Twitter to produce a better revenue stream, but why throw away core users?

The second point makes me wonder how many Odeo staffers actually use Odeo at all, let alone to create podcasts. Like I say, Odeo has an unusual property for a website, in that it virtually forces you to give up after using it for several months. Maybe the internal staffers rely on cron-powered SQL delete commands to flush their feeds, but there appears to be no solution for the rest of us.

I want to use Odeo again. Please let me know if anyone has a solution to the Amazing Indestructible Odeo Feed That Knows No Satiety.

Ajax Functionality and Usability Patterns – Podcast 2 of 4: Ajax Page Architecture

This is the second in the four-part series on Ajax functionality and usability patterns (Book: Part 4, pp 327-530).

The guest for this week is Dave Johnson of Nitobi (the Ajax component developers formerly known as E-Business Applications), widget guru and author of the upcoming Enterprise Ajax book. Dave helps me walk through the patterns and offers plenty of great insights along the way. We mention Dave’s recent presentation a couple of times; here’s the PDF.

This 54-minute podcast covers ten patterns of Ajax Architecture (Book: Chapter 15, pp 389-444):

Web 2.0: What Happened to Organic Growth???

Another day, another round of VC. From TechCrunch this week:

Online social network Multiply has closed a Series A funding round with $5 million from Transcosmos and $1 million from the company’s founders. Multiply is a service that filters all networking functions, from highlighted users to visible tag clouds, through a proximity filter with a slider. In other words, users determine whether they want to view information about people just on their contacts list, who are friends of a friend, etc.

And this from last month (via The Medici Effect):

Wine.com has become the most stubborn dot.com hangover ever.

The San Francisco online wine retailer, which depending on how you count is now on around its seventh reincarnation since launching in the 1990s, will announce tomorrow that it has raised $12 million in new venture capital from New York-based Baker Capital.

<

p>Technorati also announced a new $7 million round this week. Hardly significant for a company people speculated to have been talking about a billion-dollar acquisition a while ago, but still, it makes you wonder – if the organic growth model really works, how can a Web 2.0 poster-child still need a cash injection after being live and successful for over 2 years?

Cameron Reilly I believe had planned to go organic with ThePodcastNetwork, but he reported this week that he started to opt for the acquisition path instead.

Funding last week for some hotel version of myspace I’ve never heard of that wants to expand to Japan; Rumours of upcoming funding for HuffingtonPost; I could go on.

VC is all good if you know what you’re gonna do with the cash, but what happened to organic growth? You know, all the arguments about “everything’s cheap now” (open-source argument), “people are willing to pay for virtual goods” (37S/Flickr argument), “avoid overplanning” (agile argument), “Adsense rocks” (plentyoffish argument), “VCs burned us in 2000″ (learning from history argument), “we can outsource development” (naieve naieve argument).

Am I missing something? Has Web 2.0 entered a new phase where VC is now the thing to do? Did the organic growth model just fade away?

Ajax Programming Patterns – Podcast 3 of 4: “DOM Population” and “Code Generation and Reuse” Patterns

The third podcast in this series of Ajax Programming Patterns. The 29-minute podcast covers five patterns. As with the previous podcast, there is reason for concern about the audio quality herein. Firstly, three patterns on DOM population – taking server response data and displaying it or storing it in the DOM:

The second group of patterns (representing a different chapter in the Ajax Design Patterns book) are a couple of generic Javascript patterns to make the code more maintainable and portable: