AJAX and the Great Data Cloud in the Sky

Office apps, backup, calendars – trying to store it online was all the rage of the mid-90s. By 2000, we would be hopping on our flying scooters and whizzing past the great data cloud in the sky.

Well, it was nice to dream.

Here are two major reasons it never happened:

  • Options for rich clients – like Java applets and downloadable applications – were never feasible for a range of reasons, meaning that web applications became the only realistic choice. But web applications gave a lousy user experience and were hampered by portability issues.
  • Trusting the host. Are you crazy? You want me to store my prized data on someone else’s box?

Well, AJAX is a big step towards rich clients, and what we’re seeing now will only get better with emerging patterns and frameworks. GMail and Backpack and the thousands of wikis are all examples of web applications with important data being stored server-side.

But right now, you have to trust the host with your data. Feasible in some cases, but it would be ideal if data could be encrypted server-side. So a big challenge right now is to work out how web applications can deal with encrypted data.

It turns out that AJAX may well offer a viable solution to server-side encryption. For the past couple of weeks, Richard Schwartz has been talking about how an AJAX application could decrypt on the fly. The idea is this: in the past, decryption in the browser would have meant a Java applet – with all the usual Java problems – or Javascript-based encryption. But Javascript can’t remember data between page loads, so the user would have to type the passphrase each time (or use frames, which is feasible though messy). An AJAX application need load only once, so the user would only have to enter the password once per session. Richard has a proof-of-concept demo too.

There’s also some further discussion about how practical this idea is. Having to type your password in each session is certainly an issue, but not a showstopper in my view, at least not for techie users. The reason is because of browser plugins. As Richard mentions, it could be solved on Firefox with a GreaseMonkey script. The reality is, it could be solved on IE too with the right plugin. So this is an example of progressive enhancement. You can type your passphrase in if you need to, but install this plugin to avoid it.

AJAX Patterns: Design Patterns for AJAX Usability

I’ve been putting together some AJAX design patterns.

Update (May 15, 2005): I’ve set up AJAXPatterns.org to keep working on these patterns. I’ve also cleaned up a couple of things here, although all future changes will occur at ajaxpatterns. Thanks to Leoglas for spotting two errors here.

Why AJAX Design Patterns?

AJAX holds a lot of promise for web usability, and the underlying technology has already delivered some stunning applications. But it’s no magic bullet. Careful design is always required, and it must be based specifically on the technology at hand. As AJAX emerges, we’re going to learn more about what sort of design works, and we’ll need ways of documenting this information and talking about it. Fortunately, the evolution of this particular technology will take place at a time when design patterns are well-entrenched in the industry, and design patterns are an excellent means of knowledge representation. Thus, it makes sense to begin cataloguing AJAX design patterns. These are some thoughts based on current examples and demo systems.

Patterns being patterns, there’s not a lot of unseen information here. Patterns are just a concise way to represent the knowledge embodied in the many AJAX applications that are out there. The point is to discover best practices by investigating how developers have successfully traded off conflicting design principles. AJAX is all about usability, so the patterns focus particularly on delivering usability in the face of constraints, most notably: user capabilities and expectations, bandwidth limits, the stateless nature of HTTP, the complexity of Javascript.

This is a work-in-progress. There should eventually be more patterns, more examples, more detailed explanations. And one more disclaimer: AJAX is a new term, but XMLHttpRequest and related technologies have been around for a while. I know that, but the introduction of a single umbrella term nevertheless constitutes a tipping point which is forcing web development to move heavily in a certain direction. AJAX is only a name, but names can be tremendously important.

Related Work

  • Thomas Baekdal’s AJAX Usability Guidelines contains some great advice and inspired me to write these patterns. What I’m doing here is not just stating different guidelines. Patterns ain’t principles, and the focus here is on patterns whereas the focus above is on principles. Both forms of design advice are vital. Here, I’m explicitly distinguishing between the two, because the patterns are intended to help designers achieve the principles. The principles here are heavily influenced by Baekdal’s guidelines.

  • fiftyfoureleven.com and AJAX Matters contain great collections of AJAX examples. You can’t have patterns without examples.

  • Joel Webber illuminates the technology behind Google Maps. Chris Justus dissects Google Suggest.

  • The recent AJAX podcast and blog entry covered some of the strengths and weaknesses of AJAX. The patterns here are supposed to guide on emphasising the strengths of AJAX without being struck by its weaknesses. The Polymorphic Podcast has also covered AJAX – prior to my podcast and is recommended as alternative overview on the technology.

Design Principles for AJAX Applications

  • Minimise traffic between browser and server so that the user feels the application is responsive.
  • Be clear on the interaction mode being used – regular HTML versus AJAX versus desktop application so that the user can predict what will happen next .. no surprises.
  • While avoiding confusion, borrow from conventions of HTML and desktop applications so that the user can rapidly learn how to use your application.
  • Avoid distractions such as gratuitous animations so that the user can focus on the task at hand.
  • Stick with AJAX wherever possible – just say no to entire page downloads so that the user’s experience is consistent.
  • Adopt AJAX for usability, not just to illustrate you’re hip to where it’s at so that the user is engaged, and not immediately driven away by your nod to website splash screens, <blink tags>, popup ads, and other usability disasters of websites which have gone to a place you don’t want to be.

Some (and eventually all?) of these principles above inform the patterns below. The patterns are about trading off among the principles, and also about resolving conflict between the needs of usability and other practical constraints, such as ease-of-development.

Architectural Patterns

Local Event-Handling You want the user experience to be dynamic and responsive, but it slows things down if you have to keep going to the server. THEREFORE, ensure most events are handled locally, resorting to server-side interaction only when there is no local facility. To increase the cases where local handling is sufficient, consider using Local Caches.

Local Cache You want to support Local Activity, but many events must be handled by resorting to data on the server. THEREFORE, maintain a local cache of information. The cache might be created on initialisation or accumulated after each query. When accumulating upon query, it’s probably worthwhile performing a predictive download.

  • Example: Websites have kept local information at hand for many years. Many forms cache a mapping from countries to states, for instance.

Predictive Download You want the application to respond quickly, but many user actions will require information from the server. THEREFORE, anticipate likely user actions and pre-load the required data. Note that this auxiliary information should ideally be downloaded after the main information has been loaded. A “background thread” might even be used to continuously monitor the user’s local behaviour, regularly pulling down information which might be useful.

  • Example: Google Maps downloads more than you ask, evidenced by the fact that you can move a little in each direction without forcing any new blocks to be loaded. Another way to see this is to compare what happens when you pan slowly versus quickly. Another example is International Herald Tribune, which caches entire articles to provide immediate gratification when you click on “next page” (an example taken from here. Google asks the browser to prefetch the first search result in case the user clicks on it.

Submission Throttling You want the application to respond quickly, but it will be overloaded if the server was accessed for every trivial input event. THEREFORE, instead of submitting upon each Javascript event, use the events to change local state. Store events in a buffer variable, or some DOM component, and frequently (e.g. once a second) poll to see if anything should be submitted. As a variant to periodic polling, check the time since last submission whenever a new Javascript event occurs. Note: If the period is long (e.g. 10 minutes), show the Synchronisation Status and consider allowing explicit submission as well.

Explicit Submission (An alternative to Submit Throttling.) You want the application to respond quickly, but it will be overloaded if the server was accessed for every trivial input event. THEREFORE, instead of submitting upon each Javascript event, require the user to explicitly request it, e.g. submit upom clicking a button. Ensure the user knows what’s been submitted with Synchronisation Status.

  • Example: Like many chat applications, the LiveChat demo requires you to click a “submit” button in order to complete your chat message.

Periodic Refresh You want the browser to be synchronised with the server, but information may change server-side after the user last made a change (e.g. because of another user on the same system). THEREFORE, the browser should periodically call the server so as to refresh any volatile information. Furthermore, indicate information staleness is with Age Displays.

  • Example: Open two browsers, and you’ll see that this chat demo polls the server for changes every few seconds.

Distinct URLs You want to let users send and bookmark information, but the application is dynamic and the results of XMLHttpRequest interaction will not alter the URL. THEREFORE, use a URL-based scheme or write distinct URLs whenever the input will cause a fresh new browser state, one that does not depend on previous interaction.

  • Example: A9 interacts via AJAX, but nevertheless provides a clean URL when you perform a new search. Google Maps has a “Link to this Page” link, which will always provide a link to whatever the user is looking at.

Display Patterns

Rich CSS You want the user-interface to be rich and colourful, but you have to minimise download size in order to make the interface responsive. THEREFORE, lean heavily on CSS to give a rich, graphical, appearance, without having to download images.

Embedded Text You want the-user interface to be rich and graphical, but you need plain text in order to support standard web functionality such as cut-and-paste and inspection by search engine robots. THEREFORE,integrate text into graphics, and rely on Rich CSS to show text when fancy styling is desirable.

Age Display You want the user to trust the system, but it’s possible some information on the browser is stale. THEREFORE, indicate the age of any critical information. Vary background brightness, for instance.

  • Example: SkyScanner is a non-AJAX price comparison site which shows how old its prices are. Many trading applications do likewise with stock quotes. I’m not aware of any AJAX examples at this stage; this pattern is purely speculative and likely to be relevant only to public websites which cannot justify frequent updates.

Synchronisation Status You want the user to rely on the system to capture the input, but not all input is immediately submitted because you are using Submit Throttling or Explicit Submission. THEREFORE, provide hints that indicate what’s synchronised and what’s pending. For instance, use a different background colour on text fields. This is important if you are using Explicit Submission or Throttling with a long period.

  • Example: Manoloweb’s wiki-like demo tells you when the edit has been submitted (although is possibly buggy as the message is always there?). (Note: Since web forms were born, many heads and fists have been banged against keyboards after information was lost to timeouts and shutdowns. AJAX is capable of solving this problem, and this pattern is part of the solution.)

Interaction Patterns

Nothing new here. All have been available in desktop GUIs and some in non-AJAX DHTML too. This is really just a catalogue of interaction styles that are becoming common idioms in AJAX applications and can benefit from XMLHttpRequest-driven interaction.

Drag-and-drop e.g. This demo

Popup data input

Popup information

Highlighting e.g. A9’s customisation feature highlights search plugins you select them.

Drilldown e.g. Betfair’s Sidebar Menu.

Auto-Completion Most famously, Google Suggest.

Coding Patterns

Browser-Agnostic Components You want to make the system interactive with sophisticated DHTML, but portability can suffer. THEREFORE, encapsulate units of DHTML activity into browser-agnostic components. A “component” may simply be a Javascript function with a bunch of if-statements, or a factory in an Abstract Factory hierarchy, or a server-side, Javascript generation object.

Server-Side Code Generation You want to make the system interactive with sophisticated DHTML, but sophisticated DHTML can be difficult. THEREFORE, use a server-side framework to generate the HTML and associated Javascript. Note that this pattern will become more useful as frameworks emerge.

Further Issues

… which will emerge as patterns.

  • RESTful architecture and treating server requests as API calls. There’s a lot of scope for XMLHttpRequest to be dealing with a generic, public, API. For instance, it didn’t take long before Google Maps had been recruited as a geographical API.
  • The “X” in AJAX. How should the response be structured?
  • Collaborative applications. How to deal with merges, pessimistic/optimistic locking, etc.

Your Thoughts?

I’m interested to hear what people think. Leave a comment or send an email to [email protected]

Advanced Javascript Book: Professional JavaScript for Web Developers

Not your grandfather’s Javascript … looks like Nicholas Zakas’s Professional JavaScript for Web Developers is now out. This is great news, as the book’s editor – Jim – noted that there’s an AJAX chapter in there. Well, he, qualifies, “OK, not exactly. What he wrote for that book is chapter 16 “Client-Server Communication” (see the full TOC here) which is all about AJAX principles without actually using that term not coined until a couple of months ago.”

I’m about to order it and looking forward to it. With AJAX coming on strong, Javascript will only get bigger and bigger. So it’s time for the eternal optimists to swallow the pill and learn how to do it the right way. This book will do well: advanced Javascript texts are rare, and, as far as I can see, nothing has been published for several years. Given all the recent browser updates, that alone is enough reason to get hold of it.

BTW, there’s been a great response to the AJAX podcast in the past couple of weeks. I’ll mention more in my next podcast, but for now, one thing’s pretty clear: by the end of the year, AJAX is confirmed as the buzzword of ’05.

Podcast+Text: The AJAX Web Architecture

This podcast discusses AJAX, an architectural style for web applications that has become popular in recent months.

Click to download the podcast mp3

This is a Podcast – a blog entry with an embedded MP3 link. On Internet Explorer, Click the left mouse button to listen, or click the right mouse buttton and “Save As…” to download. Better yet, you can subscribe for updates straight into your PC or ipod – it’s easy, open, and free. Install the free, open-source, Ipodder client and when it starts, just paste this in: “http://www.softwareas.com/podcast/rss2”. Too easy! 15 minutes and you can be subscribed to receive thousands of MP3 podcasts – from official BBC documentaries to business personalities like Sun COO Jonathan Schwartz to scores of amateur publishers speaking about their own interests. They will download straight to your PC as soon as they’re published. You can listen to them on your PC or any portable MP3 player. If you have an IPod, programs like IPodder will push the downloaded MP3s straight into ITunes, so you can leave home each day with new content loaded on to your IPod. More info in the Podcast FAQ.

Quick Overview

Traditional webapps continue to send pages in their entirety, upon each user request. Consider a wiki such as Wikipedia:

  • User changes some text
  • Browser submits the new text
  • Server saves the text and sends the entire page again, updated this time
  • Browser clears previous page and draws all of new page

AJAX apps don’t redraw the whole page. Instead, they send a little request, receive a result, and adjust the page accordingly. The wiki of the future will look like this:

  • User changes some text
  • Browser submits the change
  • Server saves the change and sends a confirmation and maybe the latest timestamp.
  • Browser adjusts any information, e.g. shows the new timestamp.

For a glimpse of wikis to come, check out this Instant Edit webapp. Fire it up in two different browsers and see how the changes persist without reloading the page.

Most famously, Google has a few AJAX applications: Google Maps, GMail, Google Suggest.

YARC! Yet Another Rich Client!

AJAX, and the underlying XMLHttpRequest object, is the latest approach in the tradition of enriching the web platform. To put it into perspective, here are a few other attempts at rich applications over the years:

  • In the early 1990s, the Mosaic browser made the web clickable. It was the first window-based browser.
  • Images, then animations and sounds, were later added. ASCII art soon faded as quickly as silent film.
  • Java allowed for client-side applets.
  • Javascript (no relation) allowed for embedded programming inside HTML and led to a dramatic rise in cross-browser compatibility tables.
  • Flash was introduced, and with Flash MX, became more programmer-friendly.
  • Client-side GUI applications increasingly connected to the internet. e.g. Multiplayer games, Desktop search tools, ITunes Music Store, Auto-updating virus protection, Dreaded “free” spyware.
  • Frames, and even better, invisible IFrames allowed for invisible request submission and manipulation of the current web page.
  • Each browser continues to offer its own proprietary extensions (with some possible clicking of checkboxes or downloading of extra components): MS offers .NET support, Mozilla and Firefox offer a powerful plugin architecture, Opera offers a presentation package.

The Heart of AJAX: XMLHttpRequest

AJAX is a new name (Feb 2005) for a design style that has been possible, and in fact used sparingly, for the past couple of years. The underlying technology is XMLHttpRequest, a Javascript object that supports web requests. Every big modern language has a class like this: it takes a URL, fetches the content, and provides query support. XMLHttpRequest supports standard text as well as XML documents. That means a web page can wait for Javascript events, submit info to a web server, catch the resulting output, and play around with the current page.

One more thing about the technology: the request-response cycle is asynchronous. The XMLHttpRequest object is registered with a response handling Javascript function, and fires off to the server. Some time later, the server probably comes back, and the registered function catches the result. And that’s what AJAX stands for: Asynchronous Javascript + XML.

Let’s look at an example: This chat application is based on AJAX. You can see the Javascript here. (The entire thing has a Creative Commons license.)

When the user says something, the function sendComment() is called. It grabs the user’s message from the input field and passes it to httpSendChat, an XMLHtppRequest object. httpSendChat posts it to the server.

 
    httpSendChat.open("POST", SendChaturl, true);
    httpSendChat.setRequestHeader('Content-Type','application/x-www-form-urlencoded');
    httpSendChat.onreadystatechange = handlehHttpSendChat;
    httpSendChat.send(param);
 

The emphasised line is the registration: it says that the response will come back to the handlehHttpSendChat function. That function will discard any partial responses, and upon receiving a full response (state=4), it will redraw the chat (which involves another trip to the server):


function handlehHttpSendChat() {
  if (httpSendChat.readyState == 4) {
    receiveChatText(); //refreshes the chat after a new comment has been added (this makes it more responsive)
  }
}

receiveChatText() calls the server to send the recent discussion history, and ensures the response goes to handleHttpReceiveChat(). That function rearranges the chat text according to the recent message:

function handlehHttpReceiveChat() { if (httpReceiveChat.readyState == 4) { results = httpReceiveChat.responseText.split(‘—‘); //the fields are separated by — if (results.length > 2) { for(i=0;i < (results.length-1);i=i+3) { //goes through the result one message at a time insertNewContent(results[i+1],results[i+2]); //inserts the new content into the page } lastID = results[results.length-4]; } …. }

Great, it works if I say something. But it’s a chat program. What if someone else says something? The application simply polls the server, ensuring that “receiveChatText” is called every four seconds.

All About Usability

The chief beneficiary of AJAX is the user. Web applications feel much more responsive, and the user won’t hesitate to perform actions for fear of slow response times, or outright timeouts. Furthermore, form data need not be lost due to browser crashes: using a timer, it can be sent to the server every few minutes, just like auto-backup.

For standard form-based applications, that’s a nice benefit, but hardly a killer app. Where AJAX will shine is in truely rich applications. In particular, on intranets, where many corporations have already migrated traditional GUI applications. This migration process has usually been led by technologists concerned with the infrastructural overheads of administering and upgrading standalone applications. It’s much easier to have all the applications sitting on the server and the clients running a standard web browser.

These web migrations may have improved administrability, but they have often cause users pain. Ironically, they’re often left longing for applications written a decade before the web apps. It doesn’t help that most projects are clueless with regard to usability, but even if usability is considered, the web platform is inherently unusable. The control components are simplistic and server synchronisation is confusing and time-consuming. AJAX doesn’t do anything for the controls, but at least it brings the server and client closer together.

Problems with AJAX

Some objections are taken from the resources below, especially AJAX: Promise or Hype.

It’s Hard to Code

Any Javascript usually makes life more difficult, and early discussions indicate AJAX is no different. At present, coding for AJAX may well be more difficult, although if you look at the code examples around, you’ll see that you’re not exactly facing a Turing Test either. In any event, it’s inevitable that design patterns and supportive frameworks will emerge. A few frameworks already facilitate this mode of development: the always-controversial, uber-funky Ruby On Rails, JSON-RPC-Java, Dynamic Web Remoting. SAJAX, Echo 2. Fortunately for evolution, a wide variety of approaches is being taken. Cross-fertilisation will undoubtedly follow.

In particular, the best frameworks will probably generate as much Javascript as possible, so developers don’t need to co-ordinate between Javascript and server-side controllers.

Testability May Suffer

It’s nice to be able to perform system tests with a robot like httpUnit. Any use of Javascript makes that more difficult. At the same time, because AJAX promotes a more component-based architecture, unit testing may actually be improved. With a good design, it should be quite feasible to test the scripts that are accessed by the XMLHttpRequest object.

Accessibility May Suffer

Any form of interactivity is often anathema to many different types of specialised needs. Nevertheless, this should not stop the technology from progressing, and providing rich interaction to those who can use it. As always, accessibility must be maintained, and multiple mechanisms might be required. Furthermore, new technologies can improve accessibility too. It’s easy to imagine, for example, how an AJAX-enabled site could let users quickly resize and move around certain screen elements to meet their individual needs.

AJAX Will Collapse the Network

AJAX does represent a potential challenge to networking infrastructure. Traditional web applications can feel like earlier client-server applications. Submit your offerings, then receive a response and meditate on it for a while. AJAX makes the term “web application” a lot more honest. The server really is involved, possibly even after each keystroke. Interestingly, bandwidth requirements may go down because usually only small changes need to be sent each way. However, latency is another question: using an AJAX application might feel like typing against a slow telnet connection. Stuff… hap…pens…much…sl..ower…than you…can…think… .

This will probably not be a major concern on intranets, where there are relatively few users and usually good connectivity to the server (especially as it’s often nearby). However, it’s still an open question how AJAX will be used on the public web. Certainly, it can be used to incrementally improve just about any form-based application. And it can surely go beyond that, as Google demonstrates. But can it scale to the requirements of a major site, offering a fully-scaleable wiki or genuinely playable gaming?

It’s Just a Name, the Tech’s Not New At All

XMLHttpObject has been around for a few years, but it would be hard to believe anything called “XMLHttpObject” could trigger a revolution on the PCs of the world. Frames and IFrames supported this sort of interaction even earlier. History and logic would suggest that a standard name and community, combined with some flagship applications, are powerful tools indeed. And the timing is right: users have lived with static web applications long enough, broadband is now mainstream, and the economy hungers for innovation. The raw technology may have been around, and even used in doses. But all signs indicate that the new name, given the increased need and the prominent offerings by Google offerings, constitutes a tipping point.

Bill Won’t Like It

Let’s be clear. This could have happened a lot earlier. There’s a lot of unfounded nonsense about MS on the web, but there is indeed broad agreement that MS does not benefit from adoption of rich web applications. And for pretty obvious reasons. They worked hard to innovate with IE in the mid-90s, attaining the dominant position. Consequently, they managed to crush Netscape’s dreams of replacing the Office Suite and Sun’s dreams of Java on every desktop. It’s hard to see MS doing anything about XMLHttpRequest within IE though; the interaction it provides is rich, but quite frankly, not that rich.

Flash Can Do All This, and More

Flash is a bit of a mystery, since it’s extremely cross-platform, having excellent support on all the major browsers and platforms. And yet, it’s never taken off for serious application work. In fact, it’s really been used for not much more than ads and fancy presentations. It’s certainly capable of doing much more serious applications, and maybe Flash MX will still shine. It took a long time for Macromedia to target serious development. Perhaps this was a strategic mistake, or perhaps it was an intentional means of gaining wide browser share.

Examples

Further Resources