New Patterns: XMLHttpRequest Call and IFrame Call

Full drafts are now online for the two big Web Remoting patterns: XMLHttpRequest Call and IFrame Call. There are also a couple of new demos to illustrate GETting and POSTing with both call types: XMLHttpRequest Example, IFrame Example.

A few extracts follow, but first, let me ask you: Do you know of any public IFrame remoting examples? Google Maps is the only public example I know. All the other examples here are tutorials and framework demos.

The XMLHttpRequest alternatives was the most fun thing here (headings shown below). I went through the various techniques I’ve seen, read about, and tried out, to list as many ways as possible to programmatically call the server. As you might expect for these “Foundational Technology” patterns, there’s a lot more code in the Solutions than in later patterns, where the focus shifts more towards the -ilities, particularly usability and maintainability.

As usual, feedback much appreciated. (Mainly on the content, rather than woeful spelling and outright grammatical disasters at this stage.)

XMLHttpRequest Call

Solution: Use XMLHttpRequest objects for browser-server communication. XMLHttpRequest is a Javascript class capable of calling the server and capturing its response. Just like a browser or a command-line web client, an XMLHttpRequest issues standard HTTP requests and grabs responses. Note: The code shown throughout this demo is closely based on an online companion demo (http://ajaxify.com/run/xmlHttpRequestCall/).

The above example hopefully illustrates that the fundamental technology is pretty simple. However, be aware that it’s a very basic usage, not yet fit for production. Fundamental questions remain:

* How can XMLHttpRequests be created?
* How does an asynchronous call work?
* How do our code detect errors?
* What if the service requires a "POST" or "PUT" request rather than a "GET"?
* Which URLs can be accessed?
* How can you deal with XML responses?
* What's the API? 

The following sections address these questions and show how the code above needs to be modified. …

Decisions

6.1 What kind of content will web services provide?
6.2 How will caching be controlled?
6.3 How will you deal with errors?

Alternatives:

9.1 IFrame Call
9.2 Persistent Connection
9.3 Richer Plugin
9.4 On-Demand Javascript
9.5 Import XML Document
9.6 Image-Cookie Call
9.7 Stylesheet Call

IFrame Call

Solution: Use IFrames for browser-server communication. IFrames are page-like elements that can be embedded in other pages. They have their own source URL, distinct from their parent page, and it can change dynamically. IFrame Calls work by making the IFrame point to a URL we’re interested in, then reading the IFrame’s contents once the new content has loaded.

Chat: The “Hello World” of Ajax?

Day Barr:

Chat is not quite the Hello World of Ajax, but it’s one of the simplest yet useful things I could do. I didn’t learn very much by writing an Ajax Hello World example and it’s completely pointless :-)

As many are learning, an Ajax “Hello World” is pretty easy, provided you’ve already got a grounding in web/JS programming. So beyond that, what’s the quintessential Ajax tutorial app? Some of the Ajax texts are beginning to pour out, so maybe a pattern will emerge. But, for now, the learning apps of choice seem to be:

I’m surprised we haven’t seen more people playing with Ajax wikis and RSS aggregators, but I’m sure they’re coming.

BTW, a basic “Hello World” might be easy, but it’s also a useful springboard for some important variants. e.g. Add a long delay in your server and see how the browser script handles a timeout (or simulate a delay with Julien’s GM script. e.g. Host the service on another domain and have the server act as a proxy. e.g. Apply a visual effect like the famous Yellow Fade Technique when the message comes in. These variants aren’t just interesting exercises – all are important for real-world Ajax development.

Another DHTML “Game”

A new DHTML boxing “game” from the Man In Blue. Not quite DHTML Lemmings or Super Maryo World, but still more fun than it should be. A bit like the Ruby On Rael thing in reverse.

Also from the same presentation is an eerily lifelike OSX clone (Firefox-only). More evidence that Ajax might be useful in prototyping desktop apps. And since prototypes usually go on to become the real thing …

Thanks to the Web Essentials organisers for podcasting the talks. Safran on Sunday will have to go on hold for a while.

Ajax can *Improve* Performance Too

Recent Ajax apps like Kiko are sluggish according to Alexander Kirk’s “Rise of Slow Ajax Applications (via AjaxDeveloper):

Pages get more voluminous because so much code has to be loaded to the browser (which makes the browser slow again) so you could just begin to use the application. This somehow reminds me of all the flash apps. Waiting for hours to load the page and you’ll stick to that page for half a minute.

The initial delay is due to loading of Javascript and data too. Fast startup is certainly a big motivation for Ajax, so it’s a problem if that’s not happening.

Alexander gives four tips:

  • Keep it bookmarkable. Don’t load everything to one page, let users return to a certain section of your app via their bookmarks.

This is sort of mixing two concepts together. Using Unique URLs, it’s possible for an Ajax app to be bookmarkable, but still the same “page”. I think the advice here is to use real page refreshes rather than making an entire website a single Ajax application. I think that’s fair for a big corporate website, but for a standalone app like Kiko, it’s best seen as a last resort. Saying that, a reasonable use might be in switching between a general login/menu page and the core app itself. Or, in Writely, switching between the different tabs – editing, blog it, etc.

  • Don’t overuse AJAX. Often simple Javascript without server interaction will do. Try to reduce the server callbacks.

Agree with the tip, though not necessarily the implicit definition that Ajax == server interaction. Rich, standards-based, Javascript is Ajax in my book, and certainly a good way to improve performance.

  • Minimize the code to be loaded. When you don’t have any other choice, consider code downloading.

Code downloading – or On-Demand Javascript – is actually quite easy to do. It involves loading some basic Javascript immediately and continue to load the rest in the background or as needed. You can load the JS by tweaking the head element, or alternatively by pulling it down with XMLHttpRequest and eval’ing it. As an extension of multi-stage code loading, there’s also Multi-Stage Download – downloading an initial structure upfront and populating it with further calls.

  • Speed up your apps with AJAX. Use AJAX for what it was meant for: tune your existing application at points where postbacks need to reload the same page with only little change.

Great point. <rant>Too often, Ajax is blamed for making the web slow, unaccessible, unusable, etc. But what if we stop a minute and ask “What if Ajax could improve performance/accessibility/usability? How could that happen?” By attempting to answer these questions as effectively as possible, even if we disagree with the premise, we’re better equipped to weigh both sides of the argument.</rant>

In the case of performance, there are plenty of ways Ajax actually improves the situation. For starters, you don’t need to download an entire HTML page every time you talk to the server. More specifically, smart Javascript can do stuff like caching, guesstimating, and pre-fetching.

New Ajax Demos

Further to the previous post on new Ajax programming patterns, it’s also worth noting there’s a new bunch of corresponding online Ajax demos too. Being programming patterns, they tend to be “pure refactorings” rather than enhancements, thus the user experience is mostly the same – you have to look under the covers to see what’s changed. The demos are mentioned in individual patterns, but here’s a sampling:

  • The wiki has a bunch of timeout-related refactorings. These refactorings are user-visible. (They’re not actually part of the programming patterns, but they were recently added.)
  • 2 other wiki refactorings involve On-Demand Javascript . (No UI change.)
  • A JSON playground.
  • The Portal Drilldown Demo has been refactored to render with XSLT and browser templating tools. (No UI change.)
  • A couple of refactorings added to demonstrate REST (broken right now, probably due to a PHP config issue) and RPC on the Ajax Shop demo. (No UI change.)

As with all the demos, they’re tested on FF and IE6, please let me know if anything breaks.

18 New Ajax Programming Patterns

I’ve uploaded full text for 18 new Ajax patterns, completing a first-cut draft for all Programming Patterns content, which will be one part of the book. This section bridges the gap between the very basics of Ajax – XMLHttpRequest, DOM, etc – and the high-level stuff like widgets and visual effects. For instance, how do you do design the services accessed by XMLHttpRequest? Or how do you deal with a lots of Javascript? The focus here is on traditional tech concerns such as maintainability and understandability; and less about usability.

Breaking it down:

  • Web Services patterns are about overall browser-server architecture – what kinds of services and what kinds of messages? It’s relevant to the conversation of Jon Tiersen and others about what sort of responses are possible – XML, JS, HTML, JSON. It’s also relevant to another recent conversation, on [http://www.theserverside.com/news/thread.tss?thread_id=35979 Ajax and webservices], as it covers REST and RPC.

  • Browser-Server Dialogue covers a mixed bag of patterns about XMLHttpRequest handling and what you can do with it, such as loading JS on demand.

  • DOM Population is about transforming incoming XML into DOM content, e.g. XSLT.

  • Performance Optimisation covers a variety of performance patterns such as caching and pre-fetching.

  • Breakout (can you think of a better name) is about going beyond standard Ajax constraints, namely using a server-side mediator to access external domains and introducing a plugin to go where no Ajax app is allowed to go.

  • Code Generation and Reuse is strongly related to some of the emerging frameworks like Echo2 and SAJAX.

This is all early days, so any feedback on the structure, names, or content will be taken into account and acknowledged too (really!). OK, here’s the lot …

Programming Patterns

Web Services

  • RESTful Service Expose web services according to RESTful principles.
  • RPC Service Expose web services as Remote Procedural Calls (RPCs).
  • HTML Response Have the server generate HTML snippets to be displayed in the browser.
  • Semantic Response Have the server respond with abstract, semantic, data.
  • Plain-Text Message Pass simple messages between server and browser in plain-text format.
  • XML Message Pass messages between server and browser in XML format.
  • JSON Message Pass messages between server and browser in Javascript Object Notation (JSON) format.

Browser-Server Dialogue

  • Call Tracking Accommodate busy user behaviour by allocating a new XMLHtpRequest object for each request. See Richard Schwartz’s blog entry (http://smokey.rhs.com/web/blog/poweroftheschwartz.nsf/d6plinks/RSCZ-6CEQAR).Note: Pending some rewrite to take into account request-locking etc.
  • Distributed Events Keep objects synchronised with an event mechanism.
  • On-Demand Javascript Download Javascript as and when required, instead of downloading it all on page load.

DOM Population

Performance Optimisation

  • Fat Client Create a rich, browser-based, client by performing remote calls only when there is no way to achieve the same effect in the browser.
  • Browser-Side Cache Maintain a local cache of information.
  • Guesstimate Instead of grabbing real data from the server, make a guesstimate that’s good enough for most user’s needs.
  • Submission Throttling Instead of submitting upon each Javascript event, retain the data in a local buffer and upload it periodically.
  • Explicit Submission Instead of submitting upon each Javascript event, require the user to explicitly request it, e.g. submit upon clicking a button.
  • Multi-Stage Download Quickly download the page structure with a standard request, then populate it with further requests.
  • Predictive Fetch Anticipate likely user actions and pre-load the required data.

Breakout

  • Cross-Domain Mediator Allow the browser to communicate with other domains by server-based mediation.
  • Richer Plugin Make your application “more Ajax than Ajax” with a Richer Plugin.

Code Generation and Reuse

  • Ajax Stub Use an “Ajax Stub” framework which allows browser scripts to directly invoke server-side operations, without having to worry about the details of XMLHttpRequest and HTTP transfer.
  • Server-Side Code Generation Automatically generate HTML and Javascript from server-side code.
  • Cross-Browser Component Create cross-browser components, allowing programmers to reuse them without regard for browser compatibility.

Aside I was interviewed yesterday for a Japanese magazine about how I’m using the wiki. So maybe some people will be interested to know that I always write the patterns offline because I am more creative in Vim and also to avoid textarea hell. (So much for Writely, will anyone create Vimly, there’s gotta be more money it than cloning MS-Word online :-D). I also use the Mozex or ViewSourceWith extensions to make partial edits using Vim.

This time, I decided not to upload once or twice at a time; but instead all 18 at once. There’s serious overhead of introducing each new pattern to the wiki (from Vim, the sequence is: 2mins spell-check, 5-10mins markup massaging, 2 mins fixing the link and description on the homepage; sometimes exacerbated by server lag.) Uploading all at once at least allowed me to focus fully on the task and also made some aspects more efficient, particularly updating the homepage.)

Cross-Domain Portlets with Start.com Gadgets/Widgets/Startlets

Scott Isaacs on a new Start.com feature (quoted on Ajaxian):

DHTML-based Gadgets: Start.com consumes DHTML-based components called Gadgets. These Gadgets can be created by any developer, hosted on any site, and consumed into the Start.com experience. The model is completely distributed. You can develop components derived from other components on the web.

Remote portlets take the Ajax portals to another level. In fact, the whole concept is one of the biggest developments in the history of the web. A9 pre-empted it with their “Open Search” capability, which allows developers to hook into the A9 interface with live search results. Now MS is going a step further, with Konfabulator-style interaction directly on the web. It’s likely Yahoo has similar plans, given that it’s now the owner of Konfabulator. What Google does is anyone’s guess. Strategically, MS has a big edge here because of its experience working with Developers, Developers, Developers! (YES!)

The idea combines [Cross-Domain Mediator](http://ajaxpatterns.org/Cross-Domain Mediator) with Portlet. Those patterns have subsumed a now-deleted pattern that previous covered this kind of thing:

Cross-Domain Portlet: Introduce Interactive Portlets to facilitate a conversation between the user and a third-party, e.g. for services or “advertising as conversation” (http://radar.oreilly.com/archives/2005/05/remaking_advert.html)

Ajax makes it possible because there’s no page refresh. Conceptually, each portlet is autonomous and capable of conducting a conversation with another domain in parallel with other portlets. When a result comes back to one portal, the other portlets are unaffected. (You can obviously yoke portals together if you want to.) In practice, there are a couple of finer implementation points:

  • You can’t make an XMLHttpRequest Call to an external domain. Hence, the portal server must mediate.
  • Conducting parallel conversations, you’ll need some form of Call Tracking. Hopefully, you’re using a library that abstracts those details from you.

Now that a happening, here are a few things I’d like to see:

  • An open standard for conducting cross-domain conversations, one that protects the end-user and the portal host, but provides sufficient functionality to the
  • Useful ads (the advertising as conversation) concept. There’s definitely a business model here for opt-in advertising. That is, users would be willing to add advertising portlets that offer genuinely useful information. e.g. a travel portlet that lets you check flight availability.
  • RSS aggregator integration The nature of a portal can change, right? So RSS aggregators should be able to include remote portlets.
  • End-user-created portlets. Make it easy for end-users to construct their own gadgets. This idea is inspired by Charlene Li’s comments on Konfabulator: “The problem is I already know what the ultimate widget would be me … There’s only one small problem — I don’t know JavaScript! So I am at the mercy of app developers …”

A Few Ajax Gotchas At Jalecode

Andrew Sutherland offers a few Ajax Gotchas/Tips. I’ll add some comments.

  • Escape content with encodeURIComponent() which is superior to escape.

  • XMLHttpRequest’s readyState tells you how far the request has progressed. MM: If you’re confused about readyState‘s transition from 0 to 4, you have good reason to be. Read the recent posting and comments on David Flanagan’s blog, and you’ll learn that 2 and 3 are ambiguous to the point of being unusable. Essentially, you want to wait for either 4 or timeout, and probably ignore everything else.

  • Permission Denied” for XMLHttpRequest is usually due to trying to call another domain. MM: The standard security policy is that requests can only be sent to the originating server, just like the traditional policy for Java applets. To get to another domain, you can set up a Cross-Domain Mediator. This security issue has become interesting with the growing popularity of Single Page Applications (SPA). What can an HTML page sitting on you hard drive access? All domains or no domains? It would certainly be convenient if it could access the web at large. I don’t think it can access any domains on standard browsers, but it’s still possible if the user wants it to happen. Here’s what Steve Yen (TrimPath) says on this issue: “I’m shooting for now to have explicity user-driven synchronization working, which my experiments lead me to believe is workable.”

  • MM: Finally, I’ll add another gotcha-inspired tip to Andrew’s collection: Set content type to XML (in the case where you want to treat the response as XML), e.g. in PHP, header("Content-Type: text/xml");.

A Fading Habit

I’ve noticed that I’ve recently habitualised the Yellow Fade Technique, or what the Ajax patterns refer to as One-Second Spotlight.

The typical Fade example is a form field. When you change it, the field is suddenly highlighted, then gradually fades back to its original form. This tells the user that “the computer” knows something’s happened, and also serves to draw their attention to a particular element.

This is one of those things that could have been big years ago, but never happened. Despite the fact it’s only become popular recently, it’s all quite simple from a technical perspective, even more so with the new libraries coming out. I created a couple of demos a while back – one is powered by Scriptaculous, the other uses a custom-built fading engine. The first demo runs opacity through a sequence, the second demo does the same to colour, rather than opacity.

In the Scriptaculous case, running the effect is mind-numbingly simple to invoke:

new Effect.Appear(“defaultTimeLabel”);

And likewise in the case of my custom fader (which lets the user specify colour preferences):

fader.fade(defaultTimeLabel, $(“startColor”).value, $(“endColor”).value);

Anyway, I’ve lately been coding some more dynamic DOM manipulations, and it’s occurred to me that I now ask myself, whenever I add or remove an element, if I need to run an effect. Having scriptaculous handy makes it so easy to do. So fading started off as a bit of a novelty and is now standard practice.

I can hear people saying, “not the blink tag” and re-living 1995-esque motion sickness. That will definitely happen on some Ajaxian sites, but let me clarify that I’m only asking the question each time a change comes up, not actually implementing the change. Apps using the One-Second” visual effects include TiddlyWiki and Backpack (one of the first uses). They show how it can be used effectively and without going overboard.

Of all the new JS libs coming out, I haven’t seen any that augment or replace basic XHTML DOM manipulation. If one does emerge, I’d like to see it provide support for auto-effects and perhaps a parameter to indicate the effect. e.g. el.hide(Effect.FadeOut); el1.add(el2, Effect.BlindDown).

Reading Open-Source Code

Dave Crane looks into some Scriptaculous code. The main point is that Scriptaculous is easy for novice JS programmers to use, but the source code would be difficult, for novices at least, to understand. Which raises these questions:

So, going back to the bigger picture, here’s something for the authors of JavaScript libraries to consider. How much of a gap are you creating between using your library, and being able to modify or enhance it? Does such a gap present a barrier to its uptake? Should good code be there to be read as well as executed?

In theory, the implementation of a library shouldn’t matter to developers using it. That’s because a good library should offer components that are open for extension, but closed for modification – the open-closed principle. Thus, you should be able to treat the library as a black-box and only care about the API it presents – you never have to maintain it, so why read the code?

But it’s usually not so simple. You might not need to change the code, but you do need to understand it. In an ideal world, there would be abundant documentation. But agile says that all that documentation is valuable, but self-documenting code is much more valuable. In similar vein, the availability of code is often justified as a reason for open-source projects to include less documentation (among other benefits). So open-source code had better be understandable, ideally self-documenting, but at least well-commented.

I have used (by “used”, I mean “wrestled against in a cage match”) a couple of very prominent Java EE frameworks where implementation code is confusing and badly in need of refactoring, with most of the code comments in “TODO” and “Weren’t you going to fix this?” territory. I was stuck and documentation was insufficient, so I could only turn to the source code for help. And that’s actually a pretty good place to work out what’s going on anyway. But, alas, the code did not come to my aid. ** Perhaps that’s the natural consequence of an open-source project becoming a startup with a support-based revenue model, but it makes a mockery of the code-as-documentation argument for open-source.**

The Scriptaculous implementation is good for anyone who knows Javascript, which is all you could hope for … just because it’s API is simple enough for novices, doesn’t mean its implementation has to be. In fact, the implementations of “Puff”, “Fade”, and so on, are actually extensions of the base functionality, and any developer could create their own effects in the same way. The Effect framework is an example of how Javascript can provide decent support for Domain-Specific Languages, in Scriptaculous’s case, a grammar of visual effects. Also, if only want to use an existing effect, but with a little variation, some of them accept an options specification.