Basics of Ajax 2 of 3: Web Remoting (XMLHttpRequest etc) (Podcast)

Ajax Basics 2 of 3

This is the second of three podcasts on the basic Ajax patterns.

  • Podcast 1: Display Patterns and the DOM.
  • Podcast 2: Web Remoting – XMLHttpRequest, IFrame Call, HTTP Streaming.
  • Podcast 3: Dynamic Behaviour – Events and Timing.

Podcast 2: Web Remoting (XMLHttpRequest, IFrame, HTTP Streaming)

Click to download the Podcast. You can also subscribe to the
feed if you want future podcasts automatically downloaded - check out the
podcast FAQ at http://podca.st.

This 75 minute podcast covers web remoting concepts and the following specific patterns:

  • XMLHttpRequest Call Use XMLHttpRequest objects for browser-server communication. (05:00)
  • IFrame Call Use IFrames for browser-server communication. (31:45)
  • HTTP Streaming Stream server data in the response of a long-lived HTTP connection. (47:00)


Please ignore this bit, I’m claiming My Odeo Channel (odeo/bdd768d9acefcad9)

Mocks, Stubs, Dependency Injection, and … XMLHttpRequest

“A Mock Is Not So Stupid After All!”

Dave Crane’s been talking about Mocking the Server-Side:

A Mock Object is a stand-in for the real thing. Few modern programs are really standalone, and enterprise apps require a very complex context in order to operate; containers, databases, directories, web services, etc. This can make testing difficult, because to set up the test and run it, one needs to provide all the necessary context. Mock objects can be thought of as really dumb implementations of a contextual element, one that would never be useful in production, but provides the predictability and ease of setup required for a test. A mock database might always return the same four rows of data, whatever the WHERE clause. A mock web service might always serve up the same XML document.

So the idea is to create a simulation XML document (or plain-text document) to be retrieved with XMLHttpRequest.

Dave’s followed up with some good tips to that end. All good timing as I’ve recently been working on a demo for the final lot of patterns – the process patterns – and one of the refactorings uses a canned plain-text document to illustrate The Pattern Currently Known As “Server-Side Simulation Stubs” (who said alliteration had no place in software writing?).

Worth noting the terminology used, and here I’ll be pedantic. Dave refers to a Mock what I call a Simulation Stub. In the past couple of years, the terms “Mock” and “Stub” have often been used interchangeably, but I think the distinction is worth keeping, because otherwise the original meaning of Mocks is lost. I don’t really care what things are called, but when two concepts become one name, at least one of those concepts is going to get hurt. It’s a testament to the creators of Mocks that they managed to produce a name so evocative that it’s come to consume a much older concept. (“Stub” and “Shunt” have always been pretty vague terms anyway.)

Under the original definition, a Mock is an object that verifies how it’s being used by a tested object, and by inference, how things like it will be used by the class being tested. Yes, it might spit out simulation data, and it might even have a fancy interface that lets you tell it how to respond. However, those functions are only there to help verify how it’s used. It’s what goes into the mock that counts, not what the mock pumps out. Corollary: a Mock is not stupid. It might not win a Nobel Prize anytime soon, but it’s smart enough to know what to expect and how to check if that expectation’s been met. The mixing of terms has carried over to some parts of the Ruby/Rails community, as Aslak recently observed during a RubyConf presentation. (Podcast on mocks, Fowler’s Mocks Aren’t Stubs.)

In a web context, what might the distinction mean?

  • A Simulation Web Service (or a “Stub Web Service”, but I prefer “Simulation”) is typically some XML or plain-text document that’s been written manually. It could also be a bit more sophisticated, offering a means for the server-side developer to tweak its output.
  • A Mock Web Service is a server-side script that’s been configured to expect a certain sequence of XMLHttpRequest messages, and will carp as soon as the sequence is broken.

As Dave’s articles explain, a Stub Web Service is certainly a useful thing, and I suspect quite a few developers follow that practice. (Please tell me about it if you do!)So how about Mock Web Services? That’s pure speculation!! I could see the benefit, but I’ve not done it, haven’t heard of anyone doing it, and I seriously doubt there are any frameworks that facilitate it.

A variation on Simulations and Mocks would be a completely browser-side approach, in which you adopt an XMLHttpRequest wrapper, and make it output a dummy document while in “testing” mode. This raises further questions:

  • How are mocks handled in Javascript? Well, William Taysom recently noted that you can approximate method_missing, and a Mock library like the awesomely useful JMock library is clearly feasible. There are a few JUnit equivalents out there, so why not a mock lib? (Would you use such a thing?) If you say that would be over-engineering, then the same charge could be made about JS testing in general, since mocks are really a natural consequence of unit testing and TDD … any time you find yourself testing an event mechanism, a mock might be a good idea.
  • How to switch context in Javascript? Okay, so mocks are a big reason for the rise of Dependency Injection and “lightweight” ((http://www.almaer.com/blog/archives/000846.html)) containers in Java/J2EE/JEEWhiz. Well then, doesn’t Javascript need the same thing? Again, I know, “you’re over-engineering it, Mahemoff”. So maybe I am. It’s been said that Dependency Injection doesn’t do much for Ruby, and maybe the same’s true for JS. So how then, do you make your browser app switch between a mock web service and a production service? The answer is not, “change the source code”, because to achieve continuous integration, you want to keep an automated test around that calls the mock service. Maybe the best thing to do is rely on DI – or some other context-switch mechanism – in the server, so that the Javascript that it spits out will differ according to the environment.

Podcast Interview with Alexander Kirk (Blummy, Bandnews)

This 47-minute podcast is a discussion with Alexander Kirk, creator of the recently released Blummy (which I mentioned last week) and also Bandnews.

Click to download the Podcast. You can also subscribe to the
feed if you want future podcasts automatically downloaded - check out the
podcast FAQ at http://podca.st.

A few things we discussed:

  • The design behind Blummy. (Interestingly, I was incorrect to guess it uses XHR – basically, the Blummy is like a Greasemonkey script – it uses DOM manipulation to alter the page you’re on, building up the container of all the Blummlets, and interacts with other sites using standard HTTP requests.)
  • Plans for Blummy, its use of Lazy Registration, security issues.
  • Developing for cross-browser compatibility.
  • Passing data from another domain with JSON.
  • The design behind Bandnews. Most band websites don’t bother with RSS, so Bandnews creates its own RSS feed, which is what the browser script reads.
  • Why some Ajax apps are slow and what can be done about it.
  • Libraries mentioned: dojotoolkit, prototype.

New Patterns: XMLHttpRequest Call and IFrame Call

Full drafts are now online for the two big Web Remoting patterns: XMLHttpRequest Call and IFrame Call. There are also a couple of new demos to illustrate GETting and POSTing with both call types: XMLHttpRequest Example, IFrame Example.

A few extracts follow, but first, let me ask you: Do you know of any public IFrame remoting examples? Google Maps is the only public example I know. All the other examples here are tutorials and framework demos.

The XMLHttpRequest alternatives was the most fun thing here (headings shown below). I went through the various techniques I’ve seen, read about, and tried out, to list as many ways as possible to programmatically call the server. As you might expect for these “Foundational Technology” patterns, there’s a lot more code in the Solutions than in later patterns, where the focus shifts more towards the -ilities, particularly usability and maintainability.

As usual, feedback much appreciated. (Mainly on the content, rather than woeful spelling and outright grammatical disasters at this stage.)

XMLHttpRequest Call

Solution: Use XMLHttpRequest objects for browser-server communication. XMLHttpRequest is a Javascript class capable of calling the server and capturing its response. Just like a browser or a command-line web client, an XMLHttpRequest issues standard HTTP requests and grabs responses. Note: The code shown throughout this demo is closely based on an online companion demo (http://ajaxify.com/run/xmlHttpRequestCall/).

The above example hopefully illustrates that the fundamental technology is pretty simple. However, be aware that it’s a very basic usage, not yet fit for production. Fundamental questions remain:

* How can XMLHttpRequests be created?
* How does an asynchronous call work?
* How do our code detect errors?
* What if the service requires a "POST" or "PUT" request rather than a "GET"?
* Which URLs can be accessed?
* How can you deal with XML responses?
* What's the API? 

The following sections address these questions and show how the code above needs to be modified. …

Decisions

6.1 What kind of content will web services provide?
6.2 How will caching be controlled?
6.3 How will you deal with errors?

Alternatives:

9.1 IFrame Call
9.2 Persistent Connection
9.3 Richer Plugin
9.4 On-Demand Javascript
9.5 Import XML Document
9.6 Image-Cookie Call
9.7 Stylesheet Call

IFrame Call

Solution: Use IFrames for browser-server communication. IFrames are page-like elements that can be embedded in other pages. They have their own source URL, distinct from their parent page, and it can change dynamically. IFrame Calls work by making the IFrame point to a URL we’re interested in, then reading the IFrame’s contents once the new content has loaded.

New Ajax Demos

Further to the previous post on new Ajax programming patterns, it’s also worth noting there’s a new bunch of corresponding online Ajax demos too. Being programming patterns, they tend to be “pure refactorings” rather than enhancements, thus the user experience is mostly the same – you have to look under the covers to see what’s changed. The demos are mentioned in individual patterns, but here’s a sampling:

  • The wiki has a bunch of timeout-related refactorings. These refactorings are user-visible. (They’re not actually part of the programming patterns, but they were recently added.)
  • 2 other wiki refactorings involve On-Demand Javascript . (No UI change.)
  • A JSON playground.
  • The Portal Drilldown Demo has been refactored to render with XSLT and browser templating tools. (No UI change.)
  • A couple of refactorings added to demonstrate REST (broken right now, probably due to a PHP config issue) and RPC on the Ajax Shop demo. (No UI change.)

As with all the demos, they’re tested on FF and IE6, please let me know if anything breaks.

18 New Ajax Programming Patterns

I’ve uploaded full text for 18 new Ajax patterns, completing a first-cut draft for all Programming Patterns content, which will be one part of the book. This section bridges the gap between the very basics of Ajax – XMLHttpRequest, DOM, etc – and the high-level stuff like widgets and visual effects. For instance, how do you do design the services accessed by XMLHttpRequest? Or how do you deal with a lots of Javascript? The focus here is on traditional tech concerns such as maintainability and understandability; and less about usability.

Breaking it down:

  • Web Services patterns are about overall browser-server architecture – what kinds of services and what kinds of messages? It’s relevant to the conversation of Jon Tiersen and others about what sort of responses are possible – XML, JS, HTML, JSON. It’s also relevant to another recent conversation, on [http://www.theserverside.com/news/thread.tss?thread_id=35979 Ajax and webservices], as it covers REST and RPC.

  • Browser-Server Dialogue covers a mixed bag of patterns about XMLHttpRequest handling and what you can do with it, such as loading JS on demand.

  • DOM Population is about transforming incoming XML into DOM content, e.g. XSLT.

  • Performance Optimisation covers a variety of performance patterns such as caching and pre-fetching.

  • Breakout (can you think of a better name) is about going beyond standard Ajax constraints, namely using a server-side mediator to access external domains and introducing a plugin to go where no Ajax app is allowed to go.

  • Code Generation and Reuse is strongly related to some of the emerging frameworks like Echo2 and SAJAX.

This is all early days, so any feedback on the structure, names, or content will be taken into account and acknowledged too (really!). OK, here’s the lot …

Programming Patterns

Web Services

  • RESTful Service Expose web services according to RESTful principles.
  • RPC Service Expose web services as Remote Procedural Calls (RPCs).
  • HTML Response Have the server generate HTML snippets to be displayed in the browser.
  • Semantic Response Have the server respond with abstract, semantic, data.
  • Plain-Text Message Pass simple messages between server and browser in plain-text format.
  • XML Message Pass messages between server and browser in XML format.
  • JSON Message Pass messages between server and browser in Javascript Object Notation (JSON) format.

Browser-Server Dialogue

  • Call Tracking Accommodate busy user behaviour by allocating a new XMLHtpRequest object for each request. See Richard Schwartz’s blog entry (http://smokey.rhs.com/web/blog/poweroftheschwartz.nsf/d6plinks/RSCZ-6CEQAR).Note: Pending some rewrite to take into account request-locking etc.
  • Distributed Events Keep objects synchronised with an event mechanism.
  • On-Demand Javascript Download Javascript as and when required, instead of downloading it all on page load.

DOM Population

Performance Optimisation

  • Fat Client Create a rich, browser-based, client by performing remote calls only when there is no way to achieve the same effect in the browser.
  • Browser-Side Cache Maintain a local cache of information.
  • Guesstimate Instead of grabbing real data from the server, make a guesstimate that’s good enough for most user’s needs.
  • Submission Throttling Instead of submitting upon each Javascript event, retain the data in a local buffer and upload it periodically.
  • Explicit Submission Instead of submitting upon each Javascript event, require the user to explicitly request it, e.g. submit upon clicking a button.
  • Multi-Stage Download Quickly download the page structure with a standard request, then populate it with further requests.
  • Predictive Fetch Anticipate likely user actions and pre-load the required data.

Breakout

  • Cross-Domain Mediator Allow the browser to communicate with other domains by server-based mediation.
  • Richer Plugin Make your application “more Ajax than Ajax” with a Richer Plugin.

Code Generation and Reuse

  • Ajax Stub Use an “Ajax Stub” framework which allows browser scripts to directly invoke server-side operations, without having to worry about the details of XMLHttpRequest and HTTP transfer.
  • Server-Side Code Generation Automatically generate HTML and Javascript from server-side code.
  • Cross-Browser Component Create cross-browser components, allowing programmers to reuse them without regard for browser compatibility.

Aside I was interviewed yesterday for a Japanese magazine about how I’m using the wiki. So maybe some people will be interested to know that I always write the patterns offline because I am more creative in Vim and also to avoid textarea hell. (So much for Writely, will anyone create Vimly, there’s gotta be more money it than cloning MS-Word online :-D). I also use the Mozex or ViewSourceWith extensions to make partial edits using Vim.

This time, I decided not to upload once or twice at a time; but instead all 18 at once. There’s serious overhead of introducing each new pattern to the wiki (from Vim, the sequence is: 2mins spell-check, 5-10mins markup massaging, 2 mins fixing the link and description on the homepage; sometimes exacerbated by server lag.) Uploading all at once at least allowed me to focus fully on the task and also made some aspects more efficient, particularly updating the homepage.)

Must Have 5 Years “Ruby on Rails” Experience

If there’s one certainty in this industry, it’s the ongoing hilarity of job ads. David Heinemeier Hansson found a job ad that mentions “Rails” in the same breath as “enterprise” and “whitepaper”, and “struts” in the same breath as “latest and greatest”:

Should have ideally 5 years experience with all of the latest and greatest tools out there. The more the better. J2EE, struts, Ruby on Rails, Websphere, etc. Should have experience constructing Enterprise wide Java solutions from whitepapers.

“Why yes, I just pushed the Rational Rose Activationate button and Rose generated the Enterprise wide Java solution for me, along with the whitepaper. I didn’t even have to tell it anything first. Does that count?”

David’s reaction:

Rails has truly broken through the awareness barrier when its included as part of a job application that’s the antithesis of everything Rails is about. Clueless recruiters of the world, I salute you!

The “Must have 5 years Java” requirement, at a time when Java was barely out of the Sun Labs, have become legendary. There can be no finer way to attract dishonest candidates than to construct an infinitely high hurdle. I’ll donate a Pragmatic Rails text to whoever spots the first serious (as in seriously misguided) Rails equivalent.

Will Ajax be a Household Word?

Alright, it already is a household word, but how about our Ajax? Will it too become a household word? I’m seeing a trend that it will be.

Some terms should never really have escaped the tech world, but they somehow did. For example, Java and Flash. Okay, maybe not quite household, but well beyond the realm of techies and beyond that of gadget-obsessed teens and buzzword-obsessed managers. If a phone is “Java-enabled”, that means it probably has some games with graphics you’d be proud of in 1985. If a website has Flash, it’s got some fancy graphics and animation.

Monitoring the blogosphere, there’s an increasing trend of Ajax, Ajax, Everywhere. Non-techies are beginning to pick up on the term. And whereas Web 2.0 was previously a vague mixture of feeds and sharing and folksonomies and long tails and podcasts, a lot of people are homing in on the most salient aspect of Web 2.0: the <drum roll please> … Web. While I think all those other aspects of Web 2.0 are also vital, they’re quite frankly more abstract. Podcasts may be the most concrete thing about Web 2.0 – if you present it as “Ham Radio for the Y2K+” or some such – but you still have a lot of explaining to do about RSS and portable player integration and why you don’t have to have an IPod.

Okay, so post-techie early adopters are starting to wake up to the fact there’s an Ajax revolution going on. What does that mean?

I’m not sure. What do you think it means? A few ideas …

  • Once people begin to get conscious about this, the effect will snowball. You’ll see Ajax in business mags and journalists will begin to write about how companies can benefit from internal Ajax websites, which will create demand on intranets. The effects of Ajax are far-reaching, because Ajax is the converging point of the two biggest genres: web apps and desktop apps. Most of the apps in both genres are Ajax-bound.
  • If you’re a programmer, learn Ajax now. I’ve been encouraging a few students and grads to learn JS and DOM backwards, because clients are getting fat. Pure UI in JS – the ultimate presentation-model separation. Learn it now and walk into all the Ajax jobs out there in 12 months. Or hold off and compete against those who spent the past 12 months working it, as well as the 500 others who have flicked through “Advanced Ajax in 60 seconds”. (And learn Ruby too, but that’s a mostly unrelated story.)
  • If you’re a public website, better think long and hard about Ajax. GMail versus Hotmail? Google Maps versus MapBlast etc? Leapfrogging anyone?
  • If you’re the leader in desktop apps, think even harder. The MS perspective is interesting. One thing MS has never been is complacent, as they showed last time round when Marc Andreeson told reporters how the web would make Office obsolete and they’re showing with Ajax too. We’re starting to see where Ajax fits on Microsoft’s Atlas (sorry) and it looks like they’re embracing the other browsers, but perhaps the server needs to be MS. If that’s the case, I don’t think there’s much for people to complain about, but with Ajax apps getting more and more desktop-like each day, there’s still room for things toget nasty. Regarding Office vs Writely and others, MS’s best bet is to fully embrace Ajax and at the same time exploit the full power of the desktop for the standard version. That’s costly maintaining both versions, but if they don’t follow a dual strategy, they really have no case over a decent Ajax offering.
  • If you have an idea, now’s the time.