What Everyone Should Know About REST: Talk at Web Directions Code

Here are the slides:

Slides: What Everyone Should Know About REST

Sketchnotes

Thanks UX Mastery for the sketchnotes, they are awesome! (Seriously, I would be much more swayed to speak at any conference with sketchnotes because it’s a straightforward permanent memento, a better snapshot than slides or video.)

Overall, it was great to be associated with another fine Web Directions conference and the Melbourne Town Hall venue was amazing. I only regret that we were so busy scrambling on the Android app, after launching just a few days earlier, to be around the whole time. But this being my hometown — I’ll be back!

Talk Structure

I spoke at Web Directions Code on Friday, a talk on REST. I’ve been putting a lot of this into practice lately, and the talk was really an attempt to convey the main practical things every developer should know. The structure was:

  • Everyone should know about REST because it’s not just about websites anymore. Devices, whether computers, fridges, or wearable glasses – are connected, and device-to-device communication happens with web standards, i.e. HTTP. The talk covered three things about REST: Simplicity+Consistency; Security; Caching.
  • Simplicity+Consistency: Emphasising Developer Experience (#devexp) was a way to frame the general concepts, ie URLs, HTTP methods, response types.
  • Security: How the web is becoming SSL-only, and various authentication schemes. I referenced the latest Traffic and Weather, which has a good discussion on this.
  • Performance+Scalability: Mostly about caching. I’ve been musing on REST caching quite a bit for Player FM’s API (most recently thinking about a kind of reverse patch protocol, where the server can send out diffs that get cached), and explained some of the standards and tricks for squeezing efficiency out of the network.

What Wasn’t Covered

  • I didn’t go into the REST acronym or the general theory of REST as an architectural pattern arising from specific forces.
  • SSL and caching. Good Twitter conversation afterwards about this point, that you can’t cache in the middle of an SSL connection. The answer is to split the connection in the middle and run SSL on either side, with a trusted cache seeing plain-text in the middle. This is how Cloudflare works, and the CEO Matthew Prince chimed in to say it will be free soon. (At least, SSL from client to Cloudflare.) So that means the SSL-protected web could triple overnight.

The Weirdness of Ajax Redirects: Some Workarounds

I’ve been dealing with the situation when your server redirects after a DELETE. Which is actually a fairly common use case. User deletes a blog post, redirect them to the index of all posts. The article below has some Rails workarounds, but is relevant to anyone using XHR and perhaps HTML5 History, ie many single-page web apps.

It’s important stuff to know if you’re HTML5ing in earnest, because there was a recent change to Chrome which makes it standard-compliant, but fundamentally different to other browsers and also makes it handle DELETE and other verbs different to how it handles POST and GET.

DELETE redirection

This redirection becomes troublesome if you want to do this in a way which will seamlessly support Ajax calls, as well as standard web requests. It’s troublesome because of a lethal combination of two browser facts.

Firstly, XHR automatically follows redirects. Your JavaScript has no way to jump in and veto or alter any redirect that comes from the server. XHR call fires off to server, server responds with 302 status code and a location header, the browser automatically issues a new request to the specified location.

Secondly, a DELETE request, when it’s redirected, will actually cause another DELETE request to the new location. Yes, if you’re redirecting to the index, congratulations … your server now receives a request to DELETE the fraking index. You lose!!! So the very normal thing to do for a non-Ajax app suddenly becomes an epic tragedy. It’s completely unintuitive, but it’s actually the standard! I learned as much by filing an erroneous Chrome bug. Turns out Chrome’s unintuitive behaviour is actually correct and Firefox and Opera were wrong. Even more confusing, I was seeing POST requests being converted to GETs, and it turns out this is also part of the 302 standard – POSTs can be converted to GETs, but not DELETEs (or PUTs etc I assume).

Just now, I made a little workaround which seems to be working nicely in my case. In my AppController, which is my app’s own subclass of ActionController, I simply convert any 302 response (the default) to a 303, for redirect_to calls. I should maybe do this for XHR calls only, to be “pure” when a normal call comes in, but no real difference anyway.

  1. class ApplicationController < ActionController::Base
  2.  
  3.   ...
  4.  
  5.   # see src at https://github.com/rails/rails/blob/master/actionpack/lib/action_controller/metal/redirecting.rb
  6.   def redirect_to(options = {}, response_status = {})
  7.     super(options, response_status)
  8.     self.status = 303 if self.status == 302
  9.   end
  10.  
  11. end

Finding the new location

While I’m blogging this, let me tell you about another related thing that happens with redirects (this section applies to any type of request – GET, POST, DELETE, whatever). As I said above, XHR will automatically follow requests. The problem is, how do you know where the ultimate response actually came from? XHR will only tell you where the initial request went; it refuses to reveal what it did afterwards. It won’t even tell you if there was any redirect. Knowing the ultimate location is important for HTML5 apps using the History API, because you may want to pushState() to that location. The solution is to have the web service output its own location in a header. Weird that you have to, but gets the job done.

  1. class ApplicationController < ActionController::Base
  2.   ...
  3.   before_filter proc { |controller| controller.response.headers['x-url'] = controller.request.fullpath }
  4.  
  5. end

REST is good, but REST-* is …

The above image is inspired by Roy’s Rightful REST Rant, kanyelicious and kanyegate, uses @psd’s Roy Has a Posse, and is me getting up to speed with Photoshop (not that it shows) after using certain less worthy tools for image manipulation.

I’ve said a bit about this on Twitter, but basically a couple of years ago I saw REST beginning to take off in the enterprise, it was inevitable, and was amused by the prospect of vendors “embracing and extending” it for the enterprise; just as other useful terms like “synergy” and “agile” get sucked into the vortex of hollow lip service.

And now, with REST-*, it’s happened.

So it was nice to see Roy Fielding wrote the following on REST-discuss, a much tweeted tirade:

Bill, if you want people to have an open mind about what you are trying to do, then the respectful thing would be to remove REST from the name of your site.

Quite frankly, this is the single dumbest attempt at one-sided “standardization” of anti-REST architecture that I have ever seen. It even manages to one-up the previous all-time-idiocy of IBM when they renamed their CORBA toolkit “Web Services” in a deliberate attempt to confuse customers into thinking they had something to do with the Web.

Distributed transactions are an architectural component of non-REST interaction. Message queues are a common integration technique for non-REST architectures. To claim that either one is a component of “Pragmatic REST” is the equivalent of putting a giant Red Dunce Hat on your head and then parading around as if it were the latest fashion statement.

The idea that the community would welcome such a pack of marketing morons as the standards-bearers of REST is simply ridiculous. Just close the stupid site down.

Scrumptious Update: Open ID Support, UI Enhancements

I’ve released a new version of Scrumptious. Main change is it now supports Open ID. You can click a “login” link to comment by Open ID. It’s optional by default, though a Scrumptious site operator could easily make it mandatory for read, write, or both by just changing a couple of words in the “policy” config files. Similar to other CMSs like WordPress, non logged in users can indicate their name and URL when they submit a comment.

There are also UI enhancements – the design is cleaner and looks closer to the original original TiddlyWiki comments plugin. Interestingly, I retained almost identical markup, so I was able to cut-and-paste the original CSS for the comments plugin and it mostly worked. I also now include the default TiddlyWiki stylesheets as well. It’s not just look-and-feel which is closer to the original plugin, but the content – you now have info like modifier, modified date, and a permalink available.

I also added something I always wanted to add on the original plugin, which is some animation, e.g. when you add a new reply, the page scrolls to that reply, and a yellow fade effect highlights it. This is a genuinely useful feature as I was finding it difficult to see which reply I’d just added, when there are a lot of comments around.

I’ve also begun work on a Comments Report showing recent comments. Obvious related enhancement is to take the TiddlyWeb Atom plugin and make a comments feed.

Right now, all this is only tested on Firefox (the original was tested on all browsers, at least the full website view); my next priority is to work on browser compatibility, and after that, extract a modular JQuery comments plugin.

Implementation Notes

Regarding the implementation, TiddlyWeb ships with Open ID by default (Open ID is one of two default challengers, the other being the usual simple user-pass key pair config). The most challenging thing here was getting the UI right for both anonymous users and logged in users, as well as handling a redirect in the popup after a successful login; but at the back end, Open ID “just works”.

In summary, I added Open ID support as follows:

  • Add a “login” link to the TiddlyWeb OpenID challenger UI, using a “target” attribute so the challenger opens in a popup.
  • The challenger URL in that link also contains a redirect param, which I redirected to a new static page. This static page shows the user their login ID (by inspecting the “tiddlyweb_user” cookie value), calls a callback “onLogin” method on the original page, and closes itself.
  • The “onLogin” callback updates the login display to show the logged in user and a logout link; the logout link simply runs some Javascript to remove the cookie. The callback also updates any forms that are open prompting for bio info; it hides this info and in its place, shows the current user ID (read-only).

Thanks to Ben and Chris for pointing me in the right direction on TiddlyWeb’s Open ID support.

PS I discovered late in the day (literally) that TiddlyWeb lets the client specify whatever modifier they want

When Resource IDs are URLs

Being a web app about websites and other resources, Scrumptious has a resource which is basically a URL, called “Pages”. A Scrumptious resource modelling google.com/about looks like:

http://scrumptious.tv/comments/bags/ pages/tiddlers/http%3A%2F%2Fgoogle.com%2Fabout

See? The resource ID is http://google.com/about, encoded. (encodeURIComponent("http://google.com/about")

This was working fine on my dev machine, running “twanager server comments.boz 8080″ (comments.boz being an alias for my local machine). But on the server, and run through apache and TiddlyWeb’s apache.py, it failed:

The fix was twofold – both of the following were required:

  • AllowEncodedSlashes On in apache config. This option ensures encoded slashes (%2F) are passed through to the end-app.
  • PathInfoHack plugin As with any TiddlyWeb plugin, I downloaded it and added it to ‘system_plugins’ list in tiddlywebconfig.py. (Thanks Chris for the pointer.)

And now we can happily talk resources with URLs as IDs to the server.

A URL Parameter Engine (and Rambling about PHP Globals)

I’ve recently been playing around with PHP again, because (a) it’s vastly simpler to deploy personal projects in PHP than any other platform (aside from pure client-side Ajax of course!) (b) it’s so easy to get simple stuff done.

Anyway, one thing I’m doing is creating a RESTful service where parameter crunching is at an all-time high. The ancient way this was done was with magically created globals. i.e. For the URL http://example.com?id=10, you would end up with a global called $id, set to 10. But the more contemporary way to do it is to use a global array called $_GET, i.e. $_GET[$id]. This is considered safer, to ensure malicious users don’t set important globals. And in PHP 6.0, the former way is being obliterated altogether, since the relevant configuration option – register_globals is being altogether removed.

Well, to me, this is throwing out the baby with the bathwater.

This comes down to how you view PHP. For me, if I want to build a massive, complex, piece of software, I will use Rails or Java. So I consider PHP a scripting language and I value concise expression highly. Features like namespaces (to be added in PHP 6) and even OO don’t mean all that much to me in this language. I am much more concerned with syntactic sugar.

Thus, I would prefer to see templates look like this:

  1. Your name: <?= $name ?>

… and not:

  1. Your name: <?= $_GET["name"] ?>

A little thing, but in the context of a large corpus of templates, each with many variables, it makes a big difference.

So I’m working on a little parameter-processing engine which will let me use the “evil” $globalVar notation, but in a way that requires each script to pass in a whitelist of acceptable variable names. It also has some other value-adds, such as passing in default values. And these default values can be other parameters that were passed in.

(Incidentally, even if I wasn’t building this script, the first thing I would do would be to globally alias $G or to the uglier and more verbose $_GET.)

REST turns the URL into a command line, and this should push web framework developers towards smarter ways of processing parameters. Think Getopts, but for the URL instead of the shell command line. And that’s where I’m heading.

Did you hear the one about enterprise reuse?

Confirming that enterprise reuse can be a bit of a joke at times, Jason Gorman shares this fable on enterprise reuse (via another inspired Jason). Short summary: Two ladies could save 8 cents by boiling tea in the same kettle. But the cunning analyst forgets that, since they live 20 miles from each other, there will be overheads to the tune of a $20 cab ride and the travelling time.

Viewed from a high level, enterprise reuse is a noble goal; what’s the point of being a single company if everyone writes their own code? In practice, it can be fraught. Ironically, it’s usually easier to reuse publicly-available libraries (e.g. open-source libs on sourceforge) and public web services than those in the same company. The following things make reuse more digestible in an enterprise setting:

  • Language-agnostic, industry-standard, technologies Using obscure or proprietary technologies can work in an individual team, but rarely in a large enterprise; in most cases, there are simply too many factions with different skill sets and legacy code bases. There are companies that describe themselves as a “pure Java shop”, for example, but you will indefinitely find pockets working in Python, .Net, and so on. Getting an enterprise to truly standardise (not just lip service) on something non-industry-standard is futile. It takes several months for people to get really competent in a new language; in an environment full of contractors and staff turning over every few years, and full of legacy systems, you can count on the fact that there will be disparate technologies at play. It’s a good thing, too; no one language (or paradigm, for that matter), not even Java *gasp*, is the right solution to all problems.
  • Service-oriented SOA as in “built on a needs-driven basis”. The stuff that’s available for reuse is stuff that’s been abstracted from real-world projects, where at least one project already built it and at least one other project actually needs it. (Rails is successful because 37Signals uses it; there aren’t dozens of 24-month working groups involved.) Perhaps the biggest mistake enterprises make in this whole area is pushing out functionality no-one else actually wants to reuse.
  • Support trumps standardisation The best way IMO to encourage a certain technology or library is the carrot, not the stick. Make people actually want to reuse what you have to offer, rather than forcing them to do so. I am very sceptical about any situation where architects have to act as the reuse police; if the component or service was designed, documented, easily located, and served a genuine need, wouldn’t the developer be drawn towards it? Wouldn’t they actually want to use it, and maybe even give something back to it? In an ecosystem where components and services are high-quality and easily-accessed, you can forget about mandating reuse because it will happen anyway. See Web API Patterns and Documentation as Conversation for the kinds of things that will make this happen.
  • Online As a rule of thumb, offering a centralised web service is better than offering a reusable code component. The web service can (should) be easier to use and is language-agnostic. Obviously, there are sometimes situations where code components make more sense, especially from a performance perspective. I wouldn’t use an online service to create a polygon every millisecond, for example.
  • Easy to use As with any API, it should be easy to learn and make calls. For this reason, online services should be RESTful, not SOAP or CORBA or whatever MQ if you can help it.
  • Iterative progress Don’t try to bite off more than you can chew; if you start pretending *everything* can be reused, you’ll soon find that nothing gets reused.
  • Simple and parsimonious Factor out the trivial factors that relate only to one particular client. In enterprise reuse, this can be a big problem, where client projects may be the budget holder for anything reusable. It’s difficult, but someone needs to stand up and say “no, we’re not going to include feature X because no-one else would actually need it”. In software, deciding what to leave out is usually a greater challenge than coming up with new things to put in. Any feature that won’t be used by a significant proportion of client apps is going to create more clutter than its worth. In a broad-scale service, I’d say this minimum proportion should be something like 5-10% (e.g. Each method should be exercised by 5-10% of clients who used the class.) In an enterprise context, where there may only be a few clients, I’d say the criterion should be “at least 2 clients”. (There was a podcast interview a while ago, with PragDave I think, where he was asked what he would include in Rails 2.0. He essentially replied that he’s more worried about taking things out – push them out of the core distro and into plugins.)
  • Automated Sometimes, people think “it’s all under the same roof”, so getting access and learning about an API requires a call or a meeting with the owner of the reusable service/component. Whereas, if Google offers the same thing, it will provide online doco and a means of accessing it automatically, without any human intervention. An agile enterprise should aspire to do the same thing; it doesn’t have to be as polished as a public offering, but the spirit should be the same. Otherwise, it won’t scale, and the owner will soon become fed up doing the same thing over and over.

Ruby is Rails is … REST

Will the peripheral IT community come to view REST and Rails as equivalent? It might sound ridiculous, but consider: Unix==Linux, Wiki==Wikipedia, Ajax=Web 2.0, blogging==RSS, podcast==spoken MP3. Last but not least, every knows that

Ruby==Rails

So it only stands to reason that the REST equivalence shall come to pass, as REST hops on for a free ride on the Rails worthy-hype machine.

REST==Rails

 

(For the sake of completion, yes, Ruby==REST.)

The possibility of Rails==REST didn’t occur to me until I read this – probably quite necessary – clarification:

I want also take a step back for a moment, and mention that REST is not a Rails-only thing. REST was developed long before Rails, and is simply an approach that can be applied to many kinds of software applications.

If Rails can bring REST to the fore of mainstream development, in addition to other goodies like Convention Over Configuration and Migrations, that’s one more reason to heartily <heart> Rails.