Audio/Video Tag: Don’t Forget to load() before you play()

This is a gotcha that will catch a lot of people out because it goes against the expectation that you just change an image’s appearance by setting its “src”, done. And also, it’s a deviation from the more general convention that you change DOM properties declaratively to make stuff happen. Probably for good reason, given the temporal nature of audio and video playback, but just beware it’s a different model.

Bottom line is – load() between changing src and play()ing:

javascript

  1. function play(url) {
  2.   var audio = document.querySelector("audio");
  3.   audio.src = url;
  4.   audio.load(); // !HUL|_O! PAY ATTENTI0N!
  5.   audio.play();
  6. }

A lot of online sources talk about play() and omit to mention this vital fact, because they’re only play()ing a single track…and the first track works fine. This is an oddity – you can just call play() the first time and it will automatically load. This happens whether you set “src” in the initial HTML’s “audio” tag or whether you set it programatically. I did some fiddling and it looks like the behaviour is the same in Chrome and Firefox. It’s like the track is auto-load()ed the first time you set its source. (And note that Firefox doesn’t auto-buffer so that’s nothing to do with it.)

devx has you covered on this topic, and points out the subtle issues which you would need to look at for a quality production:

However, media by its very nature is a time-spanning process. You’re working with both the need to load and cache audio and video files and the network connection, which itself can prove to be a challenge that’s usually not a factor for most non-temporal resources. For these reasons, when dealing with video and audio on the web, if you want to take control of the process in any way, you have to work asynchronously, tying into the various events that are passed back to the client from the media-serving website. Table 4 (taken directly from the HTML 5 documentation) provides a full listing of the various events and when they are dispatched. … Of all the events in Table 4, perhaps the most useful are the canplay and canplaythrough events. The first, canplay, will fire when enough data has been loaded for the video player to start actually rendering content constructively, even if not all of the data has been loaded. The canplaythrough event, on the other hand, will fire when the data has essentially completely loaded into the browser’s buffer, making it possible to play the video all the way through without the need to pause and retrieve more content.

Yoink: Extracting All Scripts and Stylesheets on the Page

This here is a script that will put the code of all scripts and stylesheets into a single variable. Usage:

javascript

  1. yoink(function(all) {
  2.   console.log("scripts", all.scripts); // string array
  3.   console.log("stylesheets", all.stylesheets); // string array
  4. });

As you can see, you have to provide a callback because this stuff is asynchronous. It downloads the files one at a time to keep things simple and without using any fancy-pants Ajax queueing library. And right now, timeouts won’t be too graceful. Also, browsers will typically allow scripts and stylesheets to work cross-domain, but of course XHR won’t. So any external scripts and stylesheets are unfortunately out of reach and will not be captured. This is an especially tragic circumstance in the case where they are hosted on a different subdomain. But, as they say, “hey”. I don’t know what that means, but, as they say, “hey”.

Yoink will download inline scripts (script tags with bodies), remote scripts (script tags having a “src” attribute), inline style attributes (elements having a “style” attribute), inline style tags (style tags with bodies), and remote stylesheets (links of type “text/css”).

There are several purposes. I did a quick hack version of this a while back to improve logging info; in principle, a logging method could introspect on the source tree to show extra context info (locate the log message I’m currently outputting, and then say which line it’s coming from, for example).

In the case of Single Page Applications a la TiddlyWiki, a plugin might use Yoink to inline all the remote content.

Right now, I’m more interested in using it for browser extensions, such as the Feature Creep extension I suggested, to show what’s happening on the page. Some of that stuff you can work out by inspecting the DOM, but other stuff … show me the code!

Get it here – http://gist.github.com/378850. Shown below for convenience:

function yoink(complete) {

var all={ scripts: [], stylesheets: [] }

function downloadResources(urls, onComplete) {

var resources = [];

(function inlineResource(index) {

  if (index==urls.length) {
    onComplete(resources);
    return;
  }

  var xhr = new XMLHttpRequest();
  xhr.open("GET", urls[index], true);
  xhr.onreadystatechange = function() {
    if (xhr.readyState!=4) return;
    if (xhr.status==200) resources.push(xhr.responseText);
    inlineResource(index+1);
  }
  xhr.send();
})(0);

}

var scripts = document.getElementsByTagName(“script”); var scriptSources = []; for (var i=0; i

var styles = document.getElementsByTagName("styles");
for (var i=0; i<styles.length; i++) { all.stylesheets.push(styles[i]); };
var allElements = document.getElementsByTagName("*");
for (var i=0; i<allElements.length; i++) {
  var inlineStyle = allElements[i].getAttribute("style")
  if (inlineStyle) all.stylesheets.push(inlineStyle);
};
var links = document.getElementsByTagName("link");
var cssHrefs = [];
for (var i=0; i<links.length; i++) {
  if (links[i].type=="text/css") cssHrefs.push(links[i].href);
};
downloadResources(cssHrefs, function(resources) {
  all.stylesheets = all.stylesheets.concat(resources);
  complete(all);
});

});

} [/javascript]

SPA Hacks: Hacks Emerging From the World of Single-Page Web Apps

This is a permalink for my JSConf.US talk. Full slides are online here:

THE LOST HACKS: Ridiculous browser tricks from the world of single-page Applications

The talk will overview TiddlyWiki and Single-Page Apps, and then cover eight specific hacks:

  • File access without browser extensions
  • Javascript-HTML chameleon files
  • SVG-VML chameleon files
  • Inline SVG
  • iFrame squriting
  • Script islands
  • Embedded images
  • Fragment IDs

I’ll be posting a link to the slideshow notes from here. I’m pleased to say the nascent TiddlySlidy app has been a pleasure to dogfood it in.

[Image credits]

Automagic Event Registration

Further to last night’s post on custom events, I’ve set things up now to use “magic event registration”. It’s a little like the auto-wiring facility of a dependency injection container. It’s quite simple really – the app’s initialisation sequence does this:

  • Register all components that might listen to something.
  • Register all events they might listen to.
  • For each listener method among the components, automatically bind the event to it.

In code:

javascript

  1. var components = [siteFrame, singleTrial, trialController, statsZone, tableView, analysis, rawView, graphView];
  2.   var eventTypes = ["trialCreate", "trialUpdate", "trialSuspend", "trialRun", "trialComplete"];
  3.   $.each(components, function(i,component) {
  4.     $.each(eventTypes, function(i,eventType) {
  5.       var handler = component[eventType];
  6.       if (handler) $(document).bind(eventType, handler);
  7.     });
  8.   })

A component looks like:

javascript

  1. statsZone = {
  2.   trialCreate: function(e, trial) {
  3.     ...
  4.   },
  5.   trialUpdate: function(e, trial) {
  6.     ...
  7.   }
  8. }

And elsewhere in the code, the events it listens to get triggered through the normal jQuery custom events mechanism:

javascript

  1. $(document).trigger("trialUpdate", trial);

Part of this is based on what I said last night, which is (a) I’m keeping things simple – hence everything happens at startup and all the instances are singleton classes; (b) consequently, events are global, rather than being attached to any particular component.

Stepping back, what were the alternatives to this design:

  • Components register themselves as listeners – this would be the purist OO answer, i.e. keeping objects autonomous. A perfectly cromulent solution, but a little more redundancy than it needs to be. If objects really do have consistently named listener methods, it’s just as easy to handle registration automagically.

  • The initialisation routine manually wires up components to events. It has a certain Python-like “explicit” feel to it, but again is quite pointless if we can instead do it automagically.

So where does the automagic model fall down? If we are doing things dynamically, it would get more complicated and would, for example, require objects to register upon their creation. So I think that’s still okay. Another problem could be if a single method was listening to more than one event type, since this technique assumes it’s one-to-one from event type to methods. But that’s okay too – I actually had this problem and it was simply solved by getting both handlers to delegate to a third, common, handler:

javascript

  1. var trialController = {
  2.   trialCompleteOrCreate: function() {
  3.     ...
  4.   }
  5. }
  6. trialController.trialCreate = trialController.trialComplete = trialController.trialCompleteOrCreate;

I realised while working on TiddlySpace there’s a lot to be said for dependency injection style patterns in Javascript, and it’s not happening yet.

Joining Google

I mentioned I’m moving on from Osmosoft a few weeks ago and after 12days and a week of doing much less than one could ever have imagined, it’s time to announce where I’m going: Google. I’ll be joining Google as a Chrome developer advocate, which means focusing on HTML5, Javascript, the Chrome browser, ChromeOS, and related technologies.

I’m very excited about the role, for reasons best explained in terms of a timeline for the web. I see the history of web technologies divided into three phases and I believe we’re currently in the transition from the second to the third phase. Which is the starting context for this role.

Dark Ages (Web as a Black Art)

Rich web apps were hard work. Black art even. During this phase, I spent most of my time on server-side languages – Perl, PHP, Java/J2EE – and Javascript was a weird sideline. Taking an interest in the user experience, I always wanted to know more about this black art, since it’s the web technologies that touch the user and have the most direct influence over their experience with any application. But what paid the bills in the late ’90s and early noughties was form submissions and page refreshes. And most Javascript content was odd little ticker tape scripts you could download, not too much to learn from, and always difficult to achieve in one’s spare time. I think this was the frustrating experience for many web developers: not enough time and hard work. I did get by and pick up a few tricks here and there, but things only took off when the A-word was unleashed …

Ajax Era (Web as an Established Practice)

Everything changed five years ago, on February 18, 2005. Jesse James Garrett hopped in the shower, came up with the term “Ajax”, and published his thoughts on this rising architectural pattern. Many people bristle at the notion that a simple term, for something people were already doing, can make such an impact. “BUT I already knew how to do that!” they cry. Good for them. But the fact is, Ajax allowed the whole developer community to rally around this style of application. Events, blogs, books, libraries, tools, the whole thing flourished. I got seriously interested, recorded a podcast explaining the topic, and started diving into all the existing applications, which ultimately led to the Ajax Patterns wiki. All those applications, and many featured in the book, were created in the Dark Ages by brilliant pioneers who had scant resources to learn from. Today, such applications can be created at exponentially faster rates. This is thanks to improved knowledge, improved libraries, and improved tools.

There are historical analogies to this era, e.g. the rapid rise of the automobile industry once all parts were in place. In that case, it was more a matter of discovering new physical and engineering principles, whereas with Ajax, it was a superficial act. The browsers were already there, with a killer feature like IE’s XMLHttpRequest sitting largely unnoticed for five years. Few people had taken the time to really understand how they worked. It was the Ajax term to trigger the boom.

HTML5 Era (Web on an Improved Platform)

(I hesitate to call this the era of HTML5, as many things happening are outside HTML5, notably improvements to CSS and the core Javascript/ECMAscript language. But HTML5 is increasingly becoming a brand, much like Ajax in its heyday, a term that’s familiar to anyone with a passing interest in technology.)

The starting point for this era is the success of open web technologies. While we continue to learn more each day, we have the basics in place today. We know how to harness HTML, CSS, Javascript to make powerful libraries and applications on the existing browsers. This is where the Ajax Era landed us. We have seen some genius hacks. In fact, we have seen MANY ingenius hacks. On-Demand Javascript and JSONP, for example, are completely fundamental to the way business is done on the web these days, and is utterly a hack. Full credit to those who came up with them, but these hacks can only go so far. They can increase programming complexity, impact on performance, and cause security threats for those not fully-versed in their capabilities. And that’s just for the things we can do. The bigger problem is that we are heavily limited in our functionality on Ajax-Era technologies. We can’t do rich graphics or video, for example. So the next stage is all about improving the underlying platform. It’s not something an application programmer, however heroic, can do on their own. It’s something that has to involve improvements from the platform providers, i.e. the browser manufacturers and others.

To draw a line in the sand, take IE6. Although it was created back in 2001, we could do far more with it in 2004 (e.g. Google Maps) than we could in 2001, and we can do far more with it in 2010 than we could in 2004. This is because of all the knowledge, libraries, and tools the Ajax Era has provided. However, we’re in serious plateau mode now. We can continue to squeeze out 1-percenters, but the low-hanging fruit was plucked and digested a long time ago. The next stage has to be an improvement to the underlying platform, and that’s what’s going on today with modern web browsers.

Once you say “okay we’re now going to build the next-gen web platform”, you’re not just going to add support for round corners. So there’s been a truckload of effort going on, in HTML5 and beyond. Graphics in the form of canvas and SVG and WebGL and CSS3, media in the form of audio and video tags, semantic markup improvements, new form controls, offline storage, support for location-awareness. And much more.

We have the new platform and alongside it, we have the activity of understanding how to use the new platform. Not just understanding the parts, but the whole. How do you put all these things together in meaningful ways. Can we show custom-font text on a canvas? If so, how? And when does it make sense? And how to avoid performance problems? Or, to take another example, what kind of CSS transforms can we use to render videos? And since we can grab image data from the videos, what are some cool things we can do with that data? And all the while, where’s the graceful degradation story here? How do these apps look in older browsers? Most of the talk so far has been on the capabilities of the platform, not patterns of usage.

All these new capabilities take us to the place we’ve been aiming for all along: true applications deluxe, not just “html++”. Of course, there’s plenty of room for both of these, but we’re now in a place where we can really build true applications with all the capabilities of a traditional native experience. We’re seeing it in a big way in mobile space. The new model is all about sandboxing, doing away with the filesystem, and explicit permissions. Exactly what the web’s good for. They’re not only a good fit technically, but they’ve become the ubiquitous lingua franca among the programming community. The technologies are familiar, as are the patterns of UI design and code composition. Those patterns will continue to evolve with HTML5 and associated technologies.

We’re just scratching the surface here; there are exciting times ahead!


I’m not an official Googler yet, nor even a Noogler, as I’ve taken time off to chill, smell the roses, and pursue a few projects (like dusting off the homepage and … watch this space. Actually, watch that space). I’m starting in about a month. And of course, even when I am official, this blog continues to be my own opinions, caveat emptor yadda.

The role covers EMEA and I’m still based in London. Despite not starting just yet, I’m looking forward to participating in tonight’s Chrome/Chromium OS and HTML5 London event. See you there!

(I’m not the only one to announce a move to Google at this time. I’m looking forward to meeting and working with many of my talented future colleagues.)

Unintended Consequences and the Inevitable “Why Would Anyone Want To Do This?”

I’m listening to this excellent BBC podcast on Unintended Consequences of Mathematics.

In his book The Mathematician’s Apology (1941), the Cambridge mathematician GH Hardy expressed his reverence for pure maths, and celebrated its uselessness in the real world. Yet one of the branches of pure mathematics in which Hardy excelled was number theory, and it was this field which played a major role in the work of his younger colleague, Alan Turing, as he worked first to crack Nazi codes at Bletchley Park and then on one of the first computers. Melvyn Bragg and guests explore the many surprising and completely unintended uses to which mathematical discoveries have been put. These include: The cubic equations which led, after 400 years, to the development of alternating current – and the electric chair. The centuries-old work on games of chance which eventually contributed to the birth of population statistics. The discovery of non-Euclidean geometry, which crucially provided an ‘off-the-shelf’ solution which helped Albert Einstein forge his theory of relativity. The 17th-century theorem which became the basis for credit card encryption.

The relevance to the topic of this blog should be clear.

Show me someone doing something cool or experimental and I’ll show you someone who sniffs, “Why would anyone want to do this?”. The answer may be because it was fun, because they wanted to see if it was possible, or because they wanted to learn something. But whatever their primary reason, one thing’s for sure: there will be unintended consequences. The examples above show it’s happened in mathematics, and we’ve seen the same thing happen time and again in web development.

The rich interactivity we see today wouldn’t have been possible if people hadn’t been willing to fool around with the not-so-always-obvious features of web browsers. Take cross-domain iframe messaging, for example. Not something people knew much about until a few years ago, when James Burke documented his experiments. A few years later, it’s a fundamental technology in OpenSocial (which means iGoogle, the Yahoo! homepage, among many other sites), Facebook’s official Javascript client, and evidently has much interest from elsewhere.

So next time you see some wit ask “Why would anyone ever need this?”, just stay schtoom, sit back, and wait six months.

Not Your Grandpa’s Framesets: Premasagar Rose shows us IFrame 2.0!

usual live blogging caveats – spelling errors, messy, etc etc

@premasagar is visiting the Osmoplex today (thanks @jayfresh for arranging it) and is taking us through his work on iframes, widgets, and sandboxing. I’ve realised we could perhaps be collaborating as my jquery-iframe plugin is so close to his. Different emphases, but much overlap.

GitHub is where you can see what he’s been working on. Basically, this guy is a guru on all things iFrame. In particular, all the quirks around squirting dynamic content into iFrames, as opposed to pointing them elesewhere using “src”.

QUOTE OF THE DAY

“sqwidget is my tiddly”

BACKGROUND – THE EMBEDDING PROBLEM

In patternesque speak, the basic problem is:

Problem: You want to embed 3rd party content into another site.

Forces:

  • You want the 3rd party content to have its own style
  • BUT it will inherit style from the parent page

SOLUTION 1

IFrame

Works great (in fact, I’m using it in my TiddlyWiki playground app, to be documented at some point, and is similarly used in Jon Lister’s TiddlyTemplating.

The problem is you sometimes want the widget to jump out of the iframe, e.g. a ligthboxed video. So …

SOLUTION 2

CleanSlateCSS

Basically a CSS reset, but whereas CSS resets will only handle browser defaults, cleanSlate blats everything! This is exactly the kind of thing I was looking for when I was trying to cancel out tiddlywiki styling. In that case, I was flipping the entire page back and forth, so I could just cheat by removing stylesheets and re-add them. (Prem pointed out there’s a “disabled” attribute on style tags – true, so I should really use this instead, assuming it’s portable, which he thinks it is.)

Problems: – Difficult to maintain CleanSlate library, because new CSS stuff and browser quirks keep coming up – IE6 and IE7 don’t support “inherit”, so need CSS expression. – When using Javascript to interact with CSS style properties, e.g. slideDown(), these will override CleanSlate. The solution is to set the “style” attribute with !important, but it becomes an arms race! – Doesn’t solve iFrame security model

SOLUTION 3

Inject (aka squirt, inject; summary) content into a fresh iframe.

The content comes from the widget site. The sqwidget library injects it. This resolves the tension with wanting independent CSS on the same page. If the sqwidget library is running on the host page, it could even (potentially) lock down capabilities, i.e. do Caja-style sanitisation.

Sqwidget also does templating, using Resig’s micro-templating. (That thing’s getting to be very popular; I’m using it myself in Scrumptious via the UnderScoreJS library after @fnd gave us a talk about them.)

Also, prem is playing around with the idea of a div, with custom (data-*) attribute pointing to the third-party URL. You could put inside it “now loading” and then the script tag will pick those things up and load them.


Various points: