• jasmid on Github
• Online demo

Read on for the complete notes/transcript of the talk (in hopefully more coherent form than the talk itself – next time I promise to spend less time on the flashy demo and more time figuring out exactly what I’m going to say…)

Right now everyone’s jolly excited about the HTML5 audio element. At last we have a standard-compliant way to drop audio clips into web pages that avoids all the gunk with external plugins and replaces it with a simple tag (well, once you’ve fought through the details of which browsers support MP3 versus Ogg anyway). It has a comprehensive API to handle all the details of buffering, programatically pausing, playing and skipping and so on – but one thing it stops short of is being able to generate the audio data on the fly, within the browser.

Why would you want that? Well, I can’t say why you’d want it, but I can tell you what I’m hoping to do with it: I’m involved in the demo scene, a community of programmers, artists and musicians who create digital art – something roughly like music videos, but with visuals generated in real time – and I’m working on a forthcoming website that will showcase those productions. This community originates from the days of the Commodore 64, when people cracked games and added little intro animations to promote themselves, which became more and more elaborate as rival groups tried to outdo each other, until they evolved into full-scale artistic creations and the game cracking side of things took a back seat. And among the artefacts to come out of this community is a hell of a lot of music – we’re talking hundreds of thousands of tracks, all preserved in the native formats of the Commodore 64, and the Amiga, and all sorts of other things. And it would be quite neat to be able to play all of these from within my website.

I’m using this site as an excuse to play around with cool technologies, and at first I figured that this was an ideal job for Amazon EC2 – set up a bunch of instances churning away in the background converting these files to MP3. However, when I started to learn about ways to generate audio in the browser, it made sense to take advantage of that and save myself a whole lot of up-front processing (not to mention bandwidth).

At the forefront of this new development is the Mozilla Audio Data API, available in the latest nightly builds of Firefox. This extends the HTML5 audio API with a few new methods, the central ones being mozSetup – which allows you to initialise an empty audio stream with a specified sample rate and number of channels – and mozWriteAudio, which lets you pass in an array of floats representing some sample data to add to that stream. Like all good up-and-coming browser innovations, we can reasonably assume that once the Mozilla developers have got this API stable enough they’re going to submit it to WHATWG for inclusion in the HTML5 spec – but for the moment, it has somewhat limited adoption. There is a remedy for that though…

At my last Barcamp London, two years ago now, I gave the first public showing of JSSpeccy, my ZX Spectrum emulator written in Javascript, which has been something of a runaway internet hit – and I’ve been somewhat surprised by the number of people taking it seriously, rather than as the crazy pointless hack I built it as (not least, the guy who ripped it off and sold it on the App Store as the first ever iPhone Spectrum emulator, despite it running at about 30% speed *cough*). But one guy who’s picked up the concept of emulation in Javascript and taken it much further than I’d ever dreamed possible is Ben Firshman, who created JSNES, the Javascript SNES NES emulator, featuring a whole host of advanced optimisations and new features, including audio support.

To achieve this, he created the dynamicaudio.js library, which sits on top of Mozilla’s Audio Data API, but also provides an invisible Flash widget for other browsers to fall back upon. Armed with this work, I was able to build the first step towards my goal: jsmodplayer, a player for the MOD music format originally introduced on the Amiga. It’s a rather messy format to implement, with every man and his dog coming up with their own custom extensions to it in a very ad-hoc way, but at its heart it consists of a set of uncompressed wave samples (typically a second or two in length), and a script detailing when to trigger them and at what pitch. Put enough of those trigger events together, throw in some effects like volume control and pitch slide, and you have a song.

Now, as we all know from the Apple versus Adobe tiff, Flash is not exactly universal across the platforms we care about – so we haven’t covered all of our bases yet. However, for certain applications, there’s a possible third path (albeit one that I haven’t properly investigated yet), using another recent browser addition: data: URIs. Unlike typical URLs, which point to some external location that contains the data we want, a data: URI embeds that data directly, as a string of base64 or URL-encoded data. This means that we could generate a string containing a valid WAV file from Javascript (or indeed an MP3 or Ogg file, although generating those from Javascript is a little bit hardcore), and use that as the source of an <audio> element.

This does depend on us being able to generate the audio up-front before starting playback, so it’s arguably not truly ‘real time’ – something like an emulator, which is generating audio in response to user interaction, couldn’t really do this. It’s good enough for our straightforward audio player, though.

For a while there was an unfortunate flaw in this plan: Chrome didn’t currently support WAV as an audio format. The bug tracker ticket relating to this featured some rather flimsy arguments defending this decision, such as “if we support WAV, people will start widely serving audio across the web as uncompressed WAV files” (um… just like everyone on the internet is using BMP files, which are supported by all major browsers, right?). Given the tendency of bug tickets to wander off onto unrelated subjects, it’s hard to tell what the eventual conclusion to this is – but if I’m reading it right, we can happily use WAVs as of Chrome 7.

(At this point, someone rightly mentioned that Internet Explorer imposes a 32K limit on URLs, so that won’t get you much of a wave file – especially with base64 or URL encoding to contend with. As such, all we can really do is hope that IE users will tend to have the Flash option available. Personally, I think it makes a refreshing change that we can even consider IE in our cutting-edge browser hacks once again…)

In fact, there’s another mechanism for feeding data into URLs dynamically, which I’d all but forgotten until I started preparing this talk: if you have a ‘javascript:’ URL which returns a string when executed, that string will be used as the data. This trick was most prominently used in Wolfenstein 5K (long before the wider world caught on to the joys of Javascript size coding contests…), which constructed its display as an XBM image, an obscure text-based format. Before canvas came along, I spent many happy hours trying to replicate that trick with GIF images, discovering along the way that Internet Explorer didn’t like strings containing zero bytes, which sent me down the garden path of constructing valid GIFs containing no zeros. Ah, happy days. In short, I don’t even know if this works at all with <audio>, and even if it does, it’s almost certainly as much of a dead end as it was back in 2004…

One project of mine where I really should have seen the value of generating the audio up-front was Fake Plastic Cubes, a demo I wrote this summer to play around with the size reduction tricks I’d seen coming out of the 10K Apart and JS1k contests, and to try and come up with something audiovisual in as small a size as possible. On the audio side, that meant ditching samples entirely, and doing the synthesis from first principles, building the sound up from plain sine waves – but in a classic case of project management fail, I spent a week building a really wonderful synthesiser framework and rushed everything else at the last minute, meaning that I had no chance to actually make something nice on top of it. As a result, it’s totally unpolished, and the animation keeps stuttering for a split second while it generates the next chunk of audio. It’s only a tiny fraction of the available processor time, but it’s enough to be very, very noticeable. I really should have just generated the entire audio track on startup and then kicked off the visuals – but there was no time for that, or to play around with web workers which would seem to be another potential way to generate audio as a background process…

I also didn’t have time to compose a decent soundtrack, or experiment with the synth enough to come up with interesting sounds, or implement a vaguely sensible way to enter musical notes (as a dodgy workaround, I remapped the names of the notes in the scale to their positions on a QWERTY keyboard and prodded out a melody from those). Fortunately, I didn’t have to let a good routine go to waste: a week or so back, I encountered Sergi Mansilla’s jsmidi project, which provides an easy way to create standard MIDI files from Javascript. However, it turns out that MIDI support in browsers has more or less stagnated – while wave audio goes from strength to strength, MIDI is still stuck in the world of OS-specific plugins – so there was a clear opportunity for some Javascript synthesiser love there. And so I’ve come up with jasmid, a JS library for reading MIDI files and playing them back through its own audio synthesis engine.

A MIDI file consists of a simple header, followed by a list of tracks – each of which is a list of timestamped events, which are usually note on/off events but could be a tempo change, change of instrument or various other things. The timestamps are actually a bit weird: you’d expect them to be given in something like microseconds, but they’re actually expressed as a number of ‘ticks’, where the MIDI file specifies a particular number of ticks per beat, and the tempo of the song is given in beats per second, which can change over time just to add further confusion. Ultimately it probably does make sense, because it means you can accurately use any rational number (within reason) as a tempo, and that’s handy for professional MIDI equipment that has to keep exact time for extended periods – it’s just an initial hurdle that you have to get over. Once you’ve got that in place, a MIDI synthesiser boils down to a set of generator functions that can emit audio waves for as long as you tell them to, and a main loop which picks events off the queue, running the generators until it’s time to process the next event.

For this first release, the generated sounds are not particularly interesting – just plain sine waves with a bit of attack/decay volume control applied – but now that we’ve got the initial framework in place it should be relatively straightforward to add more diverse sounds, and the synth engine should hopefully be flexible enough to support effects like harmonics and reverb.

Finally, as a glimpse of what’s in store for in-browser audio creation in the future, keep an eye on the Mozilla Rainbow project, which provides APIs for capturing audio and video from microphones / webcams. For the last few years I’ve been taking part in February Album Writing Month, a song writing community where collaborations over the internet play a major part – perhaps it won’t be too long until we’re doing that over a Google-Docs-style online equivalent of Audacity / GarageBand…?

It was internally nagging me all summer that I ought to release something at Sundown, but apart from some brief excitement around a brainwave I had involving three iPhones, Javascript and some cardboard (which sadly didn’t work out in practice), it didn’t really amount to much. Then Evoke happened, and inspired me to decide that, in the words of Haujobb’s invitation, I should make a demo.

Or, to be precise, a Javascript 64K intro. Unfortunately, there was only one week to Sundown by that point, and in a classic case of demoscene project management fail, I spent most of that week building an audio framework, leaving about 24 hours to write the actual demo. The end result is 9K of rather-flaky-performing code, hastily improvised plinky ambient music, and dreadful coder art (except to the extent that I’ve ripped it off from Fairlight demos).

• Fake Plastic Cubes – online version
• Fake Plastic Cubes – download (10Kb)
• Fake Plastic Cubes on Pouet

Still, even if the execution this time didn’t work out, I think it’s been a worthwhile exercise in bringing pieces together. Jacob Seidelin’s PNG compression hack (where JS code is stored in a PNG image to take advantage of the compression, then unpacked on a canvas using getImageData) has created a bit of a buzz in the JS development world, but this is the first time it’s been used in an actual demoscene production (which is surprising, given how the demoscene is the spiritual home of size-coding hacks). Ben Firshman’s JSNES has been dynamically generating audio for some time now, ardently chasing the moving target that is Mozilla’s Audio Data API (with a trusty Flash snippet as a fallback), and Mathieu ‘p01′ Henri was experimenting with softsynths long before then. Not even my own code is safe from this cherry-picking exercise of doom – the 3D routines are a mishmash of Gallions Reach / Canvastastic (for the lighting model) and Antisocial (for the full scene / movable camera handling). Finally, node.js makes a cameo appearance, because having an actual web server on hand makes development go a lot smoother.

Put them all together and you have the ingredients for a delicious 64K Intro cake. This time it came out a bit half-baked, but I’m passing on the recipe in the hope that someone else can make it work:

• Fake Plastic Cubes source code / framework, on Github

So far it only implements a subset of the possible sample effects, and it demands a very fast Javascript engine – luckily all the new breed of browsers are pretty competitive at that now. Even so, unless your CPU is an absolute behemoth, it’ll probably struggle to keep up – the audio output is fixed at 44100Hz, and that’s rather a lot of numbers for Javascript to crunch, especially when the MOD file gets up to 16 or more channels. Which, amusingly enough, is exactly the situation we had back when we were using Gravis Ultrasounds on 386es. Hurrah for progress!

Update 2010-06-08: Oops. In the process of testing how Safari 5 shapes up, I discovered a rather silly oversight: the audio buffering routine was set up to never use more than 10% of CPU. Now that I’ve fixed it, it turns out that Chrome and Safari (at least) have no trouble at all playing Jugi’s Dope theme in its 28-channel glory. (However, taking the brakes off the buffering does mean that we can’t reliably pause the audio any more. A small price to pay, I think you’ll agree.)

• Play online
• Download JSSpeccy v20091121 (680Kb)
• JSSpeccy subversion repository

A new version of JSSpeccy is out. It doesn’t run at full speed on an iPhone either (although it positively speeds along on recent versions of Safari on real computers), but it does boast the following changes:

• GPL v3 licenced, with prominent notices to make it clear that playing silly buggers like the above will not be tolerated (even if they do include source…)
• A bit of speed optimisation (about 15% faster maybe)
• A pimped-up user interface with shiny icons
• And most relevantly, entirely controllable via iPhone / iPod Touch touchscreen. In principle. (If you’re expecting an immersive gaming experience, you’ll be disappointed.)

So there you go – probably the best Spectrum emulator for the iPhone ever. And it’s free.

• Play online
• Download, inc source (674Kb)
• JSSpeccy Subversion repository

• Readme file
• Run JSSpeccy online (includes 10 classic games!)
• Download JSSpeccy (644Kb)
• JSSpeccy Subversion repository

Writing this wasn’t actually such a big deal – the Z80 core was ported from the one in Fuse, with the Perl-and-C-preprocessor-munging trickery still intact, and Javascript is syntactically close enough to C that that wasn’t a mammoth task. (I got 90% of it done on the train journey back from International Vodka Party alongside recording silly songs about tube stations.) The one fiddly bit was working around the places where the Fuse code used low-level C constructs to its advantage: using unions to chop and change between individual registers and 16-bit register pairs, and relying on limited-size C data types (often in pretty subtle ways) to truncate 8-bit and 16-bit values at the right time, whereas Javascript only has integers. (Actually, the really time-consuming bit was debugging it all… luckily, Fuse has a rather excellent test suite too.)

The rest is just creative abuse of the <canvas> element, as usual… it’ll take advantage of the putImageData interface to do the pixel pushing if available (on my machine Firefox has it, Safari doesn’t) and fall back on drawing 1×1 pixel rectangles otherwise. This time I’ve thrown in Google’s ExplorerCanvas as a nod to those poor unfortunates still stuck with Internet Explorer. Incidentally, I’d be curious to know how it rates on Google Chrome (I don’t have an XP/Vista box to test on) – if the hype is true (and it implements the putImageData interface like all good up-to-date browsers should) then I’d expect it to comfortably reach 100% Spectrum speed on modest hardware.

Belorussian translation of this post provided by movavi

Visit the Antisocial microsite…

With my characteristic lack of organisation, I found myself with two weeks to go to the Sundown party, having promised a demo release, and with nothing specific in the pipeline. So, I decided to take a chance and run with an idea that had been sitting on top of my “demos to write when I have more free time than I do right now” pile for the best part of a year. I had it all planned out in my head, right down to the soundtrack: a mysterious track from an unlabelled CD I picked up at a ZX Spectrum Orchestra gig in 2005 (which turned out to be Round, from their Clive Live^3 EP). A quick bit of permission-getting later, and I was at the point of no return.

I knew it would be an ambitious job, and a bit of a leap artistically and technically from my usual stuff. I pencilled in a rough project plan in my diary. I drew up storyboards. I read up on the maths that was too nasty to contemplate on previous projects. And shockingly enough, I actually enjoyed all of the above.

As it happens, browser technology (as far as the <canvas> element goes at least) has not moved on one jot in the last two years, so I was able to dust off the Canvastastic codebase and found it still pleasantly usable and not too affected by code rot. I gave it a slightly more OpenGL-ish API (within the limitations of my “someone at the pub described it to me once” knowledge of OpenGL) and patched up the more glaring omissions (like Z-plane clipping, so that you can have polygons going behind the camera. Proper frustum clipping would have been a better idea, so that I didn’t end up with it trying to plot 20000×40000 pixel triangles and sending Windows into a stroboscopic flashing fit and having to hack up a fix after the party. Macs are fine with it…).

The demo code was only half the job though; to handle the camera paths and synchronisation and the million other details where hard-coding wouldn’t cut it this time, I joined the big league of the demoscene by building a demo tool (still all in Javascript) that probably only I can understand. (It’s included in the final release, so have a play. I dare you.) As I didn’t have any meaningful experience in 3dMAX, or Flash, or Werkkzeug, or Quartz Composer, or any of the other things I really ought to have been using as reference points, it was a grand exercise in Making It Up As I Went Along. I’d almost forgotten how much fun it is to code that way, but naturally a fair few bad decisions came out of that as well. Firstly, it turns out that organising things around a timeline interface doesn’t really fit all that well; it means that every object and event is treated as independent of everything else, so things get a bit clunky when they have to share resources (such as lots of camera shots of the same scene). Looks like you can’t beat the good old boxes-and-arrows-all-over-the-place approach.

Secondly, and what probably comes as no surprise to anyone but me – Javascript is a bit horrible for building demo tools in. The demo itself, no problem – but for the user interface widgets I found myself constantly wishing for nice, sensible class-based inheritance. Yep, I know the language boffins will say that prototype-based OOP is more powerful, but I guess I’ve just had my mind addled by programming in the real world. I know of a few attempts to graft class-like behaviour onto Javascript, and that should probably be at the heart of any attempted rewrite of this. Alternatively, there’s a growing-but-not-quite-there-yet buzz around alternative scripting languages in the browser (just like there’s a buzz in the opposite direction for server-side Javascript), so perhaps it wouldn’t be out of the question to have a browser demo tool written in Ruby or Python. Or have the browser environment embedded in a ruby/python desktop app – at which point, why stop at browser demos? Could it be designed as a more general tool, where the browser is just one of many platforms that can be swapped in? It’s tempting. Onward and upward!

This started out as an experiment in Comet techniques (which allow you to actively push data out from web servers without the client having to initiate the request) which quickly ballooned in ambitiousness – I didn’t set out to write Just Another Chat Application after all. The end result is a realtime multiplayer Javascript conversion of the Spectrum wargame Chaos… or a reasonable chunk of it, at least.

Play Comet Chaos now

If you’re interested in the workings behind it, check out this video from my Oxford Geek Nights presentation to hear about how Comet is like a small child on a car journey, find out how close web developers can get to world domination, and watch a live demonstration going pear-shaped.

Other resources:

• Presentation slides
• Shooting Star, the Rails Comet library
• Shooting Star walkthrough
• Comet Chaos Subversion repository

But if you’d rather just dive in and have a play, that’s fine too… For those of you unfamiliar with Chaos, it’s a turn-based game for 2-8 players, each one playing a wizard who has to try and kill of all opponents by casting spells. Most of these come in the form of creatures which you can move across the board to attack opposing creatures and wizards. As ever, World Of Spectrum has the full instructions.

Be aware that the current release is rather incomplete and experimental: the repertoire of spells is limited to creatures, ‘disbelieve’ and lightning bolts, the combat system is virtually non-existent (everything has a 50% chance of killing everything else), killing wizards isn’t tested, and the finer points of game mechanics (flying/mountable/undead creatures, ranged combat, law vs chaos) are unimplemented. But don’t let that put you off…

An equally important omission – and possibly the major reason why it’s attracted minimal user testing and feedback right now – is that there’s no way to invite other players, other than marking your game as ‘public’ so that it appears on the homepage. (Non-public games still have guessable URLs, though.) To join up with other players you’ll need to organise things on other channels, such as IRC. Ideally it needs proper user signups, invitations, notifications and friend lists…

Gosh. I wonder if this could be one of those “social network” things all the cool kids are talking about? And that various wise people at conferences say you should never implement yourself, but piggyback on an established one instead? I suppose that means it’s time for me to swallow my pride and investigate the Facebook and/or OpenSocial APIs soon then.

It does make sense – Javascript is one of the few languages you can write graphical applications in that’s truly cross-platform – as in ‘doesn’t require Windows users to install a massive runtime and a bundle of libraries’. Well, in this case we do require them to install Firefox, but come on, who’s going to object to that?

It’s dead easy to use (painting in three shades of grey is hardly rocket science after all) and it exports directly to ZX81 .P format courtesy of a neat Basic snippet from Russell Marks. So if you’re feeling artistic, give it a go – you don’t even need to download it. (But if you do, you get the additional perk of being able to supply an overlay image to trace around.)

Update (2008-07-12): Now features a handy clickable palette, as requested by Yerzmyey – so that people with less than three buttons on their mouse can use it. You’d think that a Mac user like me would have thought of that, really…

• ZXpaintyONE – online version
• ZXpaintyONE – download (ZIP, 8Kb)

Download: Canvastastic beta_1 (20K, .tar.gz)

If you’re anything like me, you’ll probably be turning your nose up at having to download the thing just to see what the fuss is about, so I’m happy to oblige with some online demos…

• Boxes (demonstrates lighting and camera movement)
• Robot (demonstrates transforms and animation)
• and finally, the library source itself, for those wanting a spot of light bedtime reading.