an Audio Augmented Reality Query

February 9, 2012

I have thought occasionally about wayfinding applications of surround sound presented over headphones – using a bunch of filters (called Head Related Transfer Functions) to replicate the experience of 3D sound but using only a stereo signal over headphones. As you may know using HRTF you can get very good surround sound – it’s used by games such as Papa Sangre – but can be made more accurate and effective with customised filters for an individual person.

The RNIB see benefit in looking at audio related wayfinding apps for mobile phones with GPS and wanted to know what I could do for them so I started thinking about how sighted people find their way round a strange place.

Then I thought about hardware to make it happen. It would need earbuds or headphones with a compass module in (like in current smartphones) so that the software generating the sounds knew which was the user’s head was turned. That way a sound over to the east will stay over to the east even when the user turns a corner, the sounds remaining stationary relative to geography and location instead of relative to the listener’s ears. After a long search I can’t find headphones such as this that are available to buy although would be quite straightforward (mostly) to make I think. The compass module is about the size of a coin and costs around £50 and the outputs could, I’m sure,be made compatible with the control input in a smartphone’s headphone socket. I have ideas how that could work.

So if we made some of these headphones what could we do with it? Plugged into a smartphone with GPS they would allow audio virtual environments that remained convincing as you moved around a place, so…

Tourism: sound walks around locations – imaging walking round Liverpool and as well as looking at pictures of the Beatles at locations around the city in the 1960s you could hear the music coming from where the Cavern Club and other venues were. Strawberry Fields and Penny Lane are playing in the appropriate place – you could follow the songs to find the places. Historic soundwalks. Audio guides – “on your left you can see the houses of pariament….” (although not in Liverpool obviously ;))

Wayfinding: no need to hold up a smartphone with sat-nav (it’s like holding a sign up saying ‘mug me’) as you try to find your destination, just a pair of earbuds/lightweight headphones trailing from your pocket like thousands of other pedestrians with an iPod. Good for blind people too in replicating how sighted people find their way round a city (I know it’s over in this direction somewhere – I can hear it).

Gaming: audio enhanced augmented reality gaming. You may have seen clips like this where game environments are superimposed onto the real world, could take this several steps further with location aware binaurally presented surround sound. The potential when combined with visual augmented reality would be incredible, I wonder if these glasses from Google will work…

Music: virtual 5.1 or ambisonic music playback that goes where you go. It’s a 3D sound iPod 😀

Interactive 3D radio: walk through a dramatic scene, creating your own narrative depending on your location and route through the scene. Multi-threaded interactive dramas in sound.

All of these could of course be appropriately enhanced with commercial messages, “hungry? the Pizza Hut on your left has a two for one deal on now”.

Would love to build a prototype pair of headphones to generate ideas and implement some of the ideas above but it seems there are patents all over it. Has anyone come across a commercially available solution? Would love to know…

Ah ha! That clever Russell Davies was talking about this and a SAP augmented game a couple of years ago at Playful, wonder if he ever found a suitable hardware platform?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: