Apple Patent Application for Laser Room Scanner in iDevices

Credit: Patently Apple

As reported by Patently Apple here, a recent patent application from Apple shows an iPhone with a laser that you can sweep around a room to make measurements.

In on illustration, a small laser beam is shown coming out of the Lightning jack on an iPhone. The user arms an app; the laser comes on; the user points to where they want to start measuring; pushes and holds a button; sweeps the laser to the end point; and releases the button. During the actions, the iPhone is continuously tracking and recording where it is in space, and the distance to the laser dot on the wall (or what ever object the beam is pointing at). Those two data sets are crunched, and the iPhone returns the distance between to start and finish points.

Applications in construction are obvious, but the basic need to measure distance is universal and could be used in everyday life in myriad ways. Finding out exactly whether a new piece of furniture might fit, or measuring for a carpet with your iPhone would be pretty handy.

It's important to note that the iPhone in the example is actually plotting the laser dot in three-dimensional space. This invention is essentially a hand operated 3-D scanner (which they're careful not to mention in the patent application).

With this invention—and, one might suspect, adding a camera pointing in the same direction with image processing software—one might be able to accurately map out a room in your house...furniture, drapes and all. That would take a lot of smarts, but phones are getting smarter all the time.

Here's where the audio geek kicks in.
So, if you can map a room with good accuracy, you should be able to make a decent acoustic model of the room. The iPhone might even be able to take a guess at acoustic absorption from brightness and scatter of the laser spot coupled with knowing something was a sofa or wall. Heck, if we go that far we might as well "ping" the room with the iPhone speaker and listen to reverberance as another important variable. The point is, this device makes it possibly for your phone to make a map of the room you're in, where you are in it, and create a synthetic acoustic space. Couple that with the inevitable head tracking and sound spacialization that is coming in headphones, and I see some pretty interesting benefits for home theater on headphones.

Headphone virtual surround systems for movies have two acoustic components: placing the surround speakers around you in space using synthesized HRTFs (head related transfer functions); and synthesizing wall reflection and reverberant cues from a room of a particular dimension. These system would often have the ability to vary the size of the room by adjusting the cues.

In my experience, some of these systems do a good job of creating a synthetic room acoustic, but I also usually feel a sense of cognitive dissonance between the acoustics of the artificial room and the room I'm actually in. Certainly, I can become immersed in the movie soundtrack and the actual room acoustics don't mean much, but, at times, I am quite aware of the difference.

With an iPhone that I could wave around my room a bit and then recreate the acoustic space that I'm actually in, and with headphones that did a good of virtualizing sounds in that space and provide the acoustic changes from head tracking cues, I bet the suspension of disbelief would be irresistible. Movies and gaming might become remarkably more real.

I'm thinking this invention might be one of the steps toward unbelievably believable virtual sound in headphones...well, for Apple and Beats anyway.

COMMENTS
zobel's picture

Does synthesizing apparent sound source positions via headphones work the same way as it does through loudspeakers? I am amazed at what can be done through two channels with just two loudspeakers to localize phantom sounds in space, and I have heard from gamers that sound localization on some games through headphones is incredibly life like. I haven't experienced it myself, but understand that these games are able to locate sound sources so well that players can even tell if the sound is coming from behind them through headphones! I wonder how that is done. I've never been able to do much more than get a sound stage spread between my ears with headphones, even with experiments done with binaural recordings attempting to show vertical placement, and behind the head placement. I find a sense of depth discernible with these recordings, but not very realistic or powerful. What information is needed to be synthesized and added to the recording to trick the ear into placing sounds in the acoustic space that obviously doesn't exist through headphones? Tyll, can you shed some light on this please?
Thanks

Tyll Hertsens's picture
When a sound approaches your head and enters the ears it goes through a transformation due to the size and shape of the torso, head, and ears. At every angle the transform is somewhat different. In sum, these transforms are called the head related transfer function (HRTF). On a detailed level, each person's HRTF is unique---like a fingerprint---due to the differences in body and ear shape. But, there are general trends and a set of generalized HRTFs is possible.

Using digital signal processing (DSP) techniques, it is possible to modify the audio signal with HRTF data. For example, if you take the left rear surround channel and add the generalized HRTF using the angle of approach to the head of a left rear surround speaker, you can, sort of, get the impression of sound coming from behind and to the left.

The problem is the most important way your brain tells where a sound is coming from is by slight movements of the head and listening to the changes. In other words, it's the changes to the HRTF with head movement that is most important for sound localization, not the static HRTF itself. This is why adding a head tracker that monitors head position and then changes the HRTF with head movement that will be most effective in getting the sound outside your head. Without head tracking, out-of-head localization is quite poor.

I think head tracking will be the most important thing to gaining convincing out-of-head virtual sonic imaging, but I think matching the synthesized acoustic space to the acoustic space your eyes are seeing around you will also provide an element that adds significantly to the experience.

zobel's picture

Man, that must be some amazing program! Have they been employing it with virtual reality gaming I wonder? I can sure understand why modeling the actual room the listener is in would most closely represent the sonic space you see. I bet if it could be done well enough it would be scary close to having the sound stage holographically portrayed in your room, but wouldn't the recording itself need additional spatial information? Maybe it wouldn't be too hard to represent the sound stage as though it were being played through a pair of speakers in that room. Even that would be really cool!

Hi-Reality's picture

Tyll,

First, a near-future prediction: I think the most realistic audio (re)-producing systems will be based on advances in adaptive HRTF/head-tracking along with a new generation of headphones and wearables (new kinds of tactile transducers). We need more precise and efficient measurement methods, and faster and smaller DSPs and hardware components. Sort of an iDeviceification (miniaturization and advancement) of Smyth Realiser for a growing number of VR & AR applications.

I thought about 'the recreation of the acoustic space you are in' that you wrote about. Here are my thoughts. Let's look at two possible use cases for audio (re)-production with playback environment and visual system in mind:

  • Immersive systems (e.g. Virtual Reality, cinema, home theater, similar):
    This is a somewhat controlled environment. The ultimate goal is to, as accurately as possible, re-produce the sound that the designer designed and experienced. In this scenario, you probably don't want to alter the acoustic/spatial properties to match the room you are in (apart from that you would want to listen through your own HRTFs measured at the designer's studio or elsewhere). The audio reproduction system has also to match/accompany the content that the visual system reproduces. An example is the hybrid (channel/object) Auro 3D where the sound & image interplay on the screen (depth, horizontal, vertical) is the key to the audio/visual experience.

  • Augmented Reality systems (and beyond the basic Google Glass-type AR; more of Magic Leap*):
    Visually, this should actually be more immersive than the above cases; but it's different. The visual production machine creates and overlays 2D/3D holographic objects in the objective reality. The audio producing system's primary function is to generate the sound related to the visual objects in the 3D space. In this case, the acoustic of the room/space you are in is important. With faster DSPs and algorithms we will probably see real-time space measurement (on the fly). This is a non-controlled environment. Therefor, I think much better noise cancellation is needed for those cases where the natural sounds in the space/room need to be dampened for maximizing the audibility of the intended 3D sound (e.g. when delivering the voice of a museum guide or a sound clip playback in an echoey and noisy museum hall).
  • Looking forward to your feedback.
    Regards, Babak
    Founder, The Hi-Reality Project
    www.hi-reality.org

    * Google invests $542 million in mysterious augmented reality firm Magic Leap:
    http://www.theverge.com/2014/10/21/7026889/magic-leap-google-leads-542-m...

    zobel's picture

    The virtual surround I was hearing was via two channel Dolby Virtual Speaker. I didn't know about the Dolby Headphone which somehow uses this technology. Have you heard this? What do you think of it?

    Here is a block diagram of the circuits;
    http://www.njr.com/semicon/PDF/NJU26220_E.pdf

    and their site;
    http://www.dolby.com/us/en/technologies/Dolby-Headphone.html

    Thanks!

    Three Toes of Fury's picture

    On behalf of the planet XYLPLIXL we wish to thank you.
    For years we have been plotting the take over and recolonization of Earth. The one item holding us back was specific population data and geographic info. To this end we placed several humanoid robots posing as engineers and CEOs to your high end electronic suppliers. These robots are responsible for the creation and proliferation of your smart phones and tablets. The data collected from these devices is invaluable in our understanding of your strengths and weaknesses. The last piece of data we require is personal surrounding topography. This iphone laser add on will provide that nicely. Why would we divulge this now? Because we realize that like a cat chasing a laser pointer, you will flock to this shiny new toy regardless of its end game. We look forward to taking over your planet soon.

    Best Regards,

    The XYLPLIXL Armada.

    PS: Upon careful reconsideration we have decided to not invade. Given your peoples fascination with the Kardashians and processed salt/sugar "foods", we figured y'all are pretty much heading towards self imposed extinction shortly. We'll just swing by in a few years when your gone.

    Jim Tavegia's picture

    I can see more sleepless nights coming from your post. What I have known all along that my ears ARE the problem. ;>)

    drblank's picture

    it looks pretty much like a digital tape measure to be used by Home Improvement DIYers', Architects, interior designers, space planners, etc.etc. I highly doubt if Apple does in fact release a new product with a digital laser that they will be thinking in terms of better audio.

    If you look at Apple's history of patent submissions, they don't always release what is described in the patent because some of them don't prove to be market worthy for one reason or another, but they submit the patent to protect the idea/design should it come to fruition.

    zobel's picture

    Knowing the size of a space tells you only a limited amount about the acoustics of that space. For example: What are the room treatments such as carpet, drapes, ceiling type, etc.? What adjoining spaces are there and what is their size and acoustic influence? And where is the part of the patent that addresses acoustic space?
    When has Apple done anything to improve on audio technology? Wouldn't a company that does gaming be more interested in virtual sound? Maybe so, but I also think Apple is likely to patent processes like this for the sake of the patent and it's possible future value. Nothing too wrong with that I guess, but I would much rather see someone patent something they want to produce.

    drblank's picture

    When has Apple done anything to improve on audio technology?

    The answer would be why is it Apple's responsibility to improve audio technology? They aren't an audio equipment mfg. They did help with the AAC MP4 which IS an improvement over just regular MP3. They did this to improve the sound quality for compressed audio so that it sounds as good a a higher bit rate and get as close as they can to CD quality, but in a compressed format for space saving reasons. the iPod was pretty much the replacement for the Walkman CD player/cassette player, so sound quality is just for the masses and not the audiophile. The problem is that people (the masses) don't really care as much about high end audio, so Apple has really no reason to cater to the audiophile with regards to these devices, that's why others can address this other much smaller market. They were more concerned with giving people an affordable way to not steal MP3's over the internet, and it was never really meant to be a big money maker, nor were they trying to cater to the high end audiophile crowd. In order to at least do something to improve the sound of their AAC files, they've been asking for 24 Bit files from the record labels so they can have the "Mastered for ITunes' versions which in my opinion are pretty darn close to a physical CD version, as long as it was mastered properly but they don't have many labels providing enough content for this effort, which isn't Apple's fault. As far as headphones, same rules, they are catering to the masses that buy iPods/mobile devices where sound quality is more to address the needs/budgets of the masses not the audiophile.

    Other than that, Apple is a computer, OS, s/w app and internet services (app/content stores), they aren't an audio equipment mfg per se, or an audio acoustic engineering company. But to incorporate a digital tape measure for those that want a built in tape measure functionality, which I think a LOT of iPhone/iPad users definitely would want. I would use that feature if it was built in, but it would be more for just general purpose tape measure functionality.

    As far as I can tell from reading the patent, it seems to be just a electronic means of a tape measure and that's about all it is. I think if you read much more into it, then I think that would be a mistake, if they wanted to incorporate audio measurements, they would have mentioned "audio", but they didn't.

    zobel's picture

    is to measure it acoustically. Someone could patent that application for a phone and it could be done with existing technology.

    drblank's picture

    Now, I'm not up to speed with how lasers would be used, but can a cheap laser actually know what the material it's being pointed at has in terms of absorption coefficients? I don't know or would think that a cheap laser pointer meant for measuring distances is sophisticated enough to know the absorption of a material. I think the type of laser they would use is probably about as good as those cheap laser pointers people use for PowerPoint Presentations.

    zobel's picture

    Is what I was hearing doing the synthesized surround from two loudspeakers. It is sort of explained here; http://www.cyberlink.com/stat/product-tutorial/enu/dv-entertainment/dvs.jsp

    There is a surround feature called Dolby Headphone that seems to use this technology, Know anything about it? found this;
    http://www.dolby.com/us/en/technologies/Dolby-Headphone.html

    Here is a block diagram of the circuitry;
    http://www.dolby.com/us/en/technologies/Dolby-Headphone.html

    Hadn't heard of this had you?

    cundare's picture

    Tyll, if the author of the Patently Apple article you cited had bothered to read the patent application, he might not have offered such over-the-top speculation. The patent describes and claims a laser distance-measuring device, something that is well-known in the art. The claims do not mention smartphones, audio, or telephony, stating only that the laser-mapping is (necessarily) performed by an "electronic device." You can speculate that this device would *have* to be a phone, but several years from now, when this patent is likely to be issued, what seems obvious now may seem silly.
    Anyway, from my read of the claims, I’d expect them to undergo significant revision in order to be deemed patentable. The only possible point of novelty in the independent claims is a device with an ability to "monitor" position and orientation of an object via a laser. (I haven't read the spec, so I'm not sure what that is intended to mean.) Yes, it is possible that nobody has combined known laser-measuring technology with a wireless interface or local storage, but even that combination could raise obviousness issues. Regardless, Apple may have to jump through hoops to overcome prior art comprised by other products in consumer space, such as Wii. Even though such a reference may use some other type of radiation to monitor position and orientation, the PTO might deem obvious to one skilled in the art an apparatus that combines known techniques of laser-measurement with known products that use other wireless methods to accomplish a similar “position and orientation” function.
    So, as is generally the case with tech analyses of patent literature, I think that "Patently Apple" is being a bit wild-eyed about this. Seriously, how hard is it to find a patent lawyer to interpret a patent filing before publishing a hot-button article that is simply off-base. The most likely outcome here -- based on how patent prosecutions usually proceed -- is that, two years from now, a greatly revised version of the published application will issue and no Apple product will ever emerge that uses the technology.
    And "Patently Apple" seems like a bunch of scumbags, anyway. The page at your link states "We ask tech sites covering our report to kindly limit the use of our graphics to one image." Um, PA copied the graphics in question right out of the patent application and then stamped them with its name. Seriously?? I wish you hadn’t given these guys free press. And next time you find some potentially interesting patent-related content, I’d be glad to give an application or issued patent a quick readthrough gratis.
    No, I’m not trying to be a troll – I just get frustrated about how tech writers with no legal background so often try to speak authoritatively about my line of work, get things completely wrong, and nobody notices. Sort of like the people who claim that Nyquist’s Theorem proves that 192kbps sampling rates cannot produce better-sounding content than 44.1. Snarl.

    X