How I learned the difference between AR & MR
It was a fairly calm evening last weekend while my friends and I sat in the pickup truck filling up for gas. The night air in the rural backwoods of Pennsylvania was calm, and the air humid from the July heat. Out of the corner of our eye we saw another car pull up, but instead of looking to fill up their tank, two older gentlemen jumped out while one of them loudly proclaimed:
“I’ve only got three Pokeballs!”
Did we slip into the Twilight Zone? What drugs were these guys on?
But then the obvious sunk in and the lightbulb went off.
These guys were driving around town playing the latest Augmented Reality game from Nintendo and Niantic Labs, aptly named Pokémon Go.
In what seems like a total invasion of augmented reality gaming, Nintendo and Niantic have unleashed what can very well be considered a “killer app”. Within 24 hours of its release, the market share for Nintendo skyrocketed by 9 billion dollars, and has in the last week alone been deemed the most traded stock in Japanese history.
Thanks to the fad-tastic launch of Pokemon GO - more popular than porn - Nintendo stock has exploded over 93% in the last 7 days (the most ever) to 6 year highs. But the Pokemania was really in the trading volume where 476 billion yen changed hands for the highest daily turnover on the Tokyo Stock Exchange this century... – zerohedge.com
Holy crap, that’s amazing…
But of course, there is now the apparent discussion/debate about whether or not Pokémon Go is actually Augmented Reality.
Whether the discussion is from VentureBeat or Scientific American, both will tell you that Pokémon go is not Augmented Reality because it breaks some rule about immersion, and so it’s not true Augmented Reality.
Well, you already know where this is going if you’ve been a long-time reader of this blog, so let’s get right to the point: Pokémon Go is Augmented Reality.
VentureBeat doesn’t seem to understand the difference when it comes to this topic, and so their entire write up about it is describing what is known as Mixed Reality and not Augmented Reality. So if you know anything about the terminology, you’ll immediately understand that while they have their heart in the right place, they just aren’t competent enough to be calling that shot. You’d immediately realize the author of that write-up is describing Mixed Reality and not Augmented Reality.
The promotional videos for Pokémon Go essentially are confusing the terms as well – in that the above video shows a Mixed Reality view and not the bare bones Augmented Reality of the actual game. The video above is showing a digital Pikachu that is immersed in the real world and is real world aware and can interact with the real world. Which is actually doable, but you’d need better hardware and software to pull it off. SLAM mapping of the real world environment, a Depth Sensing Camera, better GPS, and so on. In this notion, yes… immersion is a criteria for sure.
This is a far cry from the reality of the situation in which a 3D render of the Pokémon is simply superimposed onto your video feed.
Ok, so let’s be straight up for a minute.
Augmented Reality isn’t really one defined thing aside from “Superimposing digital information onto the real world”. It is mostly an umbrella terminology, and often used interchangeably with Mixed Reality. Augmented Reality is a spectrum and not a singular point on the map.
From Heads Up Displays (Google Glass, Air Force Pilot Helmets) to something in the realm of HoloLens and Magic Leap (and beyond). You can pretty much just have an arrow on screen pointing directions like your GPS Map, and if it’s overlaid on the real world (via your video camera) it can be AR too… albeit very rudimentary.
So today, I’m just going to take the Mixed Reality aspect and use that to define advanced augmented reality while saying AR itself can (and often is) just rudimentary and low awareness overlay. When we use the term Augmented Reality, what most of us forget is to ask “What degree in the spectrum of AR is this experience?”
So Pokémon Go is Augmented Reality… but
Oh, you should have known this one was coming.
Pokémon Go is Augmented Reality but it is probably the lowest common denominator to barely qualify. As in, it’s nothing special and a highly stripped down experience in the AR realm.
Essentially, it is AR by technicality alone, the bare minimum requirement to qualify.
It uses the real world location and your camera to “augment” your view with an overlay of information.
I think that qualifies as augmenting your reality. Even if it’s half assed, it’s still doing it in the most basic fashion. It’s no worse than calling Google Glass augmented reality even if the information isn’t spatially aware or embedded in the real world.
There’s no requirement for Augmented Reality to be “immersive”. That is far too nebulous of a notion to take seriously for an umbrella term. Saying it’s not immersive is missing the point entirely. If you wanted immersive AR, then we’re talking about Mixed Reality (AR that is real world aware and can interact with it).
Let’s not assign the “immersion” test to AR as a whole, but only if we’re talking about Mixed Reality (other end of the AR spectrum closer to Reality). In which case, then it would be a requirement to qualify not as AR, but to qualify as Mixed Reality.
Pokémon Go is mainly designed around maps, letting players find and catch in the real world characters from Nintendo's Pokémon franchise. When they find a Pokémon, players can enter an “augmented reality” mode that lets them see their target on their phone screens, superimposed over the real world.
There’s the key term: “superimposed”.
Early AR systems do exactly this. They simply superimpose the digital content over a video feed (or the real world via see through displays), but the digital objects have absolutely no flipping clue about the real world around them other than the GPS location and access to the Compass to figure out where the player is facing. There are a ton of games on Google Play that already use this form of AR – it’s been around for a very long time. The only reason you’re paying attention now is because Nintendo made a Pokémon game using it.
Technically Speaking
The problem with this debate overall is that in the bigger picture few people actually understand the terminology involved or that Augmented Reality is also used interchangeably with Mixed Reality as the umbrella term constituting the entire spectrum of augmented virtuality.
It is a nebulous sort of terminology and nobody has really sat down and figured out what they really mean, or I should say that few people have bothered to pay attention to those who have.
So let’s do a quick rundown to clarify:
Virtual Reality: an artificial environment which is experienced through sensory stimuli (as sights and sounds) provided by a computer and in which one's actions partially or wholly determine what happens in the environment.
In simple terms: The entire environment is virtual.
Augmented Virtuality: an artificial environment which is experienced through sensory stimuli (as sights and sounds) provided by a computer and in which one's actions partially or wholly determine what happens in the environment, whereby media or information from the real world is included. The real world information is virtual world aware and can interact with it accordingly as if it were the native environment.
In simple terms: The environment is mostly virtual with a dash of real world thrown into it. For instance, maybe a video feed or something.
Augmented Reality: a real world environment whereby digital information, in two dimensions or three, is superimposed onto the real world through a display. Also used as an umbrella term describing the entire spectrum from Heads Up Display to Complete Mixed Reality.
Mixed Reality: a real world environment whereby digital information, both two and three dimensional are immersed into the real world through a display and robust access to computer vision and sensory data. It is real world aware and can interact with it accordingly as if it were part of it. Also known as “Hyper Reality”. It is the juxtaposition to Augmented Virtuality.
Reality: You pretty much know what that is, unless you’re an existentialist. In which case this blog is just another illusion.
Let’s Continue…
So, I get into this breakdown simply because we really need to think about it before we have this discussion. There is an Augmented Virtuality on the scale which says that the real world is a minor component in the virtual world. Essentially it means a sort of percentage mixture in combination with the degree in which the augmented elements are immersed in that baseline space.
With Augmented Virtuality, we’re saying that the Real World is the artificial component in the Virtual World predominance. Likewise, we have the similar opposite in Mixed Reality whereby the virtual world is the artificial inclusion to the real world baseline.
But then that leaves Augmented Reality down the center, which is to say that Mixed Reality is a more in-depth version of the underlying concept of Augmented Reality. Notice when I said earlier that in AR, digital items are superimposed on the real world?
The underlying meaning of that statement was that by being “superimposed” on the camera view, basic augmented reality simply has limited or no concept of the real world it is inhabiting. That Pikachu isn’t going to be hiding in the bushes and jump out because that digital object has absolutely no idea there is even a bush there to begin with.
Basic Terminology Map
| AR Spectrum |
<— VR --- Augmented Virtuality --- AR--- Mixed Reality --- Reality –>
Augmented Reality as a spectrum goes from:
1. Heads Up Display
all the way to
2. Pikachu just ran under my couch and Team Rocket just blew a hole in my living room wall to battle me (Mixed Reality)
And before you say that last one doesn’t even seem possible, it’s been done already as a demonstration by (of course) Microsoft and their HoloLens:
Deliberate Ambiguities?
Maybe Nintendo just forgot to add the small print saying “Not actual in-game footage” to their promotional videos? Who knows… It’s a hype word that covers a large spectrum, much like using Virtual Reality is at the moment.
What the basic augmented world knows is simply the GPS location and your Compass reading, maybe some other MEMS data (Altimeter, Barometer, etc) which are more or less built into every smart-phone today. That’s pretty much it. Pikachu in “AR” is the bare minimum required to be augmented reality. Again, no worse than calling Google Glass augmented reality.
It works within the limited confines of your smart phone which isn’t using things like a Project Tango enabled phone (SLAM Mapping, RGB-D Depth Camera, etc) so Pikachu isn’t going to run under the couch or jump on the table anytime soon.
Now, Mixed Reality on the other hand, is the type of Augmented Reality that has full out immersion and likely has the sensors and equipment to actually have a full three dimensional understanding of the real world the digital world is inhabiting. Mixed Reality is when the Virtual World and Real World co-exist as one entity and the digital world can interact and act upon the real world (and vice versa).
By this understanding, then, an Air Force pilot using a Heads Up Display (again, similar to Google Glass) is also using a rudimentary form of Augmented Reality.
The underlying notion at the bare minimum level is “Does this technology superimpose information onto the real world?”
There’s no criteria in there about “is it immersive?”
For that criteria, we’re talking about Mixed Reality and not just Augmented Reality as the overarching catch-all. And since it’s a spectrum we’re not talking definitive but to what degree of immersion?
Pretty much none –> all the way to “I can’t tell the difference”
At the end of the day, it’s not about whether it is “Augmented Reality” but instead we should be asking “What is the quality of the experience within the spectrum?”
You can call both a Bugatti Veyron and a Power Wheels toy a “car”, and I think that’s what we really need to better understand.
Pokémon Go is closer to the Power Wheels toy end of the AR Spectrum while the commercials promoting the game are at the Bugatti Veyron (Mixed Reality) end of the spectrum.
Alright kids… Class Dismissed!
Every time I had to type Pokémon in this post, I had to go find that damned accent for the e. Feel free to tip me L$ in Second Life (Aeonix Aeon) to help pay for the therapy from this trauma.
0 Comments:
Post a Comment