Pokémon Go or;

How I learned the difference between AR & MR

 

 

 

It was a fairly calm evening last weekend while my friends and I sat in the pickup truck filling up for gas. The night air in the rural backwoods of Pennsylvania was calm, and the air humid from the July heat. Out of the corner of our eye we saw another car pull up, but instead of looking to fill up their tank, two older gentlemen jumped out while one of them loudly proclaimed:

 

“I’ve only got three Pokeballs!”

 

Did we slip into the Twilight Zone? What drugs were these guys on?

 

But then the obvious sunk in and the lightbulb went off.

 

These guys were driving around town playing the latest Augmented Reality game from Nintendo and Niantic Labs, aptly named Pokémon Go.

 

In what seems like a total invasion of augmented reality gaming, Nintendo and Niantic have unleashed what can very well be considered a “killer app”. Within 24 hours of its release, the market share for Nintendo skyrocketed by 9 billion dollars, and has in the last week alone been deemed the most traded stock in Japanese history.

 

Thanks to the fad-tastic launch of Pokemon GO - more popular than porn - Nintendo stock has exploded over 93% in the last 7 days (the most ever) to 6 year highs. But the Pokemania was really in the trading volume where 476 billion yen changed hands for the highest daily turnover on the Tokyo Stock Exchange this century... – zerohedge.com

 

 

Holy crap, that’s amazing…

 

But of course, there is now the apparent discussion/debate about whether or not Pokémon Go is actually Augmented Reality.

 

Whether the discussion is from VentureBeat or Scientific American, both will tell you that Pokémon go is not Augmented Reality because it breaks some rule about immersion, and so it’s not true Augmented Reality.

 

Well, you already know where this is going if you’ve been a long-time reader of this blog, so let’s get right to the point: Pokémon Go is Augmented Reality.

 

VentureBeat doesn’t seem to understand the difference when it comes to this topic, and so their entire write up about it is describing what is known as Mixed Reality and not Augmented Reality. So if you know anything about the terminology, you’ll immediately understand that while they have their heart in the right place, they just aren’t competent enough to be calling that shot. You’d immediately realize the author of that write-up is describing Mixed Reality and not Augmented Reality.

 

---

 

 

This Pokemon Go commercial is showing Mixed Reality

 

 

---

 

The promotional videos for Pokémon Go essentially are confusing the terms as well – in that the above video shows a Mixed Reality view and not the bare bones Augmented Reality of the actual game. The video above is showing a digital Pikachu that is immersed in the real world and is real world aware and can interact with the real world. Which is actually doable, but you’d need better hardware and software to pull it off. SLAM mapping of the real world environment, a Depth Sensing Camera, better GPS, and so on. In this notion, yes… immersion is a criteria for sure.

 

This is a far cry from the reality of the situation in which a 3D render of the Pokémon is simply superimposed onto your video feed.

 

Ok, so let’s be straight up for a minute.

 

Augmented Reality isn’t really one defined thing aside from “Superimposing digital information onto the real world”. It is mostly an umbrella terminology, and often used interchangeably with Mixed Reality. Augmented Reality is a spectrum and not a singular point on the map.

 

From Heads Up Displays (Google Glass, Air Force Pilot Helmets) to something in the realm of HoloLens and Magic Leap (and beyond). You can pretty much just have an arrow on screen pointing directions like your GPS Map, and if it’s overlaid on the real world (via your video camera) it can be AR too… albeit very rudimentary.

 

So today, I’m just going to take the Mixed Reality aspect and use that to define advanced augmented reality while saying AR itself can (and often is) just rudimentary and low awareness overlay. When we use the term Augmented Reality, what most of us forget is to ask “What degree in the spectrum of AR is this experience?”

 

 

 

 

So Pokémon Go is Augmented Reality… but

 

Oh, you should have known this one was coming.

 

Pokémon Go is Augmented Reality but  it is probably the lowest common denominator to barely qualify. As in, it’s nothing special and a highly stripped down experience in the AR realm.

 

Essentially, it is AR by technicality alone, the bare minimum requirement to qualify.

 

It uses the real world location and your camera to “augment” your view with an overlay of information.

 

I think that qualifies as augmenting your reality. Even if it’s half assed, it’s still doing it in the most basic fashion.  It’s no worse than calling Google Glass augmented reality even if the information isn’t spatially aware or embedded in the real world.

 

There’s no requirement for Augmented Reality to be “immersive”. That is far too nebulous of a notion to take seriously for an umbrella term. Saying it’s not immersive is missing the point entirely. If you wanted immersive AR, then we’re talking about Mixed Reality (AR that is real world aware and can interact with it).

 

Let’s not assign the “immersion” test to AR as a whole, but only if we’re talking about Mixed Reality (other end of the AR spectrum closer to Reality). In which case, then it would be a requirement to qualify not as AR, but to qualify as Mixed Reality.

 

Pokémon Go is mainly designed around maps, letting players find and catch in the real world characters from Nintendo's Pokémon  franchise. When they find a Pokémon, players can enter an “augmented reality” mode that lets them see their target on their phone screens, superimposed over the real world.

 

There’s the key term: “superimposed”.

 

Early AR systems do exactly this. They simply superimpose the digital content over a video feed (or the real world via see through displays), but the digital objects have absolutely no flipping clue about the real world around them other than the GPS location and access to the Compass to figure out where the player is facing. There are a ton of games on Google Play that already use this form of AR – it’s been around for a very long time. The only reason you’re paying attention now is because Nintendo made a Pokémon game using it.

 

---

 

Technically Speaking

 

The problem with this debate overall is that in the bigger picture few people actually understand the terminology involved or that Augmented Reality is also used interchangeably with Mixed Reality as the umbrella term constituting the entire spectrum of augmented virtuality.

 

It is a nebulous sort of terminology and nobody has really sat down and figured out what they really mean, or I should say that few people have bothered to pay attention to those who have.

 

So let’s do a quick rundown to clarify:

 

Virtual Reality: an artificial environment which is experienced through sensory stimuli (as sights and sounds) provided by a computer and in which one's actions partially or wholly determine what happens in the environment.

 

In simple terms: The entire environment is virtual.

 

Augmented Virtuality: an artificial environment which is experienced through sensory stimuli (as sights and sounds) provided by a computer and in which one's actions partially or wholly determine what happens in the environment, whereby media or information from the real world is included. The real world information is virtual world aware and can interact with it accordingly as if it were the native environment.

 

In simple terms: The environment is mostly virtual with a dash of real world thrown into it. For instance, maybe a video feed or something.

 

Augmented Reality: a real world environment whereby digital information, in two dimensions or three, is superimposed onto the real world through a display. Also used as an umbrella term describing the entire spectrum from Heads Up Display to Complete Mixed Reality.

 

Mixed Reality: a real world environment whereby digital information, both two and three dimensional are immersed into the real world through a display and robust access to computer vision and sensory data. It is real world aware and can interact with it accordingly as if it were part of it. Also known as “Hyper Reality”. It is the juxtaposition to Augmented Virtuality.

 

 

This is what Mixed Reality Looks Like

 

Reality: You pretty much know what that is, unless you’re an existentialist. In which case this blog is just another illusion.

 

---

 

Let’s Continue…

 

So, I get into this breakdown simply because we really need to think about it before we have this discussion. There is an Augmented Virtuality on the scale which says that the real world is a minor component in the virtual world. Essentially it means a sort of percentage mixture in combination with the degree in which the augmented elements are immersed in that baseline space.

 

With Augmented Virtuality, we’re saying that the Real World is the artificial component in the Virtual World predominance. Likewise, we have the similar opposite in Mixed Reality whereby the virtual world is the artificial inclusion to the real world baseline.

 

But then that leaves Augmented Reality down the center, which is to say that Mixed Reality is a more in-depth version of the underlying concept of Augmented Reality. Notice when I said earlier that in AR, digital items are superimposed on the real world?

 

The underlying meaning of that statement was that by being “superimposed” on the camera view, basic augmented reality simply has limited or no concept of the real world it is inhabiting. That Pikachu isn’t going to be hiding in the bushes and jump out because that digital object has absolutely no idea there is even a bush there to begin with.

 

 

Basic Terminology Map

 

 

                                                         |       AR Spectrum        |

<— VR --- Augmented Virtuality --- AR--- Mixed Reality --- Reality –>

 

 

 

Augmented Reality as a spectrum goes from:

 

1. Heads Up Display

 

all the way to

 

2. Pikachu just ran under my couch and Team Rocket just blew a hole in my living room wall to battle me (Mixed Reality)

 

And before you say that last one doesn’t even seem possible, it’s been done already as a demonstration by (of course) Microsoft and their HoloLens:

 

 

The other end of the AR Spectrum: Mixed Reality

 

 

---

 

Deliberate Ambiguities?

 

 

---

 

Maybe Nintendo just forgot to add the small print saying “Not actual in-game footage” to their promotional videos? Who knows… It’s a hype word that covers a large spectrum, much like using Virtual Reality is at the moment.

 

What the basic augmented world knows is simply the GPS location and your Compass reading, maybe some other MEMS data (Altimeter, Barometer, etc) which are more or less built into every smart-phone today. That’s pretty much it. Pikachu in “AR” is the bare minimum required to be augmented reality. Again, no worse than calling Google Glass augmented reality.

 

It works within the limited confines of your smart phone which isn’t using things like a Project Tango enabled phone (SLAM Mapping, RGB-D Depth Camera, etc) so Pikachu isn’t going to run under the couch or jump on the table anytime soon.

 

Now, Mixed Reality on the other hand, is the type of Augmented Reality that has full out immersion and likely has the sensors and equipment to actually have a full three dimensional understanding of the real world the digital world is inhabiting. Mixed Reality is when the Virtual World and Real World co-exist as one entity and the digital world can interact and act upon the real world (and vice versa).

 

By this understanding, then, an Air Force pilot using a Heads Up Display (again, similar to Google Glass) is also using a rudimentary form of Augmented Reality.

 

 

 

 

The underlying notion at the bare minimum level is “Does this technology superimpose information onto the real world?”

 

There’s no criteria in there about “is it immersive?”

 

For that criteria, we’re talking about Mixed Reality and not just Augmented Reality as the overarching catch-all. And since it’s a spectrum we’re not talking definitive but to what degree of immersion?

 

Pretty much none –> all the way to “I can’t tell the difference”

 

At the end of the day, it’s not about whether it is “Augmented Reality” but instead we should be asking “What is the quality of the experience within the spectrum?”

 

You can call both a Bugatti Veyron and a Power Wheels toy a “car”, and I think that’s what we really need to better understand.

 

Pokémon Go is closer to the Power Wheels toy end of the AR Spectrum while the commercials promoting the game are at the Bugatti Veyron (Mixed Reality) end of the spectrum.

 

Alright kids… Class Dismissed!

 

---

 

Every time I had to type Pokémon in this post, I had to go find that damned accent for the e. Feel free to tip me L$ in Second Life (Aeonix Aeon) to help pay for the therapy from this trauma.

 

---

 

 

 

 

Point & Click Adventure

The paradox of not moving in virtual reality

 

 

 

 

I’m not completely sold on the hype of Virtual Reality, as much as I really want to be. I believe it is a wonderful resurgence of technology and absolutely has a place in various circumstances, but it’s not a cure-all magic entertainment bullet like the industry keeps trumping it up to be.

 

For one, the motion sickness problem is still there.

 

I know, a lot of folks who have tried the Oculus and VIVE will say “But I didn’t get sick!” and you’re right. Absolutely, 100% unequivocally correct.

 

But therein is the problem. See, the reason you hardly get sick in VR with these headsets has less to do with the technology (though that does play a part), and more about the fact that instead of solving the problem, we’ve simply made it a point to avoid it.

 

Avoiding the problem isn’t the same thing as solving it. The problem is still there, but really we’re in this weird logic like owning a Ferrari and saying don’t actually drive it.

 

In the Oculus Best Practices document, they simply say that allowing the user to move around on their own should be avoided, and instead we should minimize the flow of movement. Of course, there’s the obligatory part about not forcing their camera view out of their control and such, but let’s focus on the main aspect here for the sake of this post.

 

The main issue is that there is a disconnect between what the eyes perceive as motion and what the inner ear is telling the brain about said lack of motion.

 

No amount of fancy HMD is going to fix that. No matter how well you can trick the visuals, you’ll always have the disconnect unless the user:

 

• A. Walks on an omnidirectional treadmill
• B. Isn’t allowed to freely move around

 

The industry at this juncture has chosen option B, which relegates a majority of VR experiences to a glorified Point and Click Adventure game from the 90s. A good example is Myst. It was roughly the same mechanic for movement… you pointed and clicked, then your view would fade to the new one scene by scene.

 

Of course, VR is more complex than that, but the analogy is the same. They’re not going to let you freely roam around anytime soon because you’d get sick almost 100% of the time the moment they do.

 

So we have the point and click teleport movement standard in VR today which neatly avoids the movement issue altogether without actually solving it.

 

 

 

Not really a solution so much as it is a blatant admission that the motion sickness problem wasn’t actually solved.

 

After all, Oculus games come with a “Comfort Rating” from Comfortable to Intense, with Intense being a high likelihood the game will make you lose your lunch (high motion, etc).

 

As eloquently explained on PC Invasion:

 

The comfort rating is a warning to people who may suffer from nausea, a physical response to the effects of VR that makes you sweat and then feel sick and disoriented. It’s extremely unpleasant as I have found out on many occasions on quite a few different VR systems in the past decade or so.

 

Oculus are covering their backs making sure users know there’s a real risk of a game being unplayable. There’s been no mention of refunds or a test period so owners should be aware and think hard before spending $60 on a game from the Oculus store.

 

 

In Second Life, you’d simply move from point A to B in a fluid motion. Unless you are travelling long distances (and then you would teleport). But in any game that allows free-form exploration for VR, you’re going to find this point and click movement mechanic. The ones that ignore the teleport movement mechanic are the ones you will almost certainly get sick using. Which is why a lot of people who used the Oculus headset for Second Life ended getting nauseous.

 

It’s really just the discontinuity between the movement you are seeing and your inner ear saying “Hey, we’re not moving… something is wrong”

 

So in order to truly have solved the VR Movement problem, we’re going to need something like this Walk Mouse omnidirectional treadmill.

 

That’s when we can start actually moving around in VR without getting sick. Otherwise, we’re relegated to the point and click VR adventure.

 

Then, of course, there is my personal beef with referring to 360 videos as “VR” because by god it simply is not virtual reality.

 

I think, in the grand scheme of things, what will kill the VR industry quickest will be companies looking to make a quick buck by calling their technology or product a VR product just to lend credence to it.

 

We’ll then end up with all kinds of gimmicks with “VR” in the name or “intended for VR use” when in all reality it’s a lot of quick fads at best. I consider 360 video one of those gimmicks – the lowest common denominator at best. I mean, look… we had that years ago. That’s the sort of thing that Myst actually used in the 90s (QTVR) which is really just a panorama video. Now, when it started, it was just a panorama viewer (still images), but later it allowed for video as well… the Myst games series used it extensively (as I recall) and there were video portions. It was still rudimentary, but you know… so was everything in 1995.

 

Of course, Myst IV was released in 2004, but still… it’s 2016 as of this writing, so the idea here is still a valid point to make.

 

 

Notice the player looking around?

 

To see it again making a comeback is cool and all, but I wouldn’t call it virtual reality. By doing so it actually cheapens the image of VR all over again.

 

The controllers for VR aren’t even a new idea either. Hate to be the bearer of bad news on that one. There was a similar wireless controller set for VR in the 1990s as well. It used a magnetometer to track the position.

 

 

 

 

There is, of course the Magnus VR Glove system currently in development, which to me is still a rehash of the Dataglove from VPL in the 1990s (Jaron Lanier), but still I would prefer a Dataglove over the VIVE or Oculus controllers simply because it would allow me to actually use my hands naturally.

 

Just in case i wanted to pick something up or manipulate things with actual detail and not pointing giant wands at it.

 

 

Ultimately though, the real eye opener about all of the advancements in VR is that the end-game is actually not virtual reality, but instead its cousin Mixed Reality.

 

Before this goes any farther, let’s take a moment to understand that Augmented Reality and Mixed Reality are not the same thing. A lot of folks tend to use the terms as interchangeable but they really aren’t.

 

Augmented Reality is not real world aware. It does not interact with the actual world in which the digital objects “reside”. In Mixed Reality, the digital objects are real world aware and can interact with the real world.

 

It’s the difference between overlaying a 3D version of Mario on a video feed, but it has no idea about the real world – in effect the video is just an inserted flat background, while Mixed Reality would allow Mario to jump on your couch and duck under the kitchen table.

 

As much as Pokemon Go! is a big hit now, it’s actually just Augmented Reality. The 3D Pokemon in the game are just overlaid on the background video but have no real idea about the surroundings (see the picture). Which is why we have so many hilarious Pokemon Go! pictures online now (Pokemon appearing in odd places).

 

But therein is the bigger picture. I am not totally against Virtual Reality… I just think it’s very nascent in implementation and has a lot of room to actually improve to solve the bigger issues (instead of trying to ignore them).

 

We could use a lot more intuitive control interfaces for VR that wireless sticks with buttons on them. The headsets could be wireless (and should be). Resolution per eye should be 1080 or higher, not 1080 total. We absolutely need to solve the motion sickness problem once and for all, and I believe a low cost, folding (for storage) omnidirectional treadmill platform would do this job nicely.

 

But in the end… again… all of those advancements are going to be utilized by Mixed Reality systems, which will end up just blowing the VR out of the water and would come with almost none of the motion sickness problems.

 

After all, you’re actually moving around  in Mixed Reality, so there’s no treadmill needed. There are plenty of CV algorithms that can track your hands internally via built in cameras, which a Mixed Reality system would have natively to begin with. So all in all… I look forward to using Mixed Reality far more than Virtual Reality.

 

 

Who wouldn’t want to treat reality like Second Life?

 

Yeah… it would be amazing.

 

---

 

Oh, and by the way… I’m back =^-^=

 

---

 

Hiatus

Hello blogger, my old friend…

 

 

---

 

As it has become obvious, I have taken an extended hiatus from blogging on the A3D site in order to pursue other projects and clear my mind for subjects to write about going forward.

 

I would like to extend a sincere apology for the abrupt departure after the December post The Gathering: Of Scissors, Candy and Conversation but it was unavoidable.

 

If all goes well, and my chaotic schedule normalizes, I would like to return to this blog and continue writing on a regular basis, A heartfelt thanks to those who have remained subscribed in my absence is in order, without you (my readers) I would be merely writing into the void. Though I’ve always written as though this were the case, regardless.

 

Please remain patient as I reorganize priorities and put this blog back on track (hopefully) bringing to you more of my musings and ramblings about the virtual reality industry and related materials in technology. I will note, however, that this will not only be my focus in writing.

 

Sometimes I just have some random thought experiments I’d like to explore further, some science fiction and fiction I’ve been working on, as well as a lot of interesting books I’ve been reading which I’d like to write commentary, and so this will also become the norm (if it hasn’t already over the years).

 

No, the blog isn’t “dead” so to speak.

 

Yes, I will be returning “soon” to pick up where we have left off.

 

~ Will

 

---

 

 

The reports of my death have been exaggerated…

 

 

---