Give me something to believe in: Is #Euclideon Unlimited Detail a hoax?
Every revolutionary idea seems to evoke three stages of reaction. They may be summed up by the phrases:
- It's completely impossible.
- It's possible, but it's not worth doing.
- I said it was a good idea all along.
- Arthur C. Clarke
Where to begin?
Recently I made a post on Google+ about the future of graphics technology, citing both the Outerra and the upcoming Euclideon Engine. In some regard, they both represent an interesting crossroads in the industry where one company (Outerra) is using the polygon methodology mixed with procedural systems, while Euclideon is using a variant of a voxel based system with procedural methodologies. What transpired was a really interesting debate between myself and one of the staff from Outerra (Brano Kemen) who falls somewhere between the first and second category of debate as listed above from Arthur C. Clarke.
His insistence that Euclideon was likely just a hoax wasn’t what raised my eyebrow, but instead it was the reasoning he was using. In a typical fashion, he put forth his best arguments against Euclideon from the perspective of a current generation technology programmer.
The obvious stance being – Well, this isn’t a new idea and it’s been tried before, but nobody has managed to pull it off like Euclideon claims they have.
I’m a little perplexed at this statement because it’s really no different than saying:
Thousands of people in the past have tried this and failed, therefore it’s not possible that somebody will come along and succeed.
At best, that’s a bit of faulty logic.
Generally speaking, if you’re coming at this from the perspective of what you already know in the industry, then the Euclideon engine looks like nothing more than witchcraft and magic. Of course, Arthur C. Clarke also notably commented on this phenomenon as well:
Any sufficiently advanced technology is indistinguishable from magic.
– Arthur C. Clarke
Past Tense Future
I've done a few articles on this blog awhile back concerning the Euclideon engine and what I believed at the time were the general mechanics of the engine which made it so powerful (and unique). For the most part, the more I research into this system and read (or listen) to the details that are made available, the more I am convinced it is not a hoax. If you would like to read the original posts on this subject, feel free to hop over to Quantum Rush and Quantum Rush [Redux] for some perspective.
To be fair, the Outerra engine is marvelous in its own right and has definitely earned my respect for what is being accomplished, but in the grand scheme of things, it’s like praising one technology at the height of advancement but knowing it’s a sort of last hurrah! before the sun sets and a new age approaches. I say the same thing about Crytek as well, and even they believe that technology like Euclideon is not a hoax because they tried to import a point-cloud model of a tiger during their research on the Crytek engine.
So in the grand scheme of things, at least one company thinks it’s possible and they sure as hell are bigger than Outerra. That being said, Crytek also acknowledged that they didn’t manage to crack the secret recipe for point-cloud data approaches and went with the approach they have now in the Crysis 3 engine.
Outerra Engine in alpha stages. Ground is procedural, but models are polygon.
In the beginning, was the Pixel
Simply put, the Euclideon engine is not a typical voxel engine. It may use similar foundations from this technology, but that’s about where the similarities end.
In order to wrap our heads around the claim of Unlimited Detail, we first start with the idea that your screen only has a certain amount of pixel space. Let’s say this is your screen resolution and go with 1900x1200 as a default resolution (which is mine).
Now, the first thing we ask is how many pixels is that on the screen?
It’s really a math problem, simple multiplication:
1900 pixels wide by 1200 pixels height.
This math problem yields us an answer of:
2,280,000 pixels on screen.
We can display photographs on screen at 1900x1200 resolution and that still image looks, well, photorealistic. But in a 3D Engine, photorealistic seems to be a word reserved for rendering farms and a year of processing.
In the case of Euclideon, they realized something that I don’t think the rest of the industry realized when dealing with point-cloud data models. Typically we’re loading the entire model into memory and shuffling it around, and that actually does get computationally expensive, but we’re talking about also moving around all the little digital atoms in the model that you can’t see, which is about 90% of the model at any given time.
Why exactly are we shuffling around 90% of a model we can’t see? That means we’re wasting 90% of our processing power doing stuff nobody will notice.
The obvious question then became:
What can the user actually see?
The answer to this is clearly:
They can only see what’s in front of them at any given moment, or whatever pixels are lit up in their resolution.
Well, this gives us something interesting to think about. If the pixel space 1900x1200 is the maximum amount of pixels on screen at any moment, then the only thing we need to know from there is what color those pixels are.
This is just one half of the equation, though. So we set this thought aside for a moment and ask:
How do we figure out what the user can see out of those 2,280,000 pixels on screen?
The World of Tiny Little Atoms
Setting aside the answer of 2,280,000 pixels on screen, we then look to figure out how we deliver only what will satisfy that prior question of what of the models can be seen in that screen space.
This is the point where another explanation from Bruce Dell in a recent interview comes to mind where he explains that current systems are pulling the entire book (model) and processing, yet what happens when you index the individual words in the books properly and make them searchable?
In a point-cloud model, we’re working with little digital atoms, so the model file itself is comprised of atomistic descriptions. The goal, then is to figure out which of those digital atoms in the model file are visible on screen at any given moment and ignore the rest of the model file.
So when you take a model of 500,000 polygons and convert it to the Euclideon point-cloud data format (which for all intents and purposes likely also indexes those points in the file) you end up with a high resolution point-cloud model where every point is indexed for searching. This is where we get the beginning of the second part of our answer for Euclideon and where the typical voxel engines stop.
The Euclideon engine is said to work a lot like a search engine, where the only search query is:
What pixels on the screen correspond to the point-cloud data in the model files that this user can see?
When we do a camera check in-world to see what is in front of the user, the engine likely returns the models it sees, and then the individual points from those files that are visible to the user within the screen space (again, 2,280,000 pixels).
Knowing this information, we ask the point-cloud data models up front to return only the points within it that correspond to the pixels on screen. Since the point-cloud models are pre-indexed to begin with, and the points (atoms) inside of it are individually indexed inside the file, only the points in the model that match the search query are returned as an answer to the engine, ignoring the other 90% of the file up front.
What we are likely left with then is something like this in the scene:
Clearly this is a good start, but we have quite a lot of empty space in between those pixels that are unanswered from just the points in the model files that are stored. Obviously, we wouldn’t want to store the point-cloud data at absolutely full resolution (because the storage then becomes the issue), so there must be a way to fill in the blanks on this point cloud scene.
Luckily for us, we have an answer.
More or less, the idea of what happens in between the points on the point cloud model are likely left to an algorithmic interpolation sequence. If we have two points and there is space between them, then the algorithm takes the average of the two sides and creates a new point algorithmically generated.
This isn’t a new idea, but when applied to something like voxels it gets interesting. The closer you get to the atoms in a voxel display system the more they enlarge and even start generating algorithmic interpolation to fill in the spaces, creating detail where there wasn’t before.
This is a common technique in console emulators to upscale 2D sprite art to modern HD screens, so Mario as sprite art can look good when made bigger on your high definition computer screen. One of the more popular algorithms for this 2D Sprite scaling is the hqx algorithm, and is available as part of most current console emulators.
In image processing, hqx ("hq" stands for "high quality" and "x" stands for magnification) is one of the pixel art scaling algorithms developed by Maxim Stepin, used in emulators such as Nestopia, bsnes, ZSNES, Snes9x, FCE Ultra and many more. There are 3 hqx filters: hq2x, hq3x, and hq4x, which magnify by factor of 2, 3, and 4 respectively. For other magnification factors, this filter is used with nearest-neighbor scaling.
In much the same manner as Outerra makes use of procedural methodologies for their terrain, Euclideon seems to make use of procedural methodologies for a majority of the engine as applied to specialized point-cloud data that is indexed efficiently.
It’s like going to Google in order to search for something. Naturally you don’t expect Google to start at the beginning of the Internet and go through everything in a linear manner looking for what you searched for?
No, instead Google crawls the Internet and indexes the content which is why when you search for something it skips 90% of the Internet and brings back just what you searched for in a few milliseconds.
Therein is the secret for Euclideon. The typical 3D Engine is the equivalent of having to travel from the beginning to the end of the Internet to find what it needs, whereas Euclideon is the equivalent of skipping 90% of it and bringing the results in milliseconds. This, in turn, makes Euclideon a highly optimized graphics engine, and the indexing of the files happens when you’re converting the high resolution polygon models into the Euclideon point-cloud data format (the equivalent of Google crawling the Internet)
The former of the two approaches uses up quite a lot of processing power with computation that is wasted because 90% of what it’s dealing with can’t be seen by the user to begin with, while Euclideon frees up that computation for other things.
Under the traditional methodologies today, the graphics cards are just sufficient enough to handle the polygon detail and all the computations, but under the Euclideon methodology, that same graphics card suddenly has nothing to do and is vastly overpowered for the engine.
So what do we do with a graphics card that is sitting around twiddling its thumbs?
Now we have room for improvement, assuming you can even imagine improving on photorealistic 3D at 25-30 FPS on the CPU alone. Stuff like astronomical resolutions for screens, ultra high fidelity, lots of GPU free to do more advanced physics and lighting calculations, and more. Instead of sharing those calculations on the GPU as an afterthought after the initial 3D calculations, now the entire card is free to unleash its full attention on it.
So the claim of Unlimited Detail is accurate, but it’s a contextual statement. Unlimited Detail, but you don’t need to load all of infinity at once to see it any more than you need to see the entire universe you live in to claim it has unlimited detail. All you can see is what is in front of you at any moment, and you take for granted that the rest of infinity exists because it’s there when you get around to seeing it.
Essentially, Euclideon engine works like the real world does. Everything is made of tiny atoms, and nothing exists except what you can see at any given moment at the level of detail you can actually see.
Polygons are a lot like a Classical Physics approach while Euclideon is the Quantum Physics approach.
Which brings us to the sticking point concerning the ability to animate point-cloud data. It’s not exactly easy, and it has come with a lot of ups and downs. Originally, it was thought to just be impossible and that was that, but we now know that’s silly because of the three points from Arthur C. Clarke at the beginning of this article. Eventually, somebody actually did manage to figure out a way to animate sparse-voxels but the conclusion was akin to the second stage of revolutionary technology statements:
Yes it can be done, but it’s not worth the effort.
With Euclideon, I’m going to say they figured out the animation aspects further and will progress into the third statement of revolutionary technology:
I said it was a good idea all along.
The people in this industry years ago who came out and said that point-cloud data was the future of the graphics industry were likely correct. The problem with being a visionary is that often times you make predictions that are completely accurate, but far ahead of their time.
Because the technology and methodologies hadn’t been figured out in a reasonable time from those visionary statements, the rest of the industry lost interest and said:
“See? It was just a false alarm. Nobody came out with it, so it must not be possible. Nothing to see here, move along. Point and laugh at the suckers who bought into this idea to begin with.”
So when a company like Euclideon comes out and says they figured it out, those same industry people who saw the original visionary statements years ago chime in again and say the same thing –
“Oh, this again? Didn’t we already conclude it wasn’t possible and move on? *yawn* Tons of people already tried this before and failed. Clearly Euclideon is a hoax, because if it was going to be accomplished, it would have been one of us with all of our training, expertise and money - not some guy in his basement programming in his free time. Since we haven’t figured it out, clearly this Bruce Dell guy isn’t able to.”
Do you know what that makes the rest of the graphics industry, with thinking like that?
Arrogant. That’s what.
Or, more likely, these are the people who have heavily invested in their current generation technologies and are subject to technology lock-in. With all of that invested in their own systems, you would obviously think every single one of them would come out against Euclideon as a hoax or impossible. It’s a biased viewpoint altogether, with absolutely every possible motivation for being biased that exists. So why are we listening to them in the first place if we know they couldn’t possibly have an unbiased view of this type of technology?
Jon from id Software says it’s not possible for a few more years. Notch from Minecraft says Bruce Dell is a snakoil salesman. Even the CEO of Epic Games predicts that photorealism in games won’t happen until between 2019 and 2024.
Those people all have one thing in common:
Each has a vested interest in Bruce Dell and Euclideon being absolutely wrong.
Bruce Dell sounds unprofessional
For a guy who has been coding the Euclideon engine since 2006 as a hobbyist programmer in his spare time, do you really think he’s going to come out and get a professional actor and marketing team to do the presentation?
Secondly, he’s Australian and that accent comes across pretty heavily. He really does talk like that normally… cut the man a break.
Does this guy know the terminology of the industry? Probably not nearly as well as most would. It doesn’t really matter if you know what he’s talking about as long as he knows what he’s talking about in his head. Just because he doesn’t know the word for it doesn’t mean he doesn’t know what it is.
We’re talking about a guy who wasn’t told this stuff was impossible, and went into coding a graphics engine from scratch with the thought that it was perfectly feasible and he just didn’t know how at that time. This likely drove him harder than most coders would for solving the problem.
Think logically for a moment… when you go to college and try to earn a degree in programming, your professor is teaching you things they studied and learned from other people in the industry. There is a bias already in what you are going to learn, because it is heavily geared toward teaching you the current practices and not encouraging you to try and invent new ones. Hell, even if you’re just learning from the latest publishing of GPU Gems, you’re learning from the accomplishments of others in the industry who are themselves following accepted practice by the book. Sometimes standardized education is far more poisonous than unconventional learning methods.
They aren’t looking to turn you into a creative, unorthodox, thinker… they just want a guy/girl who can churn out code in an existing industry that has a vested interest in specifically not drawing outside of the lines on the coloring book.
Bruce Dell wasn’t poisoned by that preconceived ideology atmosphere. He didn’t even realize the lines existed to begin with, and as such likely was free to create a masterpiece of coding.
So, Euclideon is not a hoax?
It’s not likely that Bruce Dell and Euclideon are trying anything funny. If anything they seemed to have figured something out that much (if not the entire) industry failed to do. All it took was thinking from left field and the ability to ignore all the people who failed before hand telling him it wasn’t possible.
It’s funny how that works out.
I’m not going to sit here and tell you that I know exactly how Euclideon works, but I can give a simplistic rundown on what I think is happening based on what Bruce Dell has already said. From my point of view, it seems to work out just fine, and has the potential to do exactly what he says it can do. Of course, I’m not a programmer either… so all I can do is give an analysis from a layman point of view. There’s plenty going on behind the scenes of this engine, and I’ve only simplified it as much as possible for a general audience to digest.
What does that mean for the industry as a whole?
Just what Bruce Dell said in the original Euclideon demo video below – Your graphics are about to get a whole lot better, by a factor of about 100,000 times or more.
When is this supposed to come out? That’s the number one question on people’s minds right now concerning this technology. Well, I can say that Bruce Dell made a comment concerning this if anyone was paying attention. He had a number in his head, and that number was about 16 Months, give or take polishing things up for a release.
He said that around August 2011, and 16 months from then is December 2012. Give or take, that means a near Christmas release for 2012, or Spring 2013 release. There’s a good estimate for you to shoot for.
If Bruce Dell is particularly a man of twisted humor, he’d release Euclideon as a demo on December 21st, 2012 (Last day of the Mayan Calendar). Something tells me that is a very likely date, because it’s the end of an era… and would be fitting. If Euclideon does exactly what he’s claiming it does, then it may as well be the Apocalypse for polygons, unleashing an era of photorealistic gaming and virtual environments.