Scott Hanselman

Accidental Prescience and the Secrets of Project Natal

June 23, 2009 Comment on this post [16] Posted in Channel9 | Gaming
Sponsored By

I can't remember which episode, but a few years ago I mentioned on my podcast that I didn't understand why companies were spending so much time with touch screens and multi-touch input devices when we all have a perfectly good input device staring at us, unused, everyday - our webcams. Minority Report was not only a great movie, but a great user experience idea.

Johnny Chung Lee (I thought he and I had a bromance going, but it's just a fauxmance. It's one way, sniff, he doesn't know I'm alive! ;) did some amazing work in this space using the Wii remote a while back.


Ever since I saw Minority Report, perhaps even before since it's such an obvious idea, I've been searching and trying to figure out when and how this is going to happen. From my point of view, there's just no reason I shouldn't be able to make a small gesture and push a window over to another monitor. Swipe down in the air, minimize. It if was reliable, it'd be a perfect and elegant addition to the mouse and keyboard.

Johnny now works for Microsoft, and recently we learned that he's been working with the team that is doing Project Natal. If you've been under a virtual rock, here's a video what Natal does. Basically it tracks your body and you become the game controller. If it works, it'll be epic. If it fails, it'll be sad. The real question is WHEN. My bet is Christmas, only because it's obvious.

From Johnny's Blog:

The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera.

But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I've seen in academic research, never mind a consumer product. At times, working on this project has felt like a miniature “Manhattan project” with developers and researchers from around the world coming together to make this happen.

Before the world (or I) had ever heard of Project Natal, I pounced on interviewed Johnny at Mix 09 in Las Vegas. Recently Raleigh Buckner mentioned on Twitter that there was a lot "said without actually saying" in that interview, and darn it, he's right. I asked the right questions, and Johnny answered, but we (the collective) didn't see!

Now, go watch the interview again, this time with the knowledge of Project Natal's existence...

Johnny Lee on Computer Vision

Wow. I just bumped into Johnny Lee in the halls here at Mix09. I'm a huge fanboi with a man-crush on this dude. You've seen Johnny before on Channel 9 talking to Robert Hess.  Johnny's a legend (in my mind) in the computer vision space, and he put up with me gushing at him here at Mix09. We chatted in the hall about computer vision, what he's working on, how he got the gig at Microsoft and where he sees the future of human-computer-interaction.

Crazy stuff. I'm very excited to see how far they can take this.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Hosting By
Hosted in an Azure App Service
June 23, 2009 23:32
...and Big Brother is watching...
June 24, 2009 0:13
I'm just wondering how long it will be until I have access to something like this in a non-microsoft specific package.
June 24, 2009 0:42
This stuff is way cool, but it's frustrating that Microsoft can't seem to get it to market very fast. I worry Project Natal will go the way of Surface, and by the time it finally goes mainstream either a) it will be with another company or b) be old news. Windows Mobile and Zune also seem to have this problem- iPhone's gone through three revisions in the period WM has done almost .5. I'm sure a lot of hard workers are pushing to make this stuff happen, but just once I'd like to see Microsoft drop a "one more thing" that shuts up those Apple fanbois. :)
June 24, 2009 0:45
"there's just no reason I shouldn't be able to make a small gesture and push a window over to another monitor"

Sneeze, though, and you might blow away your entire desktop :/
June 24, 2009 0:48
Daniel - Agreed...we need to "just ship it." I'll be thrilled if this comes out for Christmas. If it's next Christmas, that'll be too late!
June 24, 2009 1:40
That's brilliant. I had watched the video at the time but there was so much hidden information in there... I'm suspecting there's still a lot more we don't know about. This thing he said about a plain camera phone becoming a 6-degrees motion detector (detecting the motion of the camera itself by just looking at changes in the environment) looks like a fantastic idea. Some dude recently made something vaguely similar (but working quite differently) by sticking a screen and a gyration mouse on a toy gun.
June 24, 2009 1:56
I saw the Project Natal videos a while ago, and it blew me away. And Johnny Lee's video above is really interesting - you can see how he is trying to not give away too much information :)

Lovely stuff....
June 24, 2009 9:50
I knew it! Scott has reached the level of "PreCog". Of course, you knew that already.... ah, nevermind.

It is ashamed that you and others at Microsoft cannot at least be informed of these new initiatives well in advance to bring this stuff together in a formal manner. Or maybe, you were supposed to know, but you missed that "special meeting" in Building XX. Especially to interview Johnny Lee and he has to do the dance to be careful what he says. And you know, he's wondering what Scott may already know and who may have leaked something to him.
June 24, 2009 13:03
Since I noticed Johnny is part of Natal team, I felt pretty confident they CAN make it :)

Anyway, I wanted to share with you two blogposts... First one is not very interesting (talking about Nathan & Johnny), however you can find some useful information in second email - especially link to company that was bought recently by Microsoft and developed very similar solution :)

June 24, 2009 15:23
I think it will be like those speech recognition software noone want or will use. It looks cool but is it really practical? Personaly I want more touchscreens, I love my tablet PC and it is more intuitive to use for non-geeks like ... my dog ;-).

So I think it will be cool but geek-country only.
June 25, 2009 3:53
I realise that Project Natal is leaps and bounds ahead of this, but in 2000 I had an Intel "Create and Share" Camera...with motion sensing games included, basketball, bubble pop...very rudimentary, but really, why did it take 9 years to go from that to Natal?
Daniel, agreed...except when it comes to Mobile vs. iPhone. The iPhone is just now getting to where my PPC was four years ago. In the Mobile market, MS is losing the PR war, not the RTM war.
ciao, D
June 26, 2009 0:39
natal is very very cool :) but you're on the inside scott ;) sneak down there with a dv cam and talk to them :) there need to be some channel9 videos on natal imo :)

microsoft has said natal is pretty far away, alluding to a 2010 release. but i dont know.. shane kim mentioed in an interview that MSR have been working on the algorithms in natal for several years and that "the team was looking for some hardware to put them into" and that was where the 3dv zcam came in.. that deal was actually setup late last year and locked some time in march i think.

also, since they have devkits and a fairly large amout of them too, i belive they are in the process of designing the device hardware wise (pcb layout and packaging and stuff)

kotaku reported that the natal in the secret e3 demo looked diffrent from the posters, more in the size of a projector.. there are also several reports of a laptop connected to natal. i personally belive that the embedded processor that is going to be in natal isnt done yet. its likely that the e3 natal software ran on the laptop, not on the natal it self.

very very interesting stuff.. scanning twitter for news is good but we really need more tech demos and from microsoft :) and a windows driver ;)
June 27, 2009 6:09
I'm struck by how negative these comments are, like microsft is damned if it does, and damned if it doesn't. It's pretty clear that project natal was a big deal, so they didn't announce it before it was ready, otherwise the very same people would be moaning microsoft was full of spain ,d " where is this magic thing" and so on. Just deal with it. Sorry for the people who are excited by the technology that it's Microsoft. Oh well, someone has to come up with these things. And to whoever was complaining about how long it was taking ... It's ONLY JUST BEEN ANNOUNCED. Get a hold of yourself. Exciting technology. I'm looking forward to it. And I'll be happy whenever it arrives, not bitter it wasn't sooner. Frankly I'm blown away I could have this in my house by christmas. How cool is that! Whoever makes it. Have a word with yourself peoiple. Always complaining. It's pathetic.
June 27, 2009 6:12
Spin, not spain. Humph. Typing on a mobile isn't great
June 27, 2009 6:16
I meant spin, not spain. I'm typing on a phone. It's not ideal. Still quicker and more accurate than an iphone though (I'm not on windows mobile btw)
January 26, 2010 17:29
I like the post with such a nice information. i recommend for the newbie programmers to read this blog. thanks Valentine’s Day SMS

Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.