Scott Hanselman

Building an Embodied Social Proxy or Crazy Webcam Remote Cart Thing

February 17, 2010 Comment on this post [22] Posted in Coding4Fun | IIS | Microsoft | Remote Work
Sponsored By

Channel 9 Video on Embodied Social Proxies As a remote worker I'm always looking to find better ways to work with the team that don't involve actually going to Seattle. As much as I love/hate Microsoft, I'm not moving. However, this always puts me at a disadvantage. There's only so many places (and only so far) i can go as a remote worker.

When Phil and I made the "HanselPortal," essentially a persistent always-on Skype Chat, that gave folks a place to drop by and chat (at least those two lazy to just call me via Office Communicator) but there's already two people in Phil's office, I can't move (I'm a desktop computer) and my field of view is very small. Phil has a small inside office and I can't exactly hold meetings. It's a start, but it's not perfect.

I stumbled upon the Embodied Social Proxies project at Microsoft Research and got a tour from their team. While their system costs about $3000 to $4000 (While I'd love an Axis 213 PTZ camera I can't afford the $1300 and I'm still trying to convince the boss.) and has a great cart, monitor and camera, I figured I could at least do a proof of concept on the cheap. And by proof of concept, I mean, in the same way my 1963 Ford Falcon was a proof of concept; I kept driving it until it stops, then started walking.

The Idea

While the name "Embodied Social Proxy" is very PHDy and very Researchy, the idea is that you want a physical stand-in for yourself. Sure, people can call me and contact me in any of a dozen ways and honestly, it's one click at Microsoft. I'm in the address list, I'm on chat, and I've even got a 5 digit phone extension. I've done all this and more to integrate with headquarters. However, out of sight, out of mind is really true. People subconsciously or not like to associate something physical with you. So, until I can get a 6 foot tall mobile video conferencing robot a $20 cart with a laptop super-glued to it might help. I'll set it up in an office in Building 5 so folks can "stop by." I can be "brought" to meetings where perhaps setting up a RoundTable isn't feasible. Maybe it'll work, perhaps not. But I'm going to continue to try.Network Diagram

The Problems

The researches have done many iterations and discovered many things. You want "eye gaze" to be natural so you can tell when folks are looking at you and they can tell when you're looking at them. They recognize that you need to "look around" and watch whiteboards, so being able to pan and zoom around is important.

Security is important also. We don't want a domain-logged-in real user on a mobile cart wandering around Microsoft. However, as a remote person I need to be able to start up the camera without having to ask a person to do it. I need to control as much of the cart as possible remotely.

The General Solution

I've got Office Communicator "corporate chat" on my side, so that makes things easier. If you wanted to generalize this solution you could use Skype or other things. I also have Exchange so my free/busy details are available programmatically. Again, this could be Google Calendar, or whatever.

I'll have a laptop mounted (not movable) to a cart. It'll include an HD Webcam at a fixed position (no zoom or pan) and a second IP camera that allows Pan/Tilt/Zoom (PTZ). This should let me look around a bit. This is an IP camera because there are no good pan/tilt/zoom USB cameras on the market. PTZ cameras can be $200 or up to $3000 and more. Mine's $200.

I need the laptop to have both wired and wireless. Here's where things get a little weird. The camera will have one IP and the laptop will have another. However, the laptop will have a security certificate that will allow it to get on the network while the camera will not only not be reliable, but it'll be hard to find from a DNS perspective. I would like to be able to access both the camera and the machine as a unit, reliably. I solve that below, in what I think is a pretty clever way.

The (Cheap) Hardware

The hardware for my build is beyond cheesy cheap.

The Software

There's a custom WPF-based .NET client written by the MS Research Team. I'm gonna continue to bug them to let me release the code or Open Source it or something. I don't think the software is the important part, it's the idea. The software is a little tricky and cobbled, but workable.

Talking to Office Communicator

The machine is running on a local account that isn't Administrator. It's set to log in automatically, but the account has access to nothing. There's another Non-User Domain Account called "hanselcart" but while this account is in the Active Directory, but also has no access to resources or anything. That account has had Office Communicator access turn on so I can chat with it. Communicator is set to start up automatically as well. It's that ability to chat (really "address") the cart that makes it reasonably seamless.

You'll see that if I'm not currently visible on the screen, the program makes a call to Exchange Web Services and uses the Office Communicator Presence information to show where I am and what I'm doing generally, as well as a way to call me.

Screenshot of the Social Proxy Software

There's an Office Communicator SDK so you can automate and listen to Communicator:

this.messenger = new Messenger();
this.messenger.OnSignin += new DMessengerEvents_OnSigninEventHandler(messenger_OnSignin);
this.messenger.OnContactStatusChange += new DMessengerEvents_OnContactStatusChangeEventHandler(messenger_OnContactStatusChange);
this.messenger.OnIMWindowCreated += new DMessengerEvents_OnIMWindowCreatedEventHandler(messenger_OnIMWindowCreated);
this.messenger.OnIMWindowDestroyed += new DMessengerEvents_OnIMWindowDestroyedEventHandler(messenger_OnIMWindowDestroyed);
this.messenger.OnMyStatusChange += new DMessengerEvents_OnMyStatusChangeEventHandler(messenger_OnMyStatusChange);

The code is very COM-ish, which is unfortunate, but it's easy enough to write to. If I chat "Start Video" to the cart, the program sees it and calls me back:

void StartVideo()
IMessengerAdvanced advanced = (IMessengerAdvanced) this.messenger;
object result = advanced.StartConversation(CONVERSATION_TYPE.CONVERSATION_TYPE_VIDEO,, null, null, "1", null);

// Float a "hang up" window on top
if(hangupWindow == null) {
hangupWindow = new HangupWindow(MachineName);
hangupWindow.Closed += new EventHandler(hangupWindow_Closed);
hangupWindow.Owner = this;

// put the hang up window top-right corner
hangupWindow.Top = 0;
hangupWindow.Left = System.Windows.SystemParameters.PrimaryScreenWidth - hangupWindow.Width;

// Set the video to full-screen

The FullScreen part is a horrific SendKeys hack to press F5 for me until I talk to the OC team and figure out how to do this programmatically.

How to Address the IP Camera - "It's a WebFarm of WebCams with IIS's AAR"

I'm rather proud of this hack. Let's say the cart is called http://hanselcart. I'd like to be able to address the WebCam reliably even though it's not technically a managed-object on the MS network. Essentially, if the cart itself is on and online, I'd like to use the cart to get to the camera.

Then I remember that IIS (Internet Information Services, our Web Server) has an insanely powerful free module you can download called Application Request Routing or ARR. Regardless of what you think of the name IIS ARR, it really takes IIS to the next level. Think about it next time you are trying to do something tricky with IIS. It adds a bunch of features, but I figured I needed a Reverse Proxy, which ARR can do.

Instead, I made my local IIS a WebFarm with one node, the WebCam. ;) I put the Webcam on a local network "behind" the laptop (192.168.x.x) and told my Laptop's instance of IIS to route all traffic to the Webcam. In fact, ARR smartly did it for me when i used the Add WebFarm Wizard. Whole thing took 5 minutes, counting download time.

A Web Farm of Web Cams

NOTE: I did also try a tool called WebcamXP, and while it's amazingly cool for other things and has some really diverse features around Webcams, especially IP webcams, it used about 30% CPU for what I wanted, while IIS's ARR used 1-2%.

So now, I can just hit the cart by name and I'll get the IP Webcam. This Webcam has its own HTTP server and includes a quasi-RESTful API for Pan/Tilt/Zoom so I can video chat via Office Communicator and look around with the web interface of the IP Cam.


Here's a few shots of the cart itself. I'll be driving it up to Seattle for testing and installation this week.

HanselCart Full Body Shot Close Up of IP Cam

Thanks to Gina and John and especially Peter Provost for helping me (virtually from Dallas) build it. Wish me luck!

Related Links 

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Hosting By
Hosted in an Azure App Service
February 17, 2010 7:17
How about adding a Roomba somewhere below the cart so that you can move it around remotely :-)

Great article/idea, can't wait to see this action!
February 17, 2010 8:18
Can you package this up and fly it around to conferences? Poker nights?
February 17, 2010 10:51
Looks neat, though if it works really well people may become convinced you're some sort of android :). Maybe for version 2.0 you can add a projector so you can become full sized. If the walls are painted white you may not need a screen if you don't mind looking washed out.
February 17, 2010 11:14
Josh - the projector idea is interesting...I'd need a realtime green screen on my side though...;)
February 17, 2010 11:15
Keeron - I tried the Roomba but the power of its motor wasn't enough to move a laptop. To make this mobile, I'll need a stronger battery and a better motor.
February 17, 2010 12:34
It's a great idea, but I do wonder two things...
1) Why the cart? Seems like the proxy could get around easier, go more places, go to lunch with people if the unit were hand carry-able. I realize this is harder with oth-the-shelf/budget parts, but it seems like there's a Netbook/Windows Mobile/iPhone app for that that would be, um, handy.

2) How do you know whether someone's decorated your proxy to reflect your personality, or their view of your personality? Afraid I'm not the only person who first thought of 'kick me' jokes'. :)

February 17, 2010 15:10
OK... I totally cannot resist this... when your social proxy is tested and running free at Redmond, it should be assisted by a Mini-Scott, created by mounting a touch phone (with the same cam live feed) on a USB Tank Missile Launcher :)
February 17, 2010 18:06
I agree with Filini, or even further, just mount a couple usb rocket launchers to ensure proper remote protection, if you suddenly are being attacked by orcs.
February 17, 2010 19:17
Why when I see this do I immediately think... DJ Roomba.

You are missing on thing though. You need a cattle prod. So that you can zap people. Then you can chase people down the halls screaming "AH YAI YAI YAI YAI!"
February 17, 2010 19:48
Anyone else tempted to give Scott a call? And now back to useful comments... :)
February 17, 2010 19:50
Is the IP camera able to see the cart itself, so that you can monitor the evolution of your "embodiment" as your coworkers accessorize your cart? You might want to find a hallway with facing mirrors so you can check for "ping me" signs on your back.
February 17, 2010 19:59
That's awesome. Helps remind me the reason I got into programming and computers. We are inventors in many different ways. Good luck. Now if only you could find a way to use, then you could drive it as well :)
February 17, 2010 20:14
You just need a professional surrogate like the one George Bluth Sr. hired.
February 17, 2010 20:31
Denver... not Dallas. :)

Can't wait to get mine running. Grrrr.
February 17, 2010 20:35
"Sigh... Great. Bob isnt at work so we have to hook up his stupid video conference thing. You know lets just not call Bob because I am sick of maintain his virtual work presence all because he decided to sell his car and start riding his bike. Oh great. His battery didnt charge. Does anyone know where the power supply thing is for this. Great... Now the cord won't reach..."

Great for Chang. Bad for everyone else.

You have an office... Come to work or we are giving it to someone else.
February 17, 2010 21:50
Scott -- this looks like an awesome nerd project. I would love to have one of these since I would remotely 3 days a week.

Did you consider using a Logitech Quickcam Orbit AF? Is does have the ability to pan, tilt and zoom and also can be controlled
programmatically --
February 17, 2010 22:02
Do you really need the network switch/hub? Are you connecting the laptop to a wired network or using a wireless card?

I assume that you could use a cross-over cable from the laptop to the IP based PTZ camera along with some static IPs.

If you wanted to connect via wired network this would require adding a second NIC to the laptop (but I'll assume that you have several old PC Card NICs laying about your geeek layer).

Since you are adding the extra components to the cart, I assume that I am missing something...
February 18, 2010 0:33
Love it, love it, Scott. For flexibility in movement, definitely take a look at the top two platforms at FAQ at Four to six miles should be enough range for a remote day's work in Redmond. Add a doorknocker and you're in business within a single building. For multi-building excursions, you'd need to work out the problem of secure badge swiping. Not going to touch that one.
February 19, 2010 2:49

Great project, and timely too since I'm working at home today.

Yes, the Office Communicator SDK is a bit COM-y (well, it's all COM), but there is a new managed API coming out with the new version of Office Communications Server and Office Communicator.

Details from the PDC09 here and here.


February 19, 2010 7:18
We have been using a set-up that is similar to this (maybe a bit lower-tech) since December. I also used a similar one at a previous job where I was the remote person. It works great. I just posted a description and photos of it on my blog,
February 22, 2010 3:38
Just curious but why you went with a switch instead of a cross over cable between the laptop and the camera? Just wondering if there was some sort of limitation
February 26, 2010 20:47
The HanselNet Funding Bill is passed. The system goes on-line Febuary 16th, 2010. Human decisions are removed from software development.

HanselNet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, Febuary 29th.

In a panic, they try to pull the plug.

Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.