Scott Hanselman

Abusing the Microsoft Research's Touch Mouse Sensor API SDK with a Console-based Heat-map

January 19, '12 Comments [24] Posted in Coding4Fun | Tools
Sponsored By

In August I purchased and reviewed the Microsoft Touch Mouse. I still use my Microsoft Arc Mouse more than the touch, initially due to what I felt was dodgy scrolling performance on the Touch Mouse, as I mentioned in my review. Still, I've kept it in my backpack and I use the Touch Mouse perhaps a few times a month and have kept the software up to date in case there's some software changes made to improve performance.

I can happily say that they've changed something and the scrolling performance is WAY better. I can finally get 1 to 2 pixel precision with it while scrolling in my browsers. The other nice feature is the "three finger swipe up" which gives you effectively a Windows version of the Mac Expose window switcher view.

Today I noticed while catching up on Long Zheng's excellent blog that the Touch Mouse Sensor SDK is available for download. Per their site:

"The Microsoft Touch Mouse supports multitouch gestures via an integrated sensor. The Touch Mouse Sensor API SDK is a small library intended to enable students and researchers to experiment directly with Touch Mouse sensor output.

Using this SDK, you can create your own applications consuming the 13×15 sensor image. The SDK includes C# and C++ samples demonstrating how to read and manipulate sensor data"

OK, so what can we do with this thing? It comes with a number of samples to get the bitmap from the sensor and process it. It includes examples in both C# and C++. Because I don't have C++ on this machine I got an error opening that solution, but you can just remove that project and still build the samples.

You don't need to do your work in a graphical application, although it's nice to visualize this data of course. First sample is a console app, that tells you were the center of mass of your big fat finger is.

The smarts are in the Microsoft.Research.TouchMouseSensor namespace. You implement a callback function that the mouse effectively calls whenever something interesting happens. It looks like this:

/// <summary>
/// Function receiving callback from mouse.
/// </summary>
/// <param name="pTouchMouseStatus">Values indicating status of mouse.</param>
/// <param name="pabImage">Bytes forming image, 13 rows of 15 columns.</param>
/// <param name="dwImageSize">Size of image, assumed to always be 195 (13x15).</param>
static void TouchMouseCallbackFunction(ref TOUCHMOUSESTATUS pTouchMouseStatus,
byte[] pabImage,
int dwImageSize)
{
...
}

As you can see, the image size is 13x15, always. You get the raw bytes that represent the image, 195 bytes long.

A console app outputing the center of mass as an X,Y coordinate

The first thing I was surprised to see was just HOW MUCH of the surface area of the mouse is covered and how high the resolution is. I was thinking, "13x15? That's so small, how could it tell me anything useful with such a limited structure?" Well, that's how clueless I was.

Of course, each of these bytes is giving a number between 0 and 255 so a lot of information can be extrapolated between two pixels with different values and you can get a pretty detailed picture of what's going on. How much? Well, before I move on to a graphical example, why not abuse ASCII as much as we can?

First, I'll output just the hex values with this code, then show what it looks like when I move my figures around the surface of the mouse.

Console.SetWindowSize(50,16);
Console.SetCursorPosition(0, 0);
StringBuilder sb = new StringBuilder(300);

for (Int32 y = 0; y < pTouchMouseStatus.m_dwImageHeight; y++)
{
// Iterate over columns.
for (Int32 x = 0; x < pTouchMouseStatus.m_dwImageWidth; x++)
{
// Get the pixel value at current position.
int pixel = pabImage[pTouchMouseStatus.m_dwImageWidth * y + x];
if (pixel > 0)
sb.AppendFormat("#{0:X2}", pixel);
else
sb.Append(" ");
}
sb.Append("\n");

}
Console.Write(sb.ToString());

Here's the result, animated.

Cool. But what about with color? I could use the default Console stuff and set the colors, or I could get direct access to the Console, use the Win32 APIs and what not, but I could also use Tim Sneath's ConsoleEx class that's on NuGet thanks to Anthony Mastrean! First, I'll "install-package ConsoleEx" to bring it in.

How about a mouse touch heat-map with color? I thought about building a whole "find closest color" map and trying to map 255 different colors, but then I just did this.

if (pixel > 230)
Console.BackgroundColor = ConsoleColor.Red;
else if (pixel > 128)
Console.BackgroundColor = ConsoleColor.Yellow;
else if (pixel > 90)
Console.BackgroundColor = ConsoleColor.Green;
else if (pixel > 45)
Console.BackgroundColor = ConsoleColor.Blue;
else if(pixel > 15)
Console.BackgroundColor = ConsoleColor.DarkBlue;
else
Console.BackgroundColor = ConsoleColor.Black;

ConsoleEx.WriteAt(x*2, y, " ");

And the result is this:

You can do the same thing in WPF, of course, with better bitmap support, but seriously, with a Console-based heatmap, who needs graphics? ;)

OK, fine, here's the sample code that takes the results of the TouchMouseSensorEventArgs and creates a grayscale image.

void TouchMouseSensorHandler(object sender, TouchMouseSensorEventArgs e)
{
// We're in a thread belonging to the mouse, not the user interface
// thread. Need to dispatch to the user interface thread.
Dispatcher.Invoke((Action<TouchMouseSensorEventArgs>)SetSource, e);
}

void SetSource(TouchMouseSensorEventArgs e)
{
// Convert bitmap from memory to graphic form.
BitmapSource source =
BitmapSource.Create(e.Status.m_dwImageWidth, e.Status.m_dwImageHeight,
105, 96,
PixelFormats.Gray8, null, e.Image, e.Status.m_dwImageWidth);

// Show bitmap in user interface.
SensorImage.Source = source;
}

And the result:

256 color grayscale heatmap of the Touch Mouse

One way to look at this is as if the surface of the mouse is a tiny Xbox Kinect. What kinds of gestures could I recognize and hook up to do stuff? Control WIndows, do custom stuff inside my application, browser, or whatever. Any ideas?

The downloads are here:

Enjoy!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

Developer Stand up Comedy - Coding 4 Fun

May 5, '11 Comments [27] Posted in ASP.NET | Coding4Fun | Musings | Speaking
Sponsored By

imageThere aren't enough funny developer talks in my opinion. Lots of you folks are so funny in person and when I got to conferences or just have lunch we laugh and laugh. As with all niches or cliques, we're all part of a little club of inside jokes and shared stories.

It's always weird to get lots of comments on my technical talks. Many are positive, but sometimes I'll get one like "your not funy!" or "stop joking and start coding." Well, I realize my brand of edutainment isn't for everyone.

That said, this talk is happily content-free. If you hate my jokes, you may ignore it. ;)

Many years ago I did some local open mics and stand up, although I've never really put together a tight hour of material. I'm less of a stand up and more of a storyteller. Anyway, I got signed up for a talk in Holland last week called "Coding 4 Fun" but didn't prepare anything. I was already signed up for 6 other talks and had already presented 3 times by the time I showed up on stage at the 7pm "geek night" session.

There were a bunch of people there almost an hour early so I just sat on the end of the stage and we chatted. It was great fun. Next thing we knew, the guy was recording the video and the room was filling up.

So, I talked and told stories and chatted my co-workers and friends on IM for a while, the showed some community built NuGet packages. The only think I wish is that you could see the audience in this video. They were fun and engaged and silly.

Big thanks to Glenn Block and Clemens Vasters for sitting in the front row and playing along, and to Damian Edwards, Tim Heuer and Phil Haack for being silly over IM.

A lot of the jokes are in the IM or the code, so if you can't see them (even though I zoom in) with smooth streaming, download a higher res version so you can see them.

Coding for Fun with Scott Hanselman recorded in Holland in April of 2011

I hope you enjoy. Maybe one day I will put together a real act of "stand up for developers."

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

Building an Embodied Social Proxy or Crazy Webcam Remote Cart Thing

February 17, '10 Comments [22] Posted in Coding4Fun | IIS | Microsoft | Remote Work
Sponsored By

Channel 9 Video on Embodied Social Proxies As a remote worker I'm always looking to find better ways to work with the team that don't involve actually going to Seattle. As much as I love/hate Microsoft, I'm not moving. However, this always puts me at a disadvantage. There's only so many places (and only so far) i can go as a remote worker.

When Phil and I made the "HanselPortal," essentially a persistent always-on Skype Chat, that gave folks a place to drop by and chat (at least those two lazy to just call me via Office Communicator) but there's already two people in Phil's office, I can't move (I'm a desktop computer) and my field of view is very small. Phil has a small inside office and I can't exactly hold meetings. It's a start, but it's not perfect.

I stumbled upon the Embodied Social Proxies project at Microsoft Research and got a tour from their team. While their system costs about $3000 to $4000 (While I'd love an Axis 213 PTZ camera I can't afford the $1300 and I'm still trying to convince the boss.) and has a great cart, monitor and camera, I figured I could at least do a proof of concept on the cheap. And by proof of concept, I mean, in the same way my 1963 Ford Falcon was a proof of concept; I kept driving it until it stops, then started walking.

The Idea

While the name "Embodied Social Proxy" is very PHDy and very Researchy, the idea is that you want a physical stand-in for yourself. Sure, people can call me and contact me in any of a dozen ways and honestly, it's one click at Microsoft. I'm in the address list, I'm on chat, and I've even got a 5 digit phone extension. I've done all this and more to integrate with headquarters. However, out of sight, out of mind is really true. People subconsciously or not like to associate something physical with you. So, until I can get a 6 foot tall mobile video conferencing robot a $20 cart with a laptop super-glued to it might help. I'll set it up in an office in Building 5 so folks can "stop by." I can be "brought" to meetings where perhaps setting up a RoundTable isn't feasible. Maybe it'll work, perhaps not. But I'm going to continue to try.Network Diagram

The Problems

The researches have done many iterations and discovered many things. You want "eye gaze" to be natural so you can tell when folks are looking at you and they can tell when you're looking at them. They recognize that you need to "look around" and watch whiteboards, so being able to pan and zoom around is important.

Security is important also. We don't want a domain-logged-in real user on a mobile cart wandering around Microsoft. However, as a remote person I need to be able to start up the camera without having to ask a person to do it. I need to control as much of the cart as possible remotely.

The General Solution

I've got Office Communicator "corporate chat" on my side, so that makes things easier. If you wanted to generalize this solution you could use Skype or other things. I also have Exchange so my free/busy details are available programmatically. Again, this could be Google Calendar, or whatever.

I'll have a laptop mounted (not movable) to a cart. It'll include an HD Webcam at a fixed position (no zoom or pan) and a second IP camera that allows Pan/Tilt/Zoom (PTZ). This should let me look around a bit. This is an IP camera because there are no good pan/tilt/zoom USB cameras on the market. PTZ cameras can be $200 or up to $3000 and more. Mine's $200.

I need the laptop to have both wired and wireless. Here's where things get a little weird. The camera will have one IP and the laptop will have another. However, the laptop will have a security certificate that will allow it to get on the network while the camera will not only not be reliable, but it'll be hard to find from a DNS perspective. I would like to be able to access both the camera and the machine as a unit, reliably. I solve that below, in what I think is a pretty clever way.

The (Cheap) Hardware

The hardware for my build is beyond cheesy cheap.

The Software

There's a custom WPF-based .NET client written by the MS Research Team. I'm gonna continue to bug them to let me release the code or Open Source it or something. I don't think the software is the important part, it's the idea. The software is a little tricky and cobbled, but workable.

Talking to Office Communicator

The machine is running on a local account that isn't Administrator. It's set to log in automatically, but the account has access to nothing. There's another Non-User Domain Account called "hanselcart" but while this account is in the Active Directory, but also has no access to resources or anything. That account has had Office Communicator access turn on so I can chat with it. Communicator is set to start up automatically as well. It's that ability to chat (really "address") the cart that makes it reasonably seamless.

You'll see that if I'm not currently visible on the screen, the program makes a call to Exchange Web Services and uses the Office Communicator Presence information to show where I am and what I'm doing generally, as well as a way to call me.

Screenshot of the Social Proxy Software

There's an Office Communicator SDK so you can automate and listen to Communicator:

this.messenger = new Messenger();
this.messenger.OnSignin += new DMessengerEvents_OnSigninEventHandler(messenger_OnSignin);
this.messenger.OnContactStatusChange += new DMessengerEvents_OnContactStatusChangeEventHandler(messenger_OnContactStatusChange);
this.messenger.OnIMWindowCreated += new DMessengerEvents_OnIMWindowCreatedEventHandler(messenger_OnIMWindowCreated);
this.messenger.OnIMWindowDestroyed += new DMessengerEvents_OnIMWindowDestroyedEventHandler(messenger_OnIMWindowDestroyed);
this.messenger.OnMyStatusChange += new DMessengerEvents_OnMyStatusChangeEventHandler(messenger_OnMyStatusChange);
this.messenger.AutoSignin();

The code is very COM-ish, which is unfortunate, but it's easy enough to write to. If I chat "Start Video" to the cart, the program sees it and calls me back:

void StartVideo()
{
IMessengerAdvanced advanced = (IMessengerAdvanced) this.messenger;
object result = advanced.StartConversation(CONVERSATION_TYPE.CONVERSATION_TYPE_VIDEO, this.contact, null, null, "1", null);

// Float a "hang up" window on top
if(hangupWindow == null) {
hangupWindow = new HangupWindow(MachineName);
hangupWindow.Closed += new EventHandler(hangupWindow_Closed);
hangupWindow.Owner = this;

// put the hang up window top-right corner
hangupWindow.Top = 0;
hangupWindow.Left = System.Windows.SystemParameters.PrimaryScreenWidth - hangupWindow.Width;
}

// Set the video to full-screen
maxVideoTimer.Start();
hangupWindow.Show();
}

The FullScreen part is a horrific SendKeys hack to press F5 for me until I talk to the OC team and figure out how to do this programmatically.

How to Address the IP Camera - "It's a WebFarm of WebCams with IIS's AAR"

I'm rather proud of this hack. Let's say the cart is called http://hanselcart. I'd like to be able to address the WebCam reliably even though it's not technically a managed-object on the MS network. Essentially, if the cart itself is on and online, I'd like to use the cart to get to the camera.

Then I remember that IIS (Internet Information Services, our Web Server) has an insanely powerful free module you can download called Application Request Routing or ARR. Regardless of what you think of the name IIS ARR, it really takes IIS to the next level. Think about it next time you are trying to do something tricky with IIS. It adds a bunch of features, but I figured I needed a Reverse Proxy, which ARR can do.

Instead, I made my local IIS a WebFarm with one node, the WebCam. ;) I put the Webcam on a local network "behind" the laptop (192.168.x.x) and told my Laptop's instance of IIS to route all traffic to the Webcam. In fact, ARR smartly did it for me when i used the Add WebFarm Wizard. Whole thing took 5 minutes, counting download time.

A Web Farm of Web Cams

NOTE: I did also try a tool called WebcamXP, and while it's amazingly cool for other things and has some really diverse features around Webcams, especially IP webcams, it used about 30% CPU for what I wanted, while IIS's ARR used 1-2%.

So now, I can just hit the cart by name and I'll get the IP Webcam. This Webcam has its own HTTP server and includes a quasi-RESTful API for Pan/Tilt/Zoom so I can video chat via Office Communicator and look around with the web interface of the IP Cam.

HanselCam

Here's a few shots of the cart itself. I'll be driving it up to Seattle for testing and installation this week.

HanselCart Full Body Shot Close Up of IP Cam

Thanks to Gina and John and especially Peter Provost for helping me (virtually from Dallas) build it. Wish me luck!

Related Links 

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

Windows Home Server Twitter Notification Plugin

July 14, '09 Comments [7] Posted in Coding4Fun | Home Server | Source Code
Sponsored By

A while back, the very wise Brendan Grant sent me some sample code that would use Twitter to report the health of one's Windows Home Server. I always meant to update the code to use TweetSharp to talk to Twitter, as well as add some robustness for connected/disconnected scenarios, but I'm just never going to get around to it. Instead, here it is as he sent to me.

There's a REALLY vibrant community around Windows Home Server plugins and if you've got a WHS and you want it to do something that it doesn't do, I'd encourage you to jump in.

Even as I'm posting this, I'm sure there are better and more interesting implementations. However, I like what Brendan has done to abstract away the core COM-based API of WHS for use in managed code.

Here's the full program...note again that the PostTweet() method is hacked together and should use a more robust technique:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Xml.Serialization;
using Microsoft.HomeServer.SDK.Interop.v1;

namespace Twitter_Test
{
class Program
{
static string username = "";
static string password = "";

static void Main(string[] args)
{
IWHSInfo info = new WHSInfoClass();
//Register application name
info.Init("WHS Twitter Client");

NotificationCallbackClass notificationClass = new NotificationCallbackClass();
//Register notification callback class
info.RegisterForNotifications(notificationClass);

//Check current state
Console.WriteLine("Current State: " + notificationClass.GetHealthState().ToString());

notificationClass.HealthChanged += new EventHandler(notificationClass_HealthChanged);

Console.WriteLine("Monitoring for health changes. Press to exit.");
Console.ReadLine();
}

static void notificationClass_HealthChanged(object sender, HealthChangedEventArgs e)
{
Console.WriteLine("Current State " + e.Health.ToString());
PostTweet(username, password, "Your Windows Home Server's health is now: " + e.Health.ToString());
}

private static Status PostTweet(string username, string password, string message)
{
string user = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password));
// determine what we want to upload as a status
byte[] bytes = System.Text.Encoding.ASCII.GetBytes("status=" + message);
// connect with the update page
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://twitter.com/statuses/update.xml");
// set the method to POST
request.Method = "POST";
// set the authorisation levels
request.Headers.Add("Authorization", "Basic " + user);
request.ContentType = "application/x-www-form-urlencoded";
// set the length of the content
request.ContentLength = bytes.Length;

request.ServicePoint.Expect100Continue = false;

// set up the stream
Stream reqStream = request.GetRequestStream();
// write to the stream
reqStream.Write(bytes, 0, bytes.Length);
// close the stream
reqStream.Close();

HttpWebResponse response = request.GetResponse() as HttpWebResponse;

StreamReader sr = new StreamReader(response.GetResponseStream());
string s = sr.ReadToEnd();


XmlSerializer ser = new XmlSerializer(typeof(Status));
object o = ser.Deserialize(new StringReader(s));
Status status = o as Status;

return status;
}
}
}

There interesting part is the Eventing part where he makes changes in your Home Server turn into .NET Events via callbacks. Check the code for details. You can get events when Physical Disks are changed, when Backup States change, or when basically anything happens. There's a number of folks on Twitter already who have their Windows Home Servers tweeting.

If you've got, or you're using a plugin to report your Home Server status on Twitter (or SMS or whatever) leave a comment and I'll update the post! I'm sure there are better solutions than this little sample.

Here's the code if you want it, and remember, it may kill your pets. If so, don't blame me as I'll deny everything. It's a sample you found on the Internet, what did you expect?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

Hanselminutes Podcast 141 - Coding4Fun with Dan Fernandez and Brian Peek - Wiimotes and YouTube

December 5, '08 Comments [1] Posted in Coding4Fun | Podcast
Sponsored By

My one-hundred-and-forty-first podcast is up. I talk with Dan and Brian as they turn the successful Coding4Fun blog into a book. Brian shares how to interface with the Nintendo Wii's Wiimote, and Dan tells us how to download and convert YouTube videos in one click.

Subscribe: Subscribe to Hanselminutes Subscribe to my Podcast in iTunes

Do also remember the complete archives are always up and they have PDF Transcripts, a little known feature that show up a few weeks after each show.

Telerik is our sponsor for this show!

Building quality software is never easy. It requires skills and imagination. We cannot promise to improve your skills, but when it comes to User Interface, we can provide the building blocks to take your application a step closer to your imagination. Explore the leading UI suites for ASP.NET and Windows Forms. Enjoy the versatility of our new-generation Reporting Tool. Dive into our online community. Visit www.telerik.com.

As I've said before this show comes to you with the audio expertise and stewardship of Carl Franklin. The name comes from Travis Illig, but the goal of the show is simple. Avoid wasting the listener's time. (and make the commute less boring)

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Page 1 of 9 in the Coding4Fun category Next Page

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.