Abusing the Microsoft Research's Touch Mouse Sensor API SDK with a Console-based Heat-map
In August I purchased and reviewed the Microsoft Touch Mouse. I still use my Microsoft Arc Mouse more than the touch, initially due to what I felt was dodgy scrolling performance on the Touch Mouse, as I mentioned in my review. Still, I've kept it in my backpack and I use the Touch Mouse perhaps a few times a month and have kept the software up to date in case there's some software changes made to improve performance.
I can happily say that they've changed something and the scrolling performance is WAY better. I can finally get 1 to 2 pixel precision with it while scrolling in my browsers. The other nice feature is the "three finger swipe up" which gives you effectively a Windows version of the Mac Expose window switcher view.
Today I noticed while catching up on Long Zheng's excellent blog that the Touch Mouse Sensor SDK is available for download. Per their site:
"The Microsoft Touch Mouse supports multitouch gestures via an integrated sensor. The Touch Mouse Sensor API SDK is a small library intended to enable students and researchers to experiment directly with Touch Mouse sensor output.
Using this SDK, you can create your own applications consuming the 13×15 sensor image. The SDK includes C# and C++ samples demonstrating how to read and manipulate sensor data"
OK, so what can we do with this thing? It comes with a number of samples to get the bitmap from the sensor and process it. It includes examples in both C# and C++. Because I don't have C++ on this machine I got an error opening that solution, but you can just remove that project and still build the samples.
You don't need to do your work in a graphical application, although it's nice to visualize this data of course. First sample is a console app, that tells you were the center of mass of your big fat finger is.
The smarts are in the Microsoft.Research.TouchMouseSensor namespace. You implement a callback function that the mouse effectively calls whenever something interesting happens. It looks like this:
/// Function receiving callback from mouse.
/// <param name="pTouchMouseStatus">Values indicating status of mouse.</param>
/// <param name="pabImage">Bytes forming image, 13 rows of 15 columns.</param>
/// <param name="dwImageSize">Size of image, assumed to always be 195 (13x15).</param>
static void TouchMouseCallbackFunction(ref TOUCHMOUSESTATUS pTouchMouseStatus,
As you can see, the image size is 13x15, always. You get the raw bytes that represent the image, 195 bytes long.
The first thing I was surprised to see was just HOW MUCH of the surface area of the mouse is covered and how high the resolution is. I was thinking, "13x15? That's so small, how could it tell me anything useful with such a limited structure?" Well, that's how clueless I was.
Of course, each of these bytes is giving a number between 0 and 255 so a lot of information can be extrapolated between two pixels with different values and you can get a pretty detailed picture of what's going on. How much? Well, before I move on to a graphical example, why not abuse ASCII as much as we can?
First, I'll output just the hex values with this code, then show what it looks like when I move my figures around the surface of the mouse.
StringBuilder sb = new StringBuilder(300);
for (Int32 y = 0; y < pTouchMouseStatus.m_dwImageHeight; y++)
// Iterate over columns.
for (Int32 x = 0; x < pTouchMouseStatus.m_dwImageWidth; x++)
// Get the pixel value at current position.
int pixel = pabImage[pTouchMouseStatus.m_dwImageWidth * y + x];
if (pixel > 0)
Here's the result, animated.
Cool. But what about with color? I could use the default Console stuff and set the colors, or I could get direct access to the Console, use the Win32 APIs and what not, but I could also use Tim Sneath's ConsoleEx class that's on NuGet thanks to Anthony Mastrean! First, I'll "install-package ConsoleEx" to bring it in.
How about a mouse touch heat-map with color? I thought about building a whole "find closest color" map and trying to map 255 different colors, but then I just did this.
if (pixel > 230)
Console.BackgroundColor = ConsoleColor.Red;
else if (pixel > 128)
Console.BackgroundColor = ConsoleColor.Yellow;
else if (pixel > 90)
Console.BackgroundColor = ConsoleColor.Green;
else if (pixel > 45)
Console.BackgroundColor = ConsoleColor.Blue;
else if(pixel > 15)
Console.BackgroundColor = ConsoleColor.DarkBlue;
Console.BackgroundColor = ConsoleColor.Black;
ConsoleEx.WriteAt(x*2, y, " ");
And the result is this:
You can do the same thing in WPF, of course, with better bitmap support, but seriously, with a Console-based heatmap, who needs graphics? ;)
OK, fine, here's the sample code that takes the results of the TouchMouseSensorEventArgs and creates a grayscale image.
void TouchMouseSensorHandler(object sender, TouchMouseSensorEventArgs e)
// We're in a thread belonging to the mouse, not the user interface
// thread. Need to dispatch to the user interface thread.
void SetSource(TouchMouseSensorEventArgs e)
// Convert bitmap from memory to graphic form.
BitmapSource source =
PixelFormats.Gray8, null, e.Image, e.Status.m_dwImageWidth);
// Show bitmap in user interface.
SensorImage.Source = source;
And the result:
One way to look at this is as if the surface of the mouse is a tiny Xbox Kinect. What kinds of gestures could I recognize and hook up to do stuff? Control WIndows, do custom stuff inside my application, browser, or whatever. Any ideas?
The downloads are here: