Scott Hanselman

History is being written in a new way. It is being written by people through the big and the small events in their lives. We write history through personal Web sites, discussion boards, and the legacy of photos, taken at moments that are important to us.

Researchers at Microsoft have been working on a wide range of technologies that will help people write their personal histories through digital photography. To tell any history, it's best to start at the beginning, and move forward.

In the beginning, you buy a digital camera, take a quick look at the manual, throw it to the side, and start pushing buttons. Digital photography has made it possible to take an almost endless number of photos. Some of these photos are good; some get deleted before anyone else sees them. Many of them are almost good, with a little tweaking they'd be just fine.

Adjustable Light
A common problem with digital photography is lighting. Photos turn out either too light or too dark. Since digital cameras allow us to take lots of shots without running out of film, we're willing to throw away a lot of the bad shots. But if you want to take a great picture, it might be nice to be able to control the lighting without purchasing professional lighting equipment.

One of the research projects at Microsoft Research is called Continuous Flash. This technology allows you to take the same picture with flash and without flash and later adjust the balance between light and dark. It's better than contrast filters in photo editing tools, because it considers the reflection characteristics of each object in your picture.

"You can't compensate for having one area underexposed and one area overexposed," said Hugues Hoppe, one of the project researchers. "If an area is underexposed, you can't really get the detail back by increasing brightness, because it wasn't captured in the first place. By having two different images which both have useful information, you can merge them together."

Image Stacks
A similar technology that combines the best of multiple photographs is a project called Image Stacks. Image Stacks aligns multiple images of the same subject, allowing the user to pick and chose the best pieces from each photograph. Researchers Michael Cohen, Steven Drucker and Alex Colburn thought this would come in handy for special events, when you want to get a picture of the entire group that's suitable for framing.

Taking group photographs is difficult, because capturing a single image in which everyone looks good is almost impossible. What usually happens is that in one shot, someone has their eyes closed, but someone else has got the most adorable smile. Check the next shot, everyone has their eyes open, but one person is picking a poppy seed out of their teeth. The third shot, both previous people are behaving, but grandma is yawning, tired of waiting through multiple shots. Which shot do you pick? With Image Stacks, you can easily cut and paste to present everyone's best face. The images are automatically registered into a single composite image.

Print to Digital
Print photos are still around. They're around in shoeboxes. They're shoved under the bed and in the back of closets. Some of us have so many print photos we don't know what to do with them. Neatnik types tediously scan all of their print photos one-by-one, converting them to digital form. But most of us don't want to go to all this trouble.

Cormac Herley, a researcher in the Communications, Collaboration and Signal Processing (CCSP) group, has developed a way to allow people to scan multiple photos at one time. You can put as many photos as will fit on your scanner, and the software will recognize each photo separately. It can 'read' the edges of the photos, even if they're crowded together or tilted. When it converts them to digital photos, it will correct for orientation and position. It's a quick way to get the family memories out of the shoebox and onto your hard drive.

"It's a harder problem than it looks like on the surface. Many scanner makers have tried, but it hasn't worked before," said Herley. "But this really works, it's not just a demo."

Take Out the Red
The Media Computing group at Microsoft Research Asia has developed several cool image editing techniques to help you fix some common problems with any photograph. One of the worst that comes to mind is the glowing red eyes that result from the flash hitting the pupil in just the wrong way. Unless you're way into the vampire look, this just isn't right. The red eye fix, which currently ships in Digital Image Suite and Windows XP Media Center Edition, is one of the best photo retouch features around. All you do is move the 'target' over the red eye, click, and the red is out.

The same group has also developed a quick and effective way to fix the brightness and contrast in a picture. Even if you took a picture that is too dark, you can use the Levels Auto-Fix feature in Digital Image Pro to lighten up your picture.

Organize the Digital Shoebox
The Media Computing group has also made it easy to organize your digital photos using image recognition algorithms. Some people do the same thing they did with the shoebox, and scatter pictures all over their hard drive. Then they can't find the one picture they want to use for their holiday card. The group's algorithms can distinguish between indoor and outdoor shots, shots with people in them and shots without, and city and non-city scenes. Combined with other technologies, such as keyword annotation, it makes it a breeze to find any photo in your collection.

John Platt, a researcher in the CCSP group has developed another way to manage your photos online. His image clustering algorithm helps users find their photos by one of the most prominent markers: events. The software is effective because it doesn't look only at timestamps, which could be misled by a faulty camera clock. Instead, the software looks at photograph order plus color to find pictures taken during one particular event.

"We only compare colors locally in time," said Platt. "So if you have a pumpkin in one shot, and a few months later you wear an orange shirt, later, when you're searching for the day you visited the pumpkin patch, it won't show photographs from when you were wearing the orange shirt."

His algorithm underlies several other technologies designed to help people find their digital photos, including the Microsoft Research Media Browser. The Media Browser, developed by researchers in the Next Media group, takes advantage of the photo recognition research from the Media Computing group, and Platt's algorithms to build a unique visual experience that helps you search for and identify your photos. And it looks darn cool as it works. The interface is an impressive, futuristic presentation of photos that rearrange themselves before your eyes, sliding into place in a typical 2D presentation or a 3D stack.

"The idea behind this is annotation of large collections of photos," said Steven Drucker, the lead researcher on the project. "We know that if you put annotations on photos, that it's much easier to retrieve them. But we also know that it's tedious and difficult to do. We use the advanced techniques that are available, such as face detection and image clustering, to make it easier for you to interact with your photos. We also use a game graphics card for higher visual quality."

Fill It In
Smart Erase is a photo editing tool found in Digital Image Pro and invented by researcher Patrick Perez in Cambridge. The feature allows users to remove objects from a picture. This can come in handy in case you want to remove your ex from the family reunion picture, or you before you lost the 30 pounds.

The algorithm looks at areas of the image to see which patch of texture can be "stolen" to fill in the holes left behind when the unwanted object in the image is removed.

To fill in the hole, Smart Erase does some reasoning about texture. It views the pixels outside the object as potential replacement material. The program has some strategies for knowing exactly where to look to get this material. "The algorithm constantly reviews what pieces it's got and makes comparisons very quickly to come up with the right fit," said Andrew Blake, Senior Researcher in Cambridge.

Blend It
Another photo editing feature from the Cambridge lab, a tool code-named Blender, appeared in Digital Image Pro this year as the "Blending Brush." Blender is a seamless cloning tool that can take the wrinkles out of your face, insert a new object into a scene, and combine parts of one scene with another - all without the usual difficulties and distortions that most photo editing techniques present.

If an object inserted into a new background has complex outlines, standard cloning may not work because of the incompatibility of color and intensity between the background and the new object. And even the best, most careful cutting and pasting often yields poor results because the outlines are fuzzy or jagged. Blender 'blends' pieces of the inserted object and the background together to form a seamless whole.

Cut It Out
Cutting out an image and putting it somewhere else has always held a lot of fascination for photo aficionados. Blake and his team are developing a new algorithm they call GrabCut, that's a 'no-brainer' way to do this important task. Instead of having to carefully trace the outlines of the object you want to cut out, all you have to do is draw a rectangle around the object. The algorithm selects the object and eliminates the old background. You can then paste the cut-out object onto a different background.

Cartoon Wizard
Doesn't everyone want to star in their own anime or Disney cartoon?

Though Microsoft Asia researchers can't get you a Disney contract, they can turn your digital photograph into a cartoon. Their technology, developed in cooperation with MPD Japan, is called the Cartoon Wizard. It is currently offered in the Japanese version of Office 2003. Westerners will have to wait, as the Cartoon Wizard is only trained to work with Asian faces.

Their system is based on statistical learning techniques. The algorithm automatically generates a cartoon from an image using face detection and alignment, and training data generated by studying how a human artist renders a human image into a caricature. The resulting cartoons can be used in e-cards or personalized emoticons for chat programs.

Tell a Photostory
Now that you've stepped through the process of improving and organizing your digital photos, perhaps you'd like to share them. Microsoft Research has developed several ways to do this, in small and large ways.

When researcher Dave Vronay was working on PhotoStory, he wanted to recreate the feeling of a family sitting around an old-fashioned photo album and telling the stories connected to the pictures.

"A picture is not just a description of what is there," he said. "For instance, if you have a picture of a hotel, and you showed it to a friend, you probably wouldn't just say, 'and that's the hotel we stayed at.' You might instead launch into a story about the waiter with purple hair who served you duck soup at the hotel restaurant, even though you didn't have a picture of him. The photo would be a reminder of the stories that surrounded that photo."

With Photostory, you can add images, music, and background narration to tell the stories behind the pictures and send it to the people you'd love to have sitting on the couch next to you while you share your memories.

Share Your Photos with Friends
The Social Computing group is experimenting with an online blog and photo sharing application code-named Wallop, a project designed to help people to connect with those close to them — families and friends, and friends of friends.

The group considers Wallop a "social networking" application that provides a way for small, closely connected groups of people to share personal information and photographs online. The beta testers can send photos to their Wallop interface through email or instant messages to easily update their blog interface.

Share Your Photos with the World
The World Wide Media Exchange (WWMX) offers users from around the world the chance to upload and share their photos with millions. It provides MapPoint maps and TerraServer maps so that you can view your photos by location as well as time.

One of the advantages to this interface is the ability to communicate with people across the world. If you're planning a trip to London, for instance, maybe some nice tourist who has gone before you has posted their pictures of a trip around the city, complete with shots of their favorite tea stops. Then other tourists or locals can jump in and write annotations on the photographs, such as, "don't eat here, the crumpets aren't up to the usual standards."

Some of the contributors to the WWMX have contributed to history by adding photos of 'news' events, such as fires in Southern California and search and rescue operations.

A Visual Journey
David Salesin, a senior researcher in the Document Processing and Understanding group, has inspired many digital projects at Microsoft Research. He is also on the faculty at the University of Washington. Salesin recently became actively involved in a large digital photography project. A very large project. He contributed original digital photographs from his trip to Bhutan to the world's largest published book, a visual journey across the last unspoiled Himalayan kingdom on the planet.

The project was funded by several sources, including the Bill & Melinda Gates Foundation and the iCampus program at Microsoft Research. "Mike's project seemed like an interesting, original take on how we might be able to use technology for education," said Salesin. He helped convince the iCampus funding committee to support the project.