Scott Hanselman's Computer Zen - Measuring Satisfaction

« VS 2008 and .NET 3.5 Beta 2 Releases Mad... | Main | Upgrading your project from Silverlight ... »

Measuring Satisfaction - We are a Nielsen Family!

Posted 2007-07-28 11:41 PM in Microsoft | Musings.

Nielsen Ratings Here in the US (and 40 other countries), for the last 60+ years, Television Ratings have been managed by the Nielsen Media Research company. They have a special magic sauce that let's them calculate "Reach" - the number of unique viewers for any show at any time. There's lots of good details on this at Wikipedia.

They figure there are 115 million TV households and a ratings point is 1%, or 1,152,000ish households. A show might get a 9.2 rating meaning that 10.6 million folks watched that show.

I always figured they had some magic boxes hooked up to televisions, or that they bought data from Tivo, or from my cable company. I mean, my TV knows that I'm watching Dr. Who (love the new Dr. Who), so I figured it told someone else.

Surprise, we're a Nielsen Family, at least this week. An envelope showed up with three US$10 bills and this paper booklet. Ah, how retro. Even the font, color choice and look and feel had a decidedly Mike Bradyesque vibe to it.

Surely since this company has had every American Television network ~~by the short and curlies~~ as a loyal client for the last half-century, they must have developed an extraordinarily sophisticated system for determining what I watch. Surely there's an explanation why Jericho was canceled (fortunately they've ordered new episodes) and the secret must lie within this mysterious envelope. My expectations are high...

Nielsen Ratings2

Wow. That's...lovely. Welcome to the late 1980's. Really. I'm supposed to fill this out as I watch? This is the engine that makes the world's television ratings system spin?

This is such a confusing system that Nielsen has actually put up a website dedicated to decoding the paper booklet! I particularly like Step 5, "Returning the Booklet in the Mail" - truly a must-read if you're behind on sleep.Well, now I see why all the good shows are getting canceled. Why isn't this automated? Why aren't our TV's reporting back to the Mothership?

A TV can't tell if you're satisfied with a show, just if you're watching it. Same with a DVR. Presumably if you taped it, you wanted to watch it. This paper system is silly because it's asking me to copy down manually what someone (Tivo, Cable, whoever) must already know.

Or at the very least they could have created a data-entry website, rather than an interactive help website. Seems like if you're not clever enough to fill out this survey then you may not be able to get your computer turned on and browser navigated to http://tvdiary.tvratings.com, but perhaps that's presumptuous of me.

The only way (that I can see) to determine if someone likes a show is by asking them. Now, turning to other things that capture our attention like games. Xbox Live can tell if someone is satisfied because not only do they know when someone is playing a game, but they can see how far and how hard you're trying by looking at what Xbox Achievements you're getting. The Halo team is fanatical about their statistics. So, it seems much easier to glean the "satisfaction" one has for a game, but still, there's nothing better than asking.

Now, for a segue, but a totally honest question. If you have a piece of software that you've written, what's a good way to figure out if the end-user is really satisfied? I mean, Office, Windows, etc, all sell lots, but does that mean I'm satisfied with them? Should software include a star-ratings interface like some blog posts do on every dialog box? Perhaps when you click it, you're immediately asked for comment? Maybe a "Click here to blog about this feature" link?

SatisfactionBox

One of the coolest most out of the box thinking ideas I saw come out of Microsoft last year was the Send a Smile Tool that was used in the Office 2007 Betas. Unfortunately you had to install it yourself - it wasn't automatically part of the beta experience.

However, the idea was brilliant in its simplicity. A number of bug reporting tools like FogBugz have similar tools for reporting bugs and attaching screenshots, but Send a Smile was even simpler.

You just clicked on the happy smilie or the sad smilie, entered your comment, and checked a box to optionally send a snapshot. That's it.

As a developer, I wish this tool was automatically included with Visual Studio and any major Developer Tool. Talk to the Office team, they have a database and UI and everything to manage these screenshots and smiles. A developer version could even optionally include the text of some code you're working on.

To be clear, there's LOTS of great places to give feedback about the Developer Tools, but the lower the barrier the better. Let me get you my opinions and thoughts with one click.

Have you built satisfaction measuring tools (other than a "click here for a comments page) into your software?

Now, I want this for my TV so I can send Dr. Who and the BBC a smile.

Sunday, July 29, 2007 12:34:51 AM (Pacific Standard Time, UTC-08:00)

I'm surprised that they still have a paper based system like that in the US. I'm in Australia and our family was selected for a TV ratings box. It is a little set top box that is attached to every TV in the house. Each time the TV is turned on you have a small remote that is programmed with the name and age of each family member. You then have to press the button for each person viewing the show. If someone leaves the room you have to press their button again to remove them. If you have visitors over you can add people ad hoc it asks for their gender, and age. Every night at 2:00am the mothership in Sydney would phone our place and a special wireless device sitting on the phone line would pick up the call quickly enough that we wouldn't even hear it ring. Then it sucks up the data from each set tip box and sends it down the line. It was very nice and simple. They told us that when our whole family (my wife and 4 kids) were all watching a show together, it was representative of approx 20,000 viewers. They can also track whether you are channel surfing or watching a show proper. In the end we got a bit tired of tracking all the viewers, but 2 years was a good run at it. You can have a box for up to 5 yrs max.

Stuart

Sunday, July 29, 2007 3:06:52 AM (Pacific Standard Time, UTC-08:00)

I am totally shocked that Nielson uses a paper system. But then it occured to me that I would not want my dvr/tivo/electronic device sending info to the mothership. How would I know what info they were taking? How would I know what they were doing with it? How can I trust them? Why should I trust them?
Maybe if there was a "electronic paper system" wherein it could talk to your printer and print out a copy of all the shows you have watched and next to it the user checked their level of satisfaction. That would allow me as the user to ensure that un-necessary information was not being collected and it would also make it easier for me since I would not have to remember all the shows I have watched.

The Send a smiley idea is cool and I would be interested in data that would tell me how many people used it, and how much of that data was useful. After all, I am expecting the next version of Office to make me some tasty Ice Cream.

What about having UI windows ID'ed so that referencing them for feedback would allow me to clearly let the person reading my feedback know what I'm referring to?

What about allowing testers to provide audio feedback? If written text doesn't really give us the correct emotion the writer is feeling, having the ability to provide vocal feedback may be a step up. Developers can hear the hate or love a feature has evoked. It also may be easier for users to speak about their experiences that to write them down. After all, you normall speak with a therapist rather than write letters/email/IM.

What about allowing testers to provide video feedback? If audio is good, then video should kill it. In this day and age of YouTube users and screen captures, having a big red marker could make a difference. For instance, I'm at wits end with Visual Studio 2005's horizontal tab group always changing size without my consent. This is especially infuriating when using multiple monitors. To have the ability to show a developer when this occurs and diagram, brainstrom ideas of what it should be like would be great for me and would hopefully provide better insight for the developer.

I think whatever method of feedback is provided, it must be:

1. Easy to find.
2. Easy to fill out.
3. Capture emotion.
4. Allow users to be as expressive as they want to get.
5. Be very very careful not to collect any iformation a user might deem sensitive.
6. Provide value.
7. Be Unintrusive.
8. Evolve. Feedback mechanisms should evolve over time. The more I use a system, the more complexity and features I come across. A happy face might not cut it after 2 months of using the same interface.

Well. I think my mountain dew induced high is wearing off so I'll leave it at that. :-)

Sushant Bhatia

Sunday, July 29, 2007 4:45:15 AM (Pacific Standard Time, UTC-08:00)

I always figured they had some magic boxes hooked up to televisions

They do. About 12 years ago I was a Nielsen family for 2 years while I lived in an apartment. There was a device that my TV plugged into, and the thing quietly sent data via the phone line back to Nielsen. Part of the arrangement was confidentiality; I was not supposed to announce that I was part of the program. So I was a bit surprised to see your blog post and scans of the survey information. :-) As far as I understand it, Nielsen has several different methods of gathering statistics, low tech and high tech. Guess you got picked for the low tech method. Choose wisely, we're depending on you.

Terri Morton

Sunday, July 29, 2007 5:23:10 AM (Pacific Standard Time, UTC-08:00)

They could just stick a chip in me to monitor heart-rate and rapid eye movement while I watch Lost couldn't they? I might not like the idea of Big Brother watching me (for anyone under 25 thats a 1984 reference [1984 the book not the year]) but I really don't care if he watches what I watch. Maybe then Drive would have survive.

At least with Scott on the case it sounds as if The Doctor is safe. Yay!

Mike Minutillo

Sunday, July 29, 2007 7:17:39 AM (Pacific Standard Time, UTC-08:00)

TiVo does collect usage data, but i don't think they associate it with your account, so they don't have any demographic data. You can see what other TiVo users are watching at TiVo Central.

I'm not sure if your cable company knows what you are watching or not. They don't know if you are a basic cable subscriber or a cablecard user because those technologies are just pulling frequencies off the wire. Maybe the 2-way boxes send back data but I do consulting work for a cable company and they have never talked about it. I'm sure they could get info about Video On Demand usage and channel usage if they are using Switched Digital Video (most areas arn't).

Tony Bunce

Sunday, July 29, 2007 7:18:05 AM (Pacific Standard Time, UTC-08:00)

I always figured they had some magic boxes hooked up to televisions ...

They do. I have friends who are a Nielsen family (though they're not allowed to discuss it), and there's a box that monitors everything - the Tivo (including the multiple independent tuners), the VCRs, the TVs, you name it. It's quite modern technology - Ethernet to connect all the parts, inside-the-box mods to some of the monitored devices, etc.

Anon E. Mouse

Sunday, July 29, 2007 7:56:06 AM (Pacific Standard Time, UTC-08:00)

Hm, well no one ever sad anything to me about confidentiality, and I signed nothing. I could maybe see why a super-box family might want to stay anonymous, as their opinion probably matters, but how are 1.6M paper diary people supposed to keep mum?

Scott Hanselman

Sunday, July 29, 2007 9:23:29 AM (Pacific Standard Time, UTC-08:00)

I am surprised that every cable/dish box doesn't just collect this data and send it back nightly. So much more accurate and so simple.

Aaron Fischer

Sunday, July 29, 2007 7:04:06 PM (Pacific Standard Time, UTC-08:00)

OK guys, my father was a Nielsen family for several years, here's how it worked:

1. They attached a black box to the TV with it's own remote.
2. Whenever somebody sat down to watch tv, you would push a button on the Nielsen remote. Same button when they got up removed them from the system.
3. Nielsen viewers are randomly selected from amongst several different methods including your viewing diary and my parent's black box. The amount of time they are "viewers" is also randomly selected.
4. While my parents were on the program their cable was subsidized by Nielsen.

Your paper diary is definitely a PoS, but it's just the tip of the iceberg. Nielsen is completely losing its relevance, so this is just a small piece of the puzzle. You noticed the DVR link and that (along with satellite) are the current bane of Nielsen's system. I mean, even if DVR notes that you recorded something, how do we even know that you ever watched it? Same with FTA satellites. Add to that the fact that people are now downloading shows (left and right) and we have no way of knowing how many people actually watched these shows.

Obviously, they can't just "collect this data nightly" b/c non-random data is not relevant (and also a privacy concern). They need a small set of data from randomly selected viewers or they don't get statistical significance.

The big problem that Nielsen's is having is that their ratings are responsible for deciding which TV shows live or die and what advertisers are expected to pay for air-time. Advertisers need to have confidence or those numbers are meaningless and Nielsen's credibility is getting shakier. Add to the mix all of the begrudged TV show producers getting shows cancelled even when they're being watched by 2 million people / week and it's really obvious that Nielsen's is basically just waiting to die.

Of course, that stated, I love the "feedback button" idea. What are they odds of setting up a framework for this? Like an inheritable WinForm template with your little "5-star system"? Seems to me like a couple of days of work could go quite a ways in this field :)

Gates VP

Sunday, July 29, 2007 7:50:11 PM (Pacific Standard Time, UTC-08:00)

Hi folks,

My family has participated in Nielsen surveys in the past. They use paper forms because a significant fraction of US TV viewers do not use computers, fancy cell phones, or web/internet appliances. These census type forms allow validation of the "black box" data.

Charles

PS
I've watched Dr. Who, but I'm not impressed with it. Possibly it'll grow on me the way Stargate SG-1 did.

Charles D. Quarles

Sunday, July 29, 2007 10:12:01 PM (Pacific Standard Time, UTC-08:00)

Some cable companies use a one-way connection which can't send back data.

Nielson might have a valid reason to collect data using paper. Maybe a simple inquiry call can help. Let us know.

Abdu

Sunday, July 29, 2007 10:36:39 PM (Pacific Standard Time, UTC-08:00)

A TV can't tell if you're satisfied with a show, just if you're watching it.

TV programming revolves around how many TVs they can get tuned to their station so that they can charge more for commercials -- not about if you liked a show. In the end, the "best" shows are the ones that people actually watched, not what people thought were the best.

So I urge everyone to watch The Doctor. :-)

Tony

Monday, July 30, 2007 12:10:52 AM (Pacific Standard Time, UTC-08:00)

wow, how old school :) i work for the Swiss Public Broadcasting Company and we've been using electronic audience measuring since 1985. we've developed and fine-tuned our own system for television measurement called Telecontrol (see http://www.telecontrol.ch).

it's a small black box connected to the TV which automatically registers the programme that is beeing watched (by decoding teletext or epg data). the box is installed in 1000 statistically selected households (2254 persons), each keeping the box for 3 months. the box comes with its own remote control that permits the viewers not only to indicate their presence but also to rate the programe. every night the data is transmitted to our servers over GSM network (other versions transmit over phone line or store the data on flash memory for later retrieval).

this works well for tv. for radio mesaurement we've recently developed a watch that records what you're listening to, whether it's at home, in the shopping mall, in the car or at your workplace (see http://www.radiocontrol.ch). you keep it for three weeks then you're sending it back. the system is simple: for every minute, the watch records a 4 seconds sample, compresses it to a FFT representation and stores the result. this data is later compared to the same FFT samples recorded in parallel for all radio programs we're interested in. But see for yourself:
http://real.xobix.ch/ramgen/radiocontrol/rc_film_en1.rm
http://real.xobix.ch/ramgen/radiocontrol/rc_film_en2.rm
http://real.xobix.ch/ramgen/radiocontrol/rc_film_en3.rm

olivier

Monday, July 30, 2007 7:25:05 AM (Pacific Standard Time, UTC-08:00)

Nielsen has now partnered with a company call IMMI that uses windows mobile cell phones to monitor media.

http://nielsen-immi.com/

Good bye diary, hello big brother? Well no, the media has to be in the database for a match to occur.

jefbak

Monday, July 30, 2007 8:35:22 AM (Pacific Standard Time, UTC-08:00)

Must be hitting our area...I got the booklet this week as well. Mine came with a $20 and a $10, though.

I have a friend who really was a Nielsen family (I think we're doing some Nielsen-lite thing). They came to his house and installed a box that automatically transmitted everything back to the mothership. If I remember correctly they actually disassembled the TV and attached the box to the audio out signals. This enabled some pattern recognition algorithms to determine the show he was watching, although it seems there should be a simpler way to do this. Reading the comments it seems the system has evolved a lot over time. At this time they were having a real hard time with DVRs at the time because the pattern recognition was augmented by the time and area you were watching. This was maybe 5 years ago so I'm sure they've improved it, but it was really interesting.

My friend was definitely under a confidentiality agreement, but I don't recall anything warning me to keep mum in the booklet.

Emil Lerch

Monday, July 30, 2007 9:44:34 AM (Pacific Standard Time, UTC-08:00)

Wonder how they get around the other distribution methods? I download quite a few shows on ITunes that I never watch on the TV. Does the booklet account for this other viewing method. Obviously the black box can't. Someone responded that Neilsen was becoming "irrelevant". I suspect that is very much the case - since the TV is becoming less and less of the monopoly media driver in many households. My electronic based entertainment time is now split between music, TV, and games - and TV is probably the least exercised of the three.

I also like the new Dr. Who - but I do miss Billie Piper (for all the obvious reasons <GRIN>).

Matt Childs

Monday, July 30, 2007 10:24:55 AM (Pacific Standard Time, UTC-08:00)

I got to do Nielsen ratings for radio a few years ago. Seeing as I hardly listen to music (I tend to listen to AM talk radio), I decided that my Nielsen votes were probably important... I'm definately not the majority listener. My experience was very similar with a radio listening diary.

The problem that I see with the system, is that if you are watching something, it doesn't mean you like it, only that you were watching it. This is of course, the only thing that advertisers are insterested in too. In general, the system doesn't work for identifying programming that is interesting or even deserves to stay on the air. All it will do, is capture what people happened to be watching at that moment.

Ryan Beesley

Thursday, August 02, 2007 9:22:40 AM (Pacific Standard Time, UTC-08:00)

I'm sorry but the whole idea of a paper diary needs to be eliminated. So does the people meter. I had a friend who did the paper diary and she waited until the end of the week to fill it out. Totally against the daily input, but Nielsen is none the wiser. She told me she had forgotten some shows she watched early on in the week, so they were excluded or she LIED about what she watched, just to fill it out. Aside from that, many people think the diaries and boxes are spread throughout the nation, to capture the diversity of the viewing audience.. right? No, think again. New York has a heavy distribution of nielsen families. 5,000 boxes and diaries hold the fate of 100 million households. Not only is the sampling inaccurate, that's an extremely low, low ratio. Now I hope it is clear to many people who've always wondered why great shows get canceled. I also hope every nielsen family reading this will give "Jericho" their attention when Season Two starts. It really is a good show, I'm glad it's back.

Karen

Monday, August 06, 2007 7:53:04 AM (Pacific Standard Time, UTC-08:00)

Hi Scott,

Thanks for the inspiration: Feedback for your smart client.