Measuring Satisfaction - We are a Nielsen Family!
Here in the US (and 40 other countries), for the last 60+ years, Television Ratings have been managed by the Nielsen Media Research company. They have a special magic sauce that let's them calculate "Reach" - the number of unique viewers for any show at any time. There's lots of good details on this at Wikipedia.
They figure there are 115 million TV households and a ratings point is 1%, or 1,152,000ish households. A show might get a 9.2 rating meaning that 10.6 million folks watched that show.
I always figured they had some magic boxes hooked up to televisions, or that they bought data from Tivo, or from my cable company. I mean, my TV knows that I'm watching Dr. Who (love the new Dr. Who), so I figured it told someone else.
Surprise, we're a Nielsen Family, at least this week. An envelope showed up with three US$10 bills and this paper booklet. Ah, how retro. Even the font, color choice and look and feel had a decidedly Mike Bradyesque vibe to it.
Surely since this company has had every American Television network by the short and curlies as a loyal client for the last half-century, they must have developed an extraordinarily sophisticated system for determining what I watch. Surely there's an explanation why Jericho was canceled (fortunately they've ordered new episodes) and the secret must lie within this mysterious envelope. My expectations are high...
Wow. That's...lovely. Welcome to the late 1980's. Really. I'm supposed to fill this out as I watch? This is the engine that makes the world's television ratings system spin?
This is such a confusing system that Nielsen has actually put up a website dedicated to decoding the paper booklet! I particularly like Step 5, "Returning the Booklet in the Mail" - truly a must-read if you're behind on sleep.Well, now I see why all the good shows are getting canceled. Why isn't this automated? Why aren't our TV's reporting back to the Mothership?
A TV can't tell if you're satisfied with a show, just if you're watching it. Same with a DVR. Presumably if you taped it, you wanted to watch it. This paper system is silly because it's asking me to copy down manually what someone (Tivo, Cable, whoever) must already know.
Or at the very least they could have created a data-entry website, rather than an interactive help website. Seems like if you're not clever enough to fill out this survey then you may not be able to get your computer turned on and browser navigated to http://tvdiary.tvratings.com, but perhaps that's presumptuous of me.
The only way (that I can see) to determine if someone likes a show is by asking them. Now, turning to other things that capture our attention like games. Xbox Live can tell if someone is satisfied because not only do they know when someone is playing a game, but they can see how far and how hard you're trying by looking at what Xbox Achievements you're getting. The Halo team is fanatical about their statistics. So, it seems much easier to glean the "satisfaction" one has for a game, but still, there's nothing better than asking.
Now, for a segue, but a totally honest question. If you have a piece of software that you've written, what's a good way to figure out if the end-user is really satisfied? I mean, Office, Windows, etc, all sell lots, but does that mean I'm satisfied with them? Should software include a star-ratings interface like some blog posts do on every dialog box? Perhaps when you click it, you're immediately asked for comment? Maybe a "Click here to blog about this feature" link?
One of the coolest most out of the box thinking ideas I saw come out of Microsoft last year was the Send a Smile Tool that was used in the Office 2007 Betas. Unfortunately you had to install it yourself - it wasn't automatically part of the beta experience.
However, the idea was brilliant in its simplicity. A number of bug reporting tools like FogBugz have similar tools for reporting bugs and attaching screenshots, but Send a Smile was even simpler.
You just clicked on the happy smilie or the sad smilie, entered your comment, and checked a box to optionally send a snapshot. That's it.
As a developer, I wish this tool was automatically included with Visual Studio and any major Developer Tool. Talk to the Office team, they have a database and UI and everything to manage these screenshots and smiles. A developer version could even optionally include the text of some code you're working on.
To be clear, there's LOTS of great places to give feedback about the Developer Tools, but the lower the barrier the better. Let me get you my opinions and thoughts with one click.
- For VS2008 Install and Setup, there's this forum.
- For VS2008 (Orcas) Releases there's this forum.
- Submit Suggestions and Bugs for the .NET Framework or Visual Studio here. SQL Server here.
Have you built satisfaction measuring tools (other than a "click here for a comments page) into your software?
Now, I want this for my TV so I can send Dr. Who and the BBC a smile.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
Stuart
Maybe if there was a "electronic paper system" wherein it could talk to your printer and print out a copy of all the shows you have watched and next to it the user checked their level of satisfaction. That would allow me as the user to ensure that un-necessary information was not being collected and it would also make it easier for me since I would not have to remember all the shows I have watched.
The Send a smiley idea is cool and I would be interested in data that would tell me how many people used it, and how much of that data was useful. After all, I am expecting the next version of Office to make me some tasty Ice Cream.
What about having UI windows ID'ed so that referencing them for feedback would allow me to clearly let the person reading my feedback know what I'm referring to?
What about allowing testers to provide audio feedback? If written text doesn't really give us the correct emotion the writer is feeling, having the ability to provide vocal feedback may be a step up. Developers can hear the hate or love a feature has evoked. It also may be easier for users to speak about their experiences that to write them down. After all, you normall speak with a therapist rather than write letters/email/IM.
What about allowing testers to provide video feedback? If audio is good, then video should kill it. In this day and age of YouTube users and screen captures, having a big red marker could make a difference. For instance, I'm at wits end with Visual Studio 2005's horizontal tab group always changing size without my consent. This is especially infuriating when using multiple monitors. To have the ability to show a developer when this occurs and diagram, brainstrom ideas of what it should be like would be great for me and would hopefully provide better insight for the developer.
I think whatever method of feedback is provided, it must be:
1. Easy to find.
2. Easy to fill out.
3. Capture emotion.
4. Allow users to be as expressive as they want to get.
5. Be very very careful not to collect any iformation a user might deem sensitive.
6. Provide value.
7. Be Unintrusive.
8. Evolve. Feedback mechanisms should evolve over time. The more I use a system, the more complexity and features I come across. A happy face might not cut it after 2 months of using the same interface.
Well. I think my mountain dew induced high is wearing off so I'll leave it at that. :-)
I always figured they had some magic boxes hooked up to televisions
They do. About 12 years ago I was a Nielsen family for 2 years while I lived in an apartment. There was a device that my TV plugged into, and the thing quietly sent data via the phone line back to Nielsen. Part of the arrangement was confidentiality; I was not supposed to announce that I was part of the program. So I was a bit surprised to see your blog post and scans of the survey information. :-) As far as I understand it, Nielsen has several different methods of gathering statistics, low tech and high tech. Guess you got picked for the low tech method. Choose wisely, we're depending on you.
At least with Scott on the case it sounds as if The Doctor is safe. Yay!
I'm not sure if your cable company knows what you are watching or not. They don't know if you are a basic cable subscriber or a cablecard user because those technologies are just pulling frequencies off the wire. Maybe the 2-way boxes send back data but I do consulting work for a cable company and they have never talked about it. I'm sure they could get info about Video On Demand usage and channel usage if they are using Switched Digital Video (most areas arn't).
I always figured they had some magic boxes hooked up to televisions ...
They do. I have friends who are a Nielsen family (though they're not allowed to discuss it), and there's a box that monitors everything - the Tivo (including the multiple independent tuners), the VCRs, the TVs, you name it. It's quite modern technology - Ethernet to connect all the parts, inside-the-box mods to some of the monitored devices, etc.
1. They attached a black box to the TV with it's own remote.
2. Whenever somebody sat down to watch tv, you would push a button on the Nielsen remote. Same button when they got up removed them from the system.
3. Nielsen viewers are randomly selected from amongst several different methods including your viewing diary and my parent's black box. The amount of time they are "viewers" is also randomly selected.
4. While my parents were on the program their cable was subsidized by Nielsen.
Your paper diary is definitely a PoS, but it's just the tip of the iceberg. Nielsen is completely losing its relevance, so this is just a small piece of the puzzle. You noticed the DVR link and that (along with satellite) are the current bane of Nielsen's system. I mean, even if DVR notes that you recorded something, how do we even know that you ever watched it? Same with FTA satellites. Add to that the fact that people are now downloading shows (left and right) and we have no way of knowing how many people actually watched these shows.
Obviously, they can't just "collect this data nightly" b/c non-random data is not relevant (and also a privacy concern). They need a small set of data from randomly selected viewers or they don't get statistical significance.
The big problem that Nielsen's is having is that their ratings are responsible for deciding which TV shows live or die and what advertisers are expected to pay for air-time. Advertisers need to have confidence or those numbers are meaningless and Nielsen's credibility is getting shakier. Add to the mix all of the begrudged TV show producers getting shows cancelled even when they're being watched by 2 million people / week and it's really obvious that Nielsen's is basically just waiting to die.
Of course, that stated, I love the "feedback button" idea. What are they odds of setting up a framework for this? Like an inheritable WinForm template with your little "5-star system"? Seems to me like a couple of days of work could go quite a ways in this field :)
My family has participated in Nielsen surveys in the past. They use paper forms because a significant fraction of US TV viewers do not use computers, fancy cell phones, or web/internet appliances. These census type forms allow validation of the "black box" data.
Charles
PS
I've watched Dr. Who, but I'm not impressed with it. Possibly it'll grow on me the way Stargate SG-1 did.
Some cable companies use a one-way connection which can't send back data.
Nielson might have a valid reason to collect data using paper. Maybe a simple inquiry call can help. Let us know.
A TV can't tell if you're satisfied with a show, just if you're watching it.
TV programming revolves around how many TVs they can get tuned to their station so that they can charge more for commercials -- not about if you liked a show. In the end, the "best" shows are the ones that people actually watched, not what people thought were the best.
So I urge everyone to watch The Doctor. :-)
it's a small black box connected to the TV which automatically registers the programme that is beeing watched (by decoding teletext or epg data). the box is installed in 1000 statistically selected households (2254 persons), each keeping the box for 3 months. the box comes with its own remote control that permits the viewers not only to indicate their presence but also to rate the programe. every night the data is transmitted to our servers over GSM network (other versions transmit over phone line or store the data on flash memory for later retrieval).
this works well for tv. for radio mesaurement we've recently developed a watch that records what you're listening to, whether it's at home, in the shopping mall, in the car or at your workplace (see http://www.radiocontrol.ch). you keep it for three weeks then you're sending it back. the system is simple: for every minute, the watch records a 4 seconds sample, compresses it to a FFT representation and stores the result. this data is later compared to the same FFT samples recorded in parallel for all radio programs we're interested in. But see for yourself:
http://real.xobix.ch/ramgen/radiocontrol/rc_film_en1.rm
http://real.xobix.ch/ramgen/radiocontrol/rc_film_en2.rm
http://real.xobix.ch/ramgen/radiocontrol/rc_film_en3.rm
http://nielsen-immi.com/
Good bye diary, hello big brother? Well no, the media has to be in the database for a match to occur.
I have a friend who really was a Nielsen family (I think we're doing some Nielsen-lite thing). They came to his house and installed a box that automatically transmitted everything back to the mothership. If I remember correctly they actually disassembled the TV and attached the box to the audio out signals. This enabled some pattern recognition algorithms to determine the show he was watching, although it seems there should be a simpler way to do this. Reading the comments it seems the system has evolved a lot over time. At this time they were having a real hard time with DVRs at the time because the pattern recognition was augmented by the time and area you were watching. This was maybe 5 years ago so I'm sure they've improved it, but it was really interesting.
My friend was definitely under a confidentiality agreement, but I don't recall anything warning me to keep mum in the booklet.
I also like the new Dr. Who - but I do miss Billie Piper (for all the obvious reasons <GRIN>).
The problem that I see with the system, is that if you are watching something, it doesn't mean you like it, only that you were watching it. This is of course, the only thing that advertisers are insterested in too. In general, the system doesn't work for identifying programming that is interesting or even deserves to stay on the air. All it will do, is capture what people happened to be watching at that moment.
Comments are closed.