Are Survey Results Misleading?

Over the weekend I was presented with a request to take a Yahoo survey. I was curious what Yahoo wanted to know and thought why not kill a few minutes and see what I could learn. The survey was simple enough and seemed fairly obvious what Yahoo was hoping to discover. My answers, while honest, made me wonder if the wrong conclusions could be drawn from my answers.

Some Background

One of the first things I did after purchasing my first laptop and connecting to the internet was to sign up for a Yahoo email account. I can’t remember if my then dial up provider (the one that was through k-Mart) directed me to Yahoo or I directed myself there, but within a few minutes I had my email account. A short time later I was also using the my.yahoo version of Yahoo as my home page.

It didn’t take long before I started customizing the content and the look of the page to suit me and I’ve been using that same page with an occasional modification ever since.

I’m not completely loyal to the page and could be swayed to try something else as a home page easily enough. Most of the web portals I’ve seen offer me essentially the same content and a good enough starting point for the day’s surfing. In fact there are probably a few out there that offer more. So why do I keep my.yahoo?

  1. Habit – I’m used to the page and it’s a familiar point of departure and return.
  2. Quick link to Yahoo email – Yahoo is also the only email account (I now have dozens) that doesn’t make it easy for me to download email into Thunderbird and it’s the only one I still read online.
  3. The customized look – Yahoo recently upgraded my.yahoo, but I was still able to keep the colors I had set up years ago.

One and two above keep me using the page, but number three is the one that keeps me from experimenting with a new one. If Google for example let me choose the colors and perhaps even the layout for iGoogle instead of limiting me to their pre-designed themes it might very well become my new home page.

The Survey

The survey request appeared as an overlay ad on my home page over the weekend. The survey asked the same three questions about a handful of web pages. Nine pages if memory serves.

  1. What was your impression of the download speed of the above web page?
  2. Approximately how frequently do you visit the above web page?
  3. How cluttered or uncluttered does the above web page appear to you?

At the end of the survey there are also some questions about how I rate my web expertise and what kind of internet connection I have as well as what my home page is.

Given that Yahoo’s main page is often referred to as being cluttered and that Google’s uncluttered design is often hailed as one reason for it’s success it’s reasonable that Yahoo might want to know the impression it’s pages leave with users. Some of the nine pages in the survey were Yahoo pages and the rest were pages from other search engines.

There’s nothing wrong with the survey per se and I’m glad to see Yahoo asking questions to hopefully improve the user experience. However I think the results they get might end up being somewhat deceptive.

My Answers

Looking at the questions it seems that Yahoo is trying to determine if impressions of the speed of a page and how cluttered the page appears are what determines how often the page is visited. Again nothing wrong with that, but I wonder what my answers indicate to them. For example I think my.yahoo home page is incredibly slow, but I have my reasons for keeping it. If it were any other page I might not wait around for it to finish loading. I may not think it’s too cluttered, but only because I know where everything is after years of use. I put it there after all. Obviously I visit the page many times a day.

Should Yahoo conclude that download times and clean design aren’t important to me? I hope not since both are important. My feelings about my home page have a lot to do with my familiarity with that page. I would feel different about it if I had only been using it a week. I’ve often cursed the page while waiting for it to load and as I said if another portal would let me customize the look completely I could easily switch. Habits can be broken and if sufficiently motivated I know there are workarounds to using the email.

I hardly ever visit the Live Search page. It loads fast and is very uncluttered, but I don’t care for the search results so the only time I use the search engine is when I want to know how a site is ranking in it for a specific phrase. On the other hand Google’s page also loads fast and is also very uncluttered. I do visit that page often during the day since I think the search results are good. Same impressions of the page yet very different frequency of use.

Another page in the survey was the Yahoo News main page. I never visit that page, but once again it has nothing to do with the page’s design. My my.yahoo home page already has links to the main news items I want. I’m in and out of Yahoo News all day. Just never on the main page.

Are The Results Misleading?

I realize I’m only one person and when you put together all the numbers this survey might show Yahoo some useful things and lead them to some valid conclusions. But since I was invited to take the survey through the my.yahoo home page I’m guessing that many others taking the survey also have my.yahoo set as their home page and that many people might also feel a bias one way or the other toward their familiar page.

Maybe I’m the one misleading myself here. I can say all I want that it’s important to me for a web page to download quickly and have a clean interface, but is it really that important to me? I visit quite a few sites daily that are painfully slow and suffer from information overload. The content is so good though, that I’m willing to wait and my familiarity with the site allows me to ignore the parts I don’t need to interact with. Fortunately for those sites I had sufficient cause to wait for the download on my first visit and I started engaging with the content before looking around at what else was there. I don’t think I’d be so patient with a site new to me.

What do you think? Could these questions easily lead to the wrong conclusions? Are my questions simply my own lack of understanding about surveys? Is Yahoo looking for something different than I think they’re looking for? In the end will the statistical numbers make up for a few oddball sets of answers? Or will the averages only hide what’s really going on in the minds of users?

If you’re interested in taking the survey you can find it here.

Download a free sample from my book, Design Fundamentals.

2 comments

  1. I have often thought surveys are a bad way to collect data. For a start the only people who do them are people who don’t mind taking surveys, So technically that would skew the results straight away.

    Also, If the person asking you to conduct the survey is the focus of the survey people might not be as likely to give honest answers. Like if i go out to eat and the waiter asks if i would mind filling out a small evaluation, I always put excelent for everything, because he is going to read it (i think thats a British thing), But i will still complain to whoever i am eating the meal with..

  2. That’s true that only a certain subset of people will take the time to fill out a survey and the results will be skewed. And it’s true that being the focus of the survey can bias those filling it out.

    I think they can still be valuable, but they should be looked at cautiously and I think they can very easily lead to things like getting results you’re hoping to get. I know there are ways to take into account some of the concerns, but still you may not want to put too much stock into what a survey tells you or at least understand they shouldn’t be looked at with blind faith in the results.

    Oh and it’s not a British thing. I’m pretty sure it’s universal.

Leave a Reply

Your email address will not be published.

css.php