Consumer Reports Annual Car Survey – What It Looks Like

0 Flares 0 Flares ×

As a Consumer Reports subscriber, I received an email asking for my input on vehicles. Here’s what Consumer Reports asked and how this particular survey might come up short.

Can You Say “Complicated”?

Surveying people with accuracy is difficult. First of all, many questions are open to interpretation. Second, the method for collecting answers is often imperfect. Finally, not every survey response offers the same amount of diligence and effort.

NOTE: If you need proof that surveys are problematic, think about all those political survey results we hear…more than half the time, these scientific surveys are patently incorrect.

In my opinion, the biggest flaw in the survey I received from Consumer Reports is that it’s too damn complicated. Take a look at this survey page (click for a larger view):

Consumer Reports annual car survey sample

Here’s a nice long list of serious problems that Consumer Reports wants to know about. How many people read this list completely and carefully?

Did you read each and every line of that survey? Me neither. Quite frankly, I can’t imagine that most people read this list carefully. Instead, they problem skim the headings like “steering/suspension,” and then click if/when they think they’ve had an applicable issue. I have three problems with this:

1) Some of the headings – like “Drive System” – are incredibly¬†ambiguous. If I asked a random man or woman on the street about problems with their vehicle’s “drive system,” who knows what kind of response I might get. Granted, the survey offers some examples, but there are a few sub-headings here that just aren’t descriptive enough.

2) This detailed list requires the respondent to make some fairly nuanced distinctions. For example: Let’s say your vehicle had a manual transmission with a grinding noise caused by worn synchros. Which category would you mark this down under?

  • Transmission and Clutch: Major?
  • Transmission and Clutch: Minor?
  • Noises and Leaks?
  • Drive System?

If you review things carefully, you’d probably pick the second item on my list. OR, you might get annoyed, close the survey, and move on…which brings me to my third problem.

3) This survey requires a fair amount of effort on behalf of the respondent. Therefore, you’re more likely to get responses from people who are either really angry, really happy, and or really excited about contributing to¬†Consumer Reports.

For all these reasons, I’d argue the results from the Consumer Reports surveys are anything but definitive.

Is the Consumer Reports Survey Worthless?

Of course not. Imperfect survey data is better than no survey data, and while the length, detail, and ambiguity of the survey undoubtedly leads to erroneous conclusions, the survey data is likely good enough to spot trends.

If (for example) very few Lexus owners report problems compared to BMW owners, it’s likely that Lexus is more reliable than BMW…especially if this trend holds for a period of years. Likewise, if a greater percentage of Chrysler owners report problems compared to the average, it’s likely that Chrysler products are below-average in terms of quality.

Yet it would be foolhardy to trust Consumer Reports data exclusively. In fact, the best approach would be to consult data from Consumer Reports alongside data from JD Power, reviews from trusted entities like Edmunds.com, and of course market data (resale value), which is arguably the economic expression of a vehicle’s quality and reliability relative to competing models. Looking at historical trends would also be wise, at least in the case of JD Power and Consumer Reports.

The point? Consumer Reports survey data is imperfect. Don’t buy a car just because Consumer Reports says so.

Filed Under: Auto News

RSSComments (5)

Leave a Reply | Trackback URL

  1. Mickey says:

    I gave up on Consumer Reports some years ago. No wonder they have these ridiculous reports on things. Like you said you need several reports to make a good decision on anything.

  2. Larry says:

    I too put no value on Consumer Reports.

    The best way I have found is to constantly talk to real owners and those who service car/truck. Years ago I found people had all kinds of trouble with a certain big selling Ford Model and the owners I talked to did no research before purchasing. Many said, they just assumed they would be okay because they saw so many on the road. Never buy any model until you talk to someone who services that model and never talk to anyone from a dealer service center. Independent people only.

    As for synchros,,,,,, who needs them anyway. I still double clutch. Habit from driving big trucks.

    Tim if you you catch this post, after 19 years and 150,000 miles, my T100 clutch has just started slipping, it goes in for replacement today.

  3. GoBig says:

    I’m a former subscriber to CR online. I found some of their information useful, but didn’t need most of it.

    It’s not like I trust the trade magazines for honest reviews. They seem beholden to their sponsors and have to walk a fine line. (I still subscribe to Truck Trend)

    Push polls have been used for years in politics, so why not use them in the automotive business. Remember there are lies, darn lies, and statistics.

  4. Anonymous says:

    I quit filling out dealer surveys 20 years ago when it nipped me in the keester being honest. Unless you mark the dealer surveys ‘completely satisfied’, the dealer rips you a new rear and doesn’t want to do business with you even if you mark just ‘satisfied’. To me, what is the difference? It goes against the dealer in terms of mfg. allotment of vehicles, etc. unless mark all completely satisfied to every question. If mfgs. and dealers don’t want the truth, don’t fill out surveys. Consumer reports almost always favors their sponsors and almost always favor honda and toyota over the big 3 and others.

0 Flares Twitter 0 Facebook 0 Google+ 0 Email -- 0 Flares ×