The idea behind Consumer Reports is a good one, but their methodology is inherently flawed and can't be fixed.
Not too long ago in this country, if you wanted to buy a product, you pretty much had to go by word-of-mouth or brand-name reputation. If a product turned out to be a piece of junk, you were pretty much stuck with it. There were no consumer protection agencies, no online review sites, and the cost of litigating a bad bargain was more than the item cost itself. Warranties were non-existent or lasted only 12 months or so - 12 months on a car! Imagine that. Today, warranties last a decade.
In the 1950's and 1960's our consumer movement emerged, where people stood up and said they were tired of planned obsolescence and crappy products. And Consumer Reports magazine became popular starting about that time. They refused to take ads and refused free samples of products. Instead, they went out and bought things anonymously and tested them and published their results. The idea was a good, the execution flawed, though.
(It should be noted that since 2012, the magazine has been spun off from Consumers Union and the website does now take ads, and in some instances, Consumer Reports has done evaluations of its advertisers and business partners - calling into question its impartiality).
(It should be noted that since 2012, the magazine has been spun off from Consumers Union and the website does now take ads, and in some instances, Consumer Reports has done evaluations of its advertisers and business partners - calling into question its impartiality).
The idea was that by not taking ads, they were not influenced by ad dollars. Car magazines, for example, give out awards and favorable reviews to companies that pay for ad space. Do an unfavorable review and you might lose millions in ad revenue. You can be bought and sold. And if they are giving you free cars and cars that are meticulously gone over before testing, well, you are also being influenced. Consumer Reports set out to avoid that conflict-of-interest.
So the Consumer Reports model sounded promising. But in execution, it didn't pan out very well. If you buy ten toasters and then "test" them, you really can't say much about them other than how well they toasted, whether the buttons and knobs were easy to use, and whether any of them caught fire or electrocuted the tester (the latter being "not acceptable").
Most consumers, however, don't buy things based on how nice the knobs are, and most toasters will make toast after a fashion. If they didn't, you'd take it back and get a refund. Most people buy based on price first, and maybe features secondary. And in that, we come to the first problem with Consumer Reports - they fail to acknowledge how price affects purchasing. People will tolerate slightly less usable features or functionality if the price is substantially lower. In the marketplace, price is king.
Even within the same product line, people make this trade-off. You might buy a car that comes in three trim levels - base, SE, and LE. And each has different options. The base has cloth seats and crank windows. The SE has power windows and a stereo. The LE has a leather interior and a navigation system. You might decide that even though the LE has more functionality, the $5000 price delta isn't worth it, and get the SE. We all make this trade-off assessment every day in commerce.
In the Consumer Reports world, price has little influence. Everything is either acceptable, recommended, or unacceptable, and they often will push a higher-priced product as being the optimal choice. And in the marketing world, prices are anything but fixed. So many folks seek out the lower-price or less-popular product, as the dealers are offering rebates.
The second problem with Consumer Reports is that they don't have the staff or resources to do long-term testing on products, and thus their evaluations are somewhat worthless. We care less about superficial judgments made at the time of purchase and more about longevity and reliability - two things Consumer Reports cannot really determine.
They try to offer some half-assed long-term reporting based on comments made by subscribers but as we shall see, these comments are often from a small sample base and are often based on bizarre criteria. Once in a while they might offer long-term data on the few cars they actually keep and use. But they cannot afford to buy one of each car ever made and then systematically test them to destruction (and to really perform a proper test, you'd have to buy ten of each car ever made). It just isn't practical or possible for them to do.
Even if they could offer long-term reliability data in this manner, it would be of little use, unless you could go back in time and purchase the product. By the time long-term data is available, the product is no longer on the market. It is of no use to tell me that such-and-such a car has a long-term reliability issue ten years after the fact.
They try to offer some half-assed long-term reporting based on comments made by subscribers but as we shall see, these comments are often from a small sample base and are often based on bizarre criteria. Once in a while they might offer long-term data on the few cars they actually keep and use. But they cannot afford to buy one of each car ever made and then systematically test them to destruction (and to really perform a proper test, you'd have to buy ten of each car ever made). It just isn't practical or possible for them to do.
Even if they could offer long-term reliability data in this manner, it would be of little use, unless you could go back in time and purchase the product. By the time long-term data is available, the product is no longer on the market. It is of no use to tell me that such-and-such a car has a long-term reliability issue ten years after the fact.
I'll give you an example of what I mean.
When I bought the Nissan Pickup, I wanted something cheap, reliable, large enough to haul our stuff, and powerful enough to haul a trailer. I looked online and researched the options for over three years. Consumer reports did a piece on small pickups and recommended the Nissan and Honda Ridgeline over the venerable Toyota Tacoma.
Now, as an Engineer, I can tell you the Toyota is the best of the three. But Consumer Reports mumbled something about "rear seat legroom" and "uncomfortable ride" and gave the nod to the Nissan. The Honda came out on top in their report, but they failed to consider that as a unibody vehicle (based on the Honda Pilot SUV) with a very short bed it really wasn't even a truck at all. And its low tow rating meant it wasn't in the running.
I bought the Nissan because of the price. The market speaks with regard to Toyotas. The dealers don't have to offer sales or rebates to move iron. To buy an equivalent Toyota, I would have had to shell out another $5000. That the Nissan had more horsepower was an added bonus. In terms of quality, longevity, and resale value, though, the Toyota would beat the Nissan hands down, regardless of Consumer Reports' feelings about rear-seat legroom.
Now, fast-forward three years, and Consumer Reports does an article - based on subscriber input - about vehicles "they regret buying!" At the top of the list for small pickups is the Nissan. Why? Poor gas mileage and "too large of a turning radius" are cited as concrete reasons why they thought buying a four-door pickup truck was a bad idea.
Um, the EPA rating on the 265 HP 4.0 V-6 is only about 22 mpg highway. And on a good day, that is all about it gets. You were expecting 30 mpg? Dream on. As for the turning radius, any vehicle that is over 20 feet long isn't going to turn on a dime, period.
I suspect their "consumer data" is one pissed-off lady who never thought about the EPA ratings when she bought the thing and somehow thought a long, long vehicle would be nimble. Based on this, Consumer Reports passes judgment - just as they bashed the Toyota based on "legroom" and "ride."
Even when Consumer Reports tries to go technical, they tend to screw things up. Back in the 1980's, they bashed the Suzuki Samurai as being prone to tipping over. They could have bashed it as an underpowered, tinny and poorly-made deathtrap piece of overpriced shit. But instead, they bolted huge metal poles to the sides of the thing and drove it around for the cameras intentionally sawing at the wheel until it tipped over. It was the worst sort of "investigative journalism" out there.
Why? Because any car can be made to go up on two wheels if you saw at the wheel hard enough. Look how bored Saudi rich kids do it - changing a tire at the same time! That you can get a car to tip over by making a tight turn - after bolting steel poles to the side, well, that's not "science" but just sensationalist bullshit. Accuracy In Media claims that CR set out to "bash" the Samurai in order to jigger up magazine sales.
And it is not the first time Consumer Reports went overboard with their ratings. In 1968 they rated the AMC Ambassador as "unacceptable" because the unit they bought had a small defect that cause gasoline to exit the fuel vent under heavy braking and that they felt the headrests were too soft. Not exactly a scientific study.
And it is not the first time Consumer Reports went overboard with their ratings. In 1968 they rated the AMC Ambassador as "unacceptable" because the unit they bought had a small defect that cause gasoline to exit the fuel vent under heavy braking and that they felt the headrests were too soft. Not exactly a scientific study.
Which gets us to the ultimate problem with Consumer Reports - the people who work there and write the reviews, as well as the subscribers are often clueless. They concentrate on superficial things and ignore more substantial ones. Part of this is the nature of the organization. You review a truck you have maybe a few hours to examine it. What are you going to say other than how convenient the ashtray was to the driver? Since you can't afford destructive testing or crash testing (as the Insurance Institute for Highway Safety can afford to do) you can't really provide any meaningful analysis if you wanted to. But a big part of it is that the people who write for them are journalists first, and technicians (if at all) second, and their subscriber base isn't very sophisticated, either.
You don't see "car guys" reading consumer reports for car information (nor toaster people for toasters). The sort of subscribers they get are risk-averse frightened consumers - the kind that like to complain loud and long about everything and make a nuisance of themselves at retail establishments. You know - the sorts who always repeat the mantra, "The customer is always right."
The idea behind a "Consumer Reports" type magazine is a swell one. However, there really is no way to properly execute such a concept in any kind of meaningful way. And sadly, the folks at Consumer Reports - like so many other "save the world" people, like to laud how they are helping us little folks out as if we should kiss their ass for the privilege.
Sorry, but I find that "unacceptable". Consumer Reports is just another organization that is dedicated to self-preservation above all else. There really isn't much of the way of useful data in their magazine or website.
The idea behind a "Consumer Reports" type magazine is a swell one. However, there really is no way to properly execute such a concept in any kind of meaningful way. And sadly, the folks at Consumer Reports - like so many other "save the world" people, like to laud how they are helping us little folks out as if we should kiss their ass for the privilege.
Sorry, but I find that "unacceptable". Consumer Reports is just another organization that is dedicated to self-preservation above all else. There really isn't much of the way of useful data in their magazine or website.