How Objective Is Top Gear’s Rating System?

As I mentioned recently, I have been leafing through the Top Gear 2016 Car Buyers [sic] Guide. I made an interesting discovery.

2016 Ford S-Max: www.ford.co.uk
2016 Ford S-Max: http://www.ford.co.uk

Top Gear provides a rating for every car they review, graded as marks out of ten. I crunched the ratings for six classes of car: MPVs, city cars, hatchbacks, executive cars, sports cars and cabriolets.

In advance, what would you expect to find?  A reasonable hypothesis is that the range of talent in each class of cars is the same. That means the efforts made to craft  MPVs are the same as the efforts made to craft cabriolets. The efforts should have equal chances of success. In other words the chances of there being a good MPV should be the same as the chances of there being a good cabriolet.

2016 Top Gear car ratings
According to TG, sports cars are better than MPVs and ragtops are nearly as good. That or TG is biased.The vertical axis shows average marks out of ten.

However, while flicking through the Sports Car section I noticed that several of the 37 sports cars rated received ten out of ten. In contrast not one of the city cars got higher than eight of out of ten. What that means is that either a) city car makers are not as good at making city cars as sports car makers are at making sports cars, or b) that city cars is a dying market with less investment in new product, or c) Top Gear has a bias towards sports cars.

The statistics above show that if you offer TG a sports car or a cabriolet you have a better chance of getting 10 out of 10 than if you offer an MPV or a city car. According to TG the average rating for an MPV is 5.89 out of 10. And the average rating for a sports car is 8.43 out of ten.

The average rating of all the cars included in the survey is 6.9 out of ten. Assuming that each class of car is comprised of equally good or bad examples, you’d expect that the average value of the sub-.groups would be same as the overall average.

Think of it like this: if a restaurant critic tested 50 Italian and 50 Chinese restaurants, you’d expect the average rating to be the same in both groups, other things being equal. The results would have a normal distribution in statistispeak. If there was a variation from this, such that Italian restaurants scored markedly lower than the Chinese ones, it would make one want to look for a reason. We could look to see if there was a good supply of Italian chefs in the region, for example. We could check the supply of ingredients for Italian food. We could also query the objectivity of the restaurant reviewer.

What story can we imagine to explain why TG rates sports cars and convertibles over other types of cars? Is it really that sports car makers are better at making their products than the people working on the other types of car? And if this difference in ratings reflects a real difference in the ability of the cars to fulfill their brief then we must ask TG about what can be done to improve life for MPV buyers. The ‘best in the class’ MPV, the Ford shown above, only gets 8 out of ten. What would it take for it to get 10/10, in Top Gear’s view? A nought-to-sixty time of under 4 seconds and a removable roof? Or more MPV-ness?

The same goes for the BMW 5-series, another ‘best in class’ which gets only 8 out 10. By all accounts the 5er is the most well-balanced saloon on sale; capable, competent, durable, agile, spacious and more than fast enough. How good would that car have to be to get the 10 stars that the  Ferrari La Ferrari gets? Bear in mind that the Ferrari La Ferrari carries two people, little luggage, costs four times more and consumes vastly more fuel to cover the same ground as the BMW. By what measure is that actually a better car?

You could argue that sports cars get more resources so the results are better. However, the resources expended are proportional to the cost of the car and, it’s likely GM, for example, spent more on the 7/10 Zafira than Ferarri’s La Ferrari. Really the factors are equal within the classes. I note that TG criticises GM for the price of the Zafira but sense nothing amiss about the shockingly large bill Ferarri delivers for their car.

What TG needs to do is adjust their ratings  so that each class has the same spread of results. That would mean you could use their findings to accurately compare between classes. If they left the sports car results as they were there would be a similar proportion of ten out of tens in each class and the BMW 5 would score a ten out of ten. More reasonably, TG needs to adjust the outlier classes to bring them in line with the normal distribution.

Author: richard herriott

I like anchovies. I dislike post-war town planning.

8 thoughts on “How Objective Is Top Gear’s Rating System?”

  1. Numerical ratings are an arbitrary attempt to quantify subjective opinion. One man’s fish is another’s fowl; a number of publications have acknowledged this, eschewing imposing star ratings or scores in the hope that readers will actually read their reviews and thus gain some nuance.

    1. Examinations face the same problem. Yet grades are still given. One way to keep things organised to adjust the grades so they fit the normal distribution. I think TG might have taken a bit of a look at their grades. It´s absurd no city car is considered a very good city car. That´s TG prejudice.

    2. It certainly is prejudice. Even the most prosaic fayre can be superb, as I found with the three pot Fiesta I drove last year. That I keep going on about it like a broken record to anyone who will listen (and many who will not) is a testament to how good that car was. It also helped that I had the Fiesta for a fortnight; I do not think that many reviews are conducted over a long enough period for the car’s merits to be properly adjudged, or a relativistic score to be given.

  2. Something else might be at play here too, Richard. I do not doubt that TG, as well as other outlets might be biased towards sports cars, surely the editors think that’s what the readership wants to read. But there is also the problem of finding people with enough experience, education and integrity to be able to review and asess a non-exiting yet perfectly good Supermini or MPV , from the point of view of it’s intended purpose. Singing praises to the latest from Maranello or Zuffenhausen is surely easier

  3. The disparity that you point out between the scores assigned to MPVs and sports cars betrays their heavily experiential nature. Reviewers make their assessments in something of a bubble, without applying much in the way of rigour. Often they fail to ask the most basic questions. What do people buying these cars actually need from them? Is it fit for purpose? Is it well built? Is it good to drive? What unusual features does it have? How does it compare to rivals, both obvious and tangential? Is it good value? But no, instead they place too much emphasis on how the vehicle makes them “feel”. In this regard, compared to the latest Porsche Boxster, a Citroen Picasso will always be found wanting. But a car is hardly ever bought purely on the basis of what you actually want.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s