As I mentioned recently, I have been leafing through the Top Gear 2016 Car Buyers [sic] Guide. I made an interesting discovery.
Top Gear provides a rating for every car they review, graded as marks out of ten. I crunched the ratings for six classes of car: MPVs, city cars, hatchbacks, executive cars, sports cars and cabriolets.
In advance, what would you expect to find? A reasonable hypothesis is that the range of talent in each class of cars is the same. That means the efforts made to craft MPVs are the same as the efforts made to craft cabriolets. The efforts should have equal chances of success. In other words the chances of there being a good MPV should be the same as the chances of there being a good cabriolet.
However, while flicking through the Sports Car section I noticed that several of the 37 sports cars rated received ten out of ten. In contrast not one of the city cars got higher than eight of out of ten. What that means is that either a) city car makers are not as good at making city cars as sports car makers are at making sports cars, or b) that city cars is a dying market with less investment in new product, or c) Top Gear has a bias towards sports cars.
The statistics above show that if you offer TG a sports car or a cabriolet you have a better chance of getting 10 out of 10 than if you offer an MPV or a city car. According to TG the average rating for an MPV is 5.89 out of 10. And the average rating for a sports car is 8.43 out of ten.
The average rating of all the cars included in the survey is 6.9 out of ten. Assuming that each class of car is comprised of equally good or bad examples, you’d expect that the average value of the sub-.groups would be same as the overall average.
Think of it like this: if a restaurant critic tested 50 Italian and 50 Chinese restaurants, you’d expect the average rating to be the same in both groups, other things being equal. The results would have a normal distribution in statistispeak. If there was a variation from this, such that Italian restaurants scored markedly lower than the Chinese ones, it would make one want to look for a reason. We could look to see if there was a good supply of Italian chefs in the region, for example. We could check the supply of ingredients for Italian food. We could also query the objectivity of the restaurant reviewer.
What story can we imagine to explain why TG rates sports cars and convertibles over other types of cars? Is it really that sports car makers are better at making their products than the people working on the other types of car? And if this difference in ratings reflects a real difference in the ability of the cars to fulfill their brief then we must ask TG about what can be done to improve life for MPV buyers. The ‘best in the class’ MPV, the Ford shown above, only gets 8 out of ten. What would it take for it to get 10/10, in Top Gear’s view? A nought-to-sixty time of under 4 seconds and a removable roof? Or more MPV-ness?
The same goes for the BMW 5-series, another ‘best in class’ which gets only 8 out 10. By all accounts the 5er is the most well-balanced saloon on sale; capable, competent, durable, agile, spacious and more than fast enough. How good would that car have to be to get the 10 stars that the Ferrari La Ferrari gets? Bear in mind that the Ferrari La Ferrari carries two people, little luggage, costs four times more and consumes vastly more fuel to cover the same ground as the BMW. By what measure is that actually a better car?
You could argue that sports cars get more resources so the results are better. However, the resources expended are proportional to the cost of the car and, it’s likely GM, for example, spent more on the 7/10 Zafira than Ferarri’s La Ferrari. Really the factors are equal within the classes. I note that TG criticises GM for the price of the Zafira but sense nothing amiss about the shockingly large bill Ferarri delivers for their car.
What TG needs to do is adjust their ratings so that each class has the same spread of results. That would mean you could use their findings to accurately compare between classes. If they left the sports car results as they were there would be a similar proportion of ten out of tens in each class and the BMW 5 would score a ten out of ten. More reasonably, TG needs to adjust the outlier classes to bring them in line with the normal distribution.