Lies, Damned Lies Through Statistics: TOI’s Modi@2 Survey

A survey with a skewed sample can show an incorrect picture.

There are two kinds of statistics, the kind you look up and the kind you make up. 

Rex Stout, American Novelist

A catchy headline is a difficult thing to achieve, especially if it’s telling you something you already know. So “Narendra Modi still somewhat popular among youth in metropolitan cities” won’t quite make it to the front page of the largest English language newspaper in the world. After all, the prime minister’s popularity with urban youth has been common knowledge since before the 2014 general elections.

This morning though, the Times of India splashed the results of a survey with a questionable methodology – based only on a sample of urban youth – on its front page. IPSOS, a globally-recognised market research firm, conducted the survey to gauge how India feels about the PM and his government two years in. But who they questioned, and where, was bound to give them a particular result.

A Sample, Small and Skewed

The headline in the Times of India claims that “After 2 years, 62% give Modi Thumbs up”. This conclusion was based on interviews with just 1,348 people. India’s current population is approximately 1,251,695,584. That’s a 0.00010769391 percent sample size.

To make matters worse, the survey was conducted only among voters aged between 18-45, living in the eight largest cities in the country viz. Delhi, Mumbai, Kolkata, Chennai, Bengaluru, Hyderabad, Pune and Ahmedabad. Far from being a representative sample of the country as a whole, this sample does not even adequately represent urban voters, according to Professor Sanjay Kumar, director of the Centre for the Study of Developing Societies (CSDS).

The sample size should certainly have been bigger. And why survey only 18-45 year old voters? Why have an upper limit at all?  

Professor Sanjay Kumar, Director, CSDS

Back in 2014, right after the BJP’s massive victory in general elections, Professor Kumar had written an article in The Hindu pointing out how the shift among young voters to the BJP contributed to its win.

Young voters BJP
Create your own infographics

So young voters in urban areas have been fans of the PM for some time now. The ToI-IPSOS survey though, doesn’t even measure the urban vote comprehensively. Small sample size aside, urban India is spread far beyond the eight biggest cities. There are small cities, towns and even kasbahs that form urban conglomerations.

It is far too imaginative a leap to draw national conclusions from a sample size this small and skewed. At best, we can say it represents the sentiment of a small subset of urban India.

Professor Sanjay Kumar, Director, CSDS

The Ultimate Survey Tells a Different Story

Is there any way of actually gauging voter sentiment? Yes, it’s called an election and luckily for us, we have quite a few of them in India. Like any political party, the BJP has had ups and downs. In the 2015 Delhi elections, the BJP’s vote share was 32 percent and they won just three seats out of 70 – the rest went to Arvind Kejriwal and the Aam Aadmi Party. In the Bihar election, the BJP had just 24 percent of the vote despite a high-decibel campaign by PM Modi.

In the most recent round of Assembly elections, the BJP won no seats in Kolkata or Chennai. Both cities were surveyed in the ToI-IPSOS poll. There are two possible explanations for the discrepancy: Either the ‘sentiment’ measured by the survey has no bearing on voting, or the survey itself is flawed.

Narendra Modi may well be the most popular leader in India. This survey, however, cannot be used to draw that conclusion.