June, PC Mahalanobis, and the 'Sanctity of Data': National Statistics Day 2023

The greatest tribute that the government can pay to Mahalanobis is by kick-starting the 2021 Census process.

In the 1930s, Mahalanobis was responsible for setting up the Indian Statistical Institute (ISI) when he was teaching at the Presidency College in then Calcutta.

Image: Chetan Bhakuni / The Quint

(This is part four of a four-part 'June' series that revisited significant historical events or policies and how the lessons learned from them continue to be of relevance in present-day politics and society. Read part one here, part two here, and part three here.)

There was something about the land of Bengal in those days; the kind of artists, authors, scientists, freedom fighters, and thought leaders it threw up as India moved along towards freedom from British Imperial rule. Soon after India became independent in 1947, three uniquely diverse and different personalities emerging from Bengal created institutions that, warts and all, stand rock solid even in the 21^st century.

The first was Shyama Prasad Mukherjee, a politician who founded the Bhartiya Jan Sangh, which now goes by the name of Bharatiya Janata Party (BJP) and dominates large swathes of Indian polity. The second was Sukumar Sen, a quintessential bureaucrat who supervised and oversaw the 1952 and 1957 Lok Sabha and assembly elections as independent India’s first Chief Election Commissioner. Using meagre resources and technology, Sen set a gold standard for conducting elections that is still a marvel of modern democracy.

The third personality is less talked about even though the institutions he set up and the legacy that he has left behind are equally important; and even more relevant in contemporary India. The authors are talking about PC Mahalanobis, the man who taught India the value, sanctity, and importance of credible statistics, numbers, and data.

Major Contributions of PC Mahalanobis

In contemporary India and the world, data is indeed dirty. When media professionals use data, their commentary becomes more credible. When academicians and researchers use data, their body of work is accorded more respect. When civil society groups use data their voices, and impassioned appeals become more resonant. When opposition parties and leaders use data to attack the ruling regime, data adds heft to their argument. And when policymakers use data in a smart manner, welfare schemes deliver miracles for the poor. Data is the most powerful tool for policymakers.

This simple fact was realised by Mahalanobis almost a century ago when India was steeped in rhetoric. Delving deep into his early career as a statistical scientist can be kept for another time but his pioneering work in harnessing data to study agriculture and crop patterns is worth mentioning. But he blossomed around the time India was getting independence.

Two of his contributions to the architecture and system of data collection and analysis in India have stood the test of time.

The first and most significant was the use of random sampling to collect data that could represent the entire country. Everyone talks about random sampling these days. But what exactly does it mean in layman’s terms to the ordinary Indian? The Census that covers every household is a once-in-a-decade affair.

How does one get nationwide data on a more frequent basis on income, consumption, economic activity, growth rate, poverty, inequality, unemployment, consumer sentiments, and myriad other things that are critical for policy making and policy tweaks?

Enter random sampling. In the 1930s, Mahalanobis was responsible for setting up the Indian Statistical Institute (ISI) when he was teaching at the Presidency College in then Calcutta.

The techniques adopted back then by Mahalanobis and his colleagues are still used by institutions like the United Nations and the World Bank. Nobel Prize winners in economics have, and continue to laud ISI as a pioneering leader in the world of statistics.

The Marvel of Random Sampling

To come back to random sampling, what exactly does it mean? Take a country like India with an estimated population of 1.4 billion. How does one find out how many Indians of working age are unemployed? How does one find out how many females are active in the labour force? How does one “measure” consumer sentiment? Quite obviously, it would be impossible to knock at the doors of every Indian household.

Mahalanobis and his team found a brilliant solution by finding out that a scientific process of randomly picking up a small sample of ordinary Indians and asking them well-designed and framed questions would deliver results that are representative of the entire population. Since then, random sampling has become a powerful tool for researchers and policymakers across the world.

There are thousands of examples of how data collected and analysed through random sampling has proven to be immensely insightful and valuable. The authors would take one example: “measuring” consumer sentiments. Obviously, sentiment is subjective.

So, how does one measure it, and how can asking questions to a tiny fraction of the population deliver data that is representative of the entirety of India? It does. Many organisations do sample surveys to measure consumer sentiments.

The Reserve Bank of India conducts a bimonthly Consumer Confidence Survey that covers about 6,000 urban Indians. The CVoter Foundation has been conducting a quarterly Consumer Optimism Survey on behalf of the Economic Advisory Council to the Prime Minister that covers about 10,000 rural and urban Indians. The questions asked are similar, though not identical. Astonishingly, not only do both surveys arrive at similar results, they are uncannily accurate in estimating household expenditure patterns for the whole of India despite the sample sizes being so small.

We all owe a debt of gratitude to PC Mahalanobis for this.

Random sampling and sample surveys of the kind dreamt of and designed by Mahalanobis took off after India became independent in 1947. He was appointed statistical advisor to the Government of India in 1949 and went on to become a member of the Planning Commission of India. Together, the ISI and the Pune-based Gokhale Institute of Politics & Economics (the coauthor is an alma mater) established the National Sample Survey in 1950. The effort was led by Mahalanobis.

Dent On Data Credibility In India Today

In 1951, the Central Statistical Organisation (CSO) was set up; once again the fort was led by Mahalanobis. Since then, the National Sample Survey (NSS) and the CSO have been the bedrock of random sampling, data collection, and analysis of virtually every socioeconomic issue of relevance for more than seven decades.

In recent times, the aura has frayed a bit due to needless controversies. For instance, the current regime refused to release the National Sample Survey Office (NSSO) survey results of 2016-17 on unemployment. Without going into gory detail, that controversy did dent the credibility of the data and statistical edifice built by the likes of Mahalanobis.

The lead author has been dealing with random sampling for about 30 years now. While researching this piece, he realised how widespread the influence of Mahalanobis has been on diverse disciplines. For instance, once random sampling became accepted practice in India, another pioneer named Eric da Costa established the Indian Institute of Public Opinion in the 1950s to conduct what has now become commonly known as opinion polls. Back in those days, nobody in India conducted opinion polls, either related to elections or any other topic. Da Costa was the first to use random sampling techniques to do election-related opinion polls in the 1950s and 1960s.

The baton was picked up from the pioneer by another pioneer, Dr Pronoy Roy, who conducted opinion polls on behalf of India Today and set the gold standard for the industry. The lead author was so inspired by Dr Roy’s coverage of the historic 1984 elections that his future career as an opinion poll specialist was set that very time. Today, opinion polls are an essential and integral part of the democratic process in India. Apart from Da Costa and Dr Roy, credit must eventually go to Mahalanobis.

Every year, India celebrates 29 June as National Statistics Day in honour of PC Mahalanobis. In the opinion of the authors, the greatest tribute that the government can pay to the impressive legacy of Mahalanobis is to immediately kick-start the 2021 Census process.

The COVID 19 pandemic did delay the Census. But we are already in the middle of 2023 and the 2021 Census is yet to begin. That is gross injustice not just to the legacy of Mahalanobis, but also to the sanctity of data.     

(Yashwant Deshmukh & Sutanu Guru work with CVoter Foundation. This is an opinion piece and the views expressed are the author’s own. The Quint neither endorses nor is responsible for them.)