What Is India Doing With All Its Big Data?

Despite progress in using Big Data in other sectors, India’s development sector is yet to benefit substantially.

Showing the growth of big data’s primary characteristics of volume, velocity, and variety. Image used for representational purposes.

(Photo: Wikimedia Commons)

The data revolution is here to stay. Big Data, Artificial Intelligence, and Machine Learning are constantly changing the way data is processed and used for business decisions. While the revolution started in the industrialised world, India is not far behind.

With a growing proportion of the population using digital platforms, and the government’s efforts in the recent past to digitise vast amounts of data for various programs, India has become one of the largest producers of Big Data in the world. The use of Big Data is also not new in India. The Indian Government has been using Big Data and data mining techniques for quite some time now.

The use of data mining techniques as a part of Project Insight, to identify tax evaders, and the use of geo-tagging to zero-in on shell companies, are excellent examples of India tapping in on the potential of the data revolution. Despite the progress in using Big Data in other sectors, the development sector in India is yet to benefit substantially from the Big Data revolution.

Use of Big Data for Development

‘Big data for development’ refers to the use of Big Data for planning and policy-making for development programs. Unlike traditional sources of data used in the development sector, Big Data does not depend on large-scale surveys which are highly resource-intensive and take a long time to be collected, processed and turned into meaningful information, relevant for effective policy-making. On the contrary, Big Data gets generated digitally, is passively produced, automatically-collected, and can be analysed in real-time.

Big Data and Artificial Intelligence are slowly gaining pace to impact the development sector in India. NITI Aayog is partnering with IBM to develop models to predict crop yield using Artificial Intelligence. 

Such predictive models are especially targeted to improve the yields of marginal farmers. Some of the state governments are also taking steps to enhance the use of Big Data and Artificial Intelligence. The use of Artificial Intelligence during the Kumbh Mela this year was especially notable, where smart video analytics was used for efficient traffic movement, security, and crowd management in Prayagraj. The Government of Odisha is also preparing to use Artificial Intelligence, Data Analytics, and Mobile Computing to analyse crime data and support investigation.

Big Data & Its Achievements

While the progress is noteworthy, there is much more that can be done to support the development sector in India. The UN Global Pulse, a flagship initiative of the UN General Secretary on Big Data, is paving the way for the use of Big Data for development.

Some of its innovations include the use of mobile phone data and data extracted from mobile ‘top-ups’ to predict food consumption at a household level, and the use of telecom data generated from mobile phones have been used to map the outbreak of disease.

Big data and analytics can also be used to predict weather conditions for the energy sector as well. IBM has been paving the way in this sector using advanced cloud imaging technologies and sky facing cameras to track cloud movements. These technologies are complemented by advanced sensors on wind turbines to monitor wind-speed, direction, and temperature. Such technologies have helped in increasing the efficiency of renewable sources of energy by up to 10 percent.

Further, social media-supported ‘sentiment analysis’ has been used in the past to monitor the climate change discourse.

Such monitoring can not only help assess public interest and awareness, but also support in developing strategies to drive public engagement.

Challenges Plaguing Big Data Sector

While India is taking steps to use Big Data in commercial and industrial applications, it is important to ensure that the development sector is not left untouched by the data revolution.

One of the major challenges in the sector to enable the use of Big Data is the availability of quality data.

Most of the data in the development sector is yet to be digitised. Take for example, the records being maintained in Public Health facilities in the country, which first needs to be digitised to be used for advanced analysis. Data available from the numerous large-scale surveys conducted in the development sector often do not speak to each other and are riddled with data quality issues, rendering them not usable for Machine Learning.

Apart from the quality of data, the use of Big Data in India is also limited by a serious dearth of data scientists available to serve the growing need of the sector.

According to one report, of all the professionals working in the big data and analytics domain, less than 10 are in India. The ‘big data management policy’ introduced by the Comptroller and Auditor General of India (CAG) is an important step in this direction. However, the effects of the policy on the development sector remains to be seen.

(Arnab Dey is Assistant Vice President–Research, Sambodhi Research & Communications. This is an opinion piece and the views expressed are the author’s own. The Quint neither endorses nor is responsible for them.)