(This article is part of Paper Heist, The Quint's special project bringing out the inside story of India's paper leak scandal. Please support our coverage( and help us do more such deep-dive stories.)
Even though the National Testing Agency (NTA) submitted its response in the Supreme Court on Wednesday, 10 July over re-conducting the NEET-UG exam due to alleged malpractices, their affidavit has still failed to address some key issues.
In the last hearing, on 8 July, the top court had asked the NTA and the Union government to make a “full disclosure” on the nature of paper leak, the way the paper was disseminated on social media, whether the breach was at a systemic level and if those candidates who got benefitted from unfair means can be identified and segregated.
Chief Justice of India DY Chandrachud reportedly observed, “One thing which is very clear is that the leak has taken place. That the sanctity of the exam has been breached is beyond doubt. The question is how widespread is the leak.”
Since the Central Bureau of Investigation (CBI) is now probing allegations of paper leak in NEET-UG 2024 exam, the top court also asked the central body to submit a status report.
Even as the next hearing has been posted for next Thursday, The Quint takes a closer look at the NTA’s response, and the lack thereof, to the allegations raised by the petitioners over irregularities and malpractices in the NEET medical entrance exam.
'IIT Madras Report's Data Skewed, Inconclusive'
In order to ascertain if the cases of malpractices and paper leaks were localised incidents or more widespread, the Supreme Court had asked NTA if it was possible to identify suspicious cases using data analytics by an expert government agency.
The NTA gave this task to the Indian Institute of Technology (IIT) Madras, which submitted a report using parameters such as marks distribution, city wise and centre wise rank distribution and candidates spread over marks range. It is to be noted here that three Directors of IITs, in their ex-officio capacity, are members of the NTA's Governing Body.
Through their study, IIT Madras concluded that since the marks distribution follows a bell-shaped curve, there is no abnormality. “There is neither any indication of mass malpractice nor a localised set of candidates being benefitted, leading to abnormal scores,” it claimed.
However, on analysing the data shared by the institute, the following questions emerge:
1. What is the correlation between the number of toppers from a city and malpractices reported from a city?
The IIT Madras report noted that "only two” candidates scored the top-100 ranks from Patna (Bihar), where there were allegations of paper leak and that similar trends were seen while analysing the top-500, top-1000 and top-5000 ranks.
“The candidates who allegedly had access to the question paper a night before need not necessarily score 700/720. What needs to be analysed is if there are any students with unusually high NEET scores not in line with their performance in their class 12 board exams,” a student petitioner told The Quint on the condition of anonymity.
During the last hearing, the top court too had asked NTA to probe this “red flag” as the petitioners have highlighted the issue of inflated ranks.
"The assumption linking quantitative data set (number of top-100 rank scorers) to qualitative data (if there was malpractice or not) seems flawed. This assumption needs to be substantiated with more data and other variables," Shankar Prakash, and independent researcher told The Quint.
2. Can the correlation between reduction in syllabus and increase in candidates scoring more than 600 marks be quantified?
The report prepared by IIT Madras shows that the number of students scoring 700-720 marks has increased from 350 in 2023 to 2,321 in 2024 – an increase of 563%.
Similarly, as per the report, the number of students scoring 650-699 marks has increased from 6,939 in 2023 to 27,885 in 2024 – an increase of 301%.
%202024%20RESULTS%20(9).jpg?auto=format%2Ccompress&fmt=webp&width=720)
The institute has attributed the “overall increase in marks obtained by students in the range of 550-720 marks” to 25 percent reduction in syllabus, as the NTA had mentioned in their FAQs released a week after students had flagged irregularities in the NEET-UG result.
“How many questions from the deleted syllabus were asked in the previous NEET exam can indicate the actual difference in marks,” Prakash said.
The court too had noted the “unprecedented high number of students” scoring 720/720 marks. (More on this later)
"Again the report should not make qualitative comments on quantitative data and state a conclusion that they did not arrive at," Prakash asserted.
3. What about the exam centre in Haryana’s Jhajjar, where six students with reportedly consecutive roll numbers had scored 720/720 marks?
In its report, IIT Madras stated that there is no spike in the “number of candidates securing high ranks in a particular centre in 2024 as compared to 2023, which clearly indicates there is no localised benefit due to malpractice.”
After NEET results were declared, it was widely reported that six candidates from the same centre in Bahadurgarh, in Jhajjar district of Haryana had scored full marks. NTA had responded to this through their FAQs and attributed it to grace marks given to 1,563 candidates because of loss of time. However, during the course of the hearing, NTA withdrew grace marks and ordered a re-test for these 1,563 candidates.
Hence the ‘Centre-Wise Ranks Data’ analysed by the institute does not include the Haryana centre as those six candidates didn’t get full marks in the re-test. In fact, no centre from Haryana has candidates scoring top-60,000 ranks.
"Further, the data on Patna is not very helpful if the alleged paper leak is localised. Clusters within cities should be looked at to identify anomalies," the Data Sciences researcher said.
4. On what basis has IIT-Madras picked the data?
For the report, IIT Madras has picked the marks of candidates after the re-test was conducted. “If the scores were considered before the re-test, i.e. inclusive of grace marks awarded by NTA, the ranks would have been more inflated,” Dr. Vivek Pandey, an RTI activist and petitioner in the case, claimed.
Again, on what basis were top-5% ranks made the barometer for indicating malpractice, in case there was a spike?
Shankar explained that the report assumes that beneficiaries of malpractices will score higher ranks, it does not rule out malpractices among candidates with low ranks.
Besides, though the standalone Bell Curve in the report indicates “no abnormality,” it should be compared with similar graphs drawn from the results of 2023 and 2022; shifts in the curve (if any) can then be discerned, a researcher in Data Science told The Quint, on the condition of anonymity.
Marks distribution of NEET candidates in 2024 vs 2023.
Screenshot: From NTA's response in court
“It would still appear as a bell curve if there were spikes in the 650-720 category, considering that 24 lakh candidates took the exam and the maximum number of students will score in the lower range of 60 to 200 marks,” Shankar added.
Again, the centre and city-wise rank distribution includes data of 1.4 lakh candidates, considering that there are 1.1 lakh seats available in government colleges but excludes seats in private colleges. Further, it is a subset of the total 24 lakh students who appeared for the exam and may miss patterns and irregularities occurring beyond this range.
"The present analysis is not conclusive to draw results from. More tests are needed to identify underlying behaviour before drawing inferences," the Data Sciences researcher remarked.
5. What is the methodology, limitation and chance of error in this data?
Although IIT Madras has stated that they have used Python for data processing, PostGreSQL for data storage and Metabase for analysis, they haven’t specified the methodology for conducting the study. The institute has also not specified chances and sources of error in the data if any.
"Any statistical study should clearly spelt out the limitations, otherwise, it is considered as hiding information," Prakash said.
Viewing the data as "a bit skewed," he added that even though statistically, there are no anomalies in the scores, it cannot necessarily be linked to malpractices.
What Explains Sharp Hike in Candidates Scoring Full Marks?
When NEET-UG 2024 results were first declared on 5 May, a total of 67 students had scored 720/720 marks, a spike of almost 3,200% when compared to two such candidates in 2023.
However, after the withdrawal of grace marks and a fresh test, this number has come down to 61, with six students from the Jhajjar exam centre not getting the perfect score on the second attempt.
The NTA, in its response to the Supreme Court, clarified that now there are 61 perfect scorers, of which 44 are due to revision in answer key of Physics. However, even with 61 candidates scoring full marks, the spike can’t go unnoticed.
What About the Alleged Paper Leak in Patna?
In its response, the NTA claims that the Bihar Police had handed it names of 17 candidates, who were suspected to have benefitted from the paper leak. On analysing their performance, the NTA concluded “not much impact on the conduct of the exam at the said centre” based on the performance of those candidates being "moderate."
However, this again assumes a correlation between malpractices and high scores and ignores irregularities among low-rank scorers.
Even if the performance of the candidates is considered, as per NTA's disclosure, two candidates have scored 601 and 580 marks, corresponding to a rank of 70,925 and 1,03,234 – well within the top-1.4 lakh ranks data set, used by IIT Madras.
The matter is being investigated by the CBI, which has also submitted their response to the Supreme Court. NTA has withheld the results of these 17 candidates.
How Did Candidates from Different States Take Exam in Godhra?
The information bulletin of NEET-UG 2024 states that the exam centres will be limited to the state of permanent address of state of present address.
“The candidates may select their own city or neighbouring cities in their state of residence only.”
It may be recalled that as per a report in The Indian Express, at least 16 students belonging to Odisha, Jharkhand, Maharashtra and Karnataka had chosen to take the exam from a particular centre in Gujarat’s Godhra, where malpractices were reported, and an FIR too was filed.
However, NTA had not stated the status of these students.
Based on the performance of the candidates at the said centre, NTA claimed that “there was no impact whatsoever on the sanctity of the exam,” again limiting malpractices to top rankers only. At this centre, 14 candidates have scored marks between 600 and 680.
What About the Issue of Grace Marks?
In their response, the NTA has stated that grace marks were withdrawn on the recommendations of a High-Powered Committee on 13 June and a re-test was conducted on 23 June.
However, questions still remain:
Why was the issue of compensatory marks not disclosed to the students before the results were announced on 4 June.
Why is there no mention of grace marks or the formula for their computation in the exam prospectus?
When was the High-Powered Committee formed and who all did it comprise of?
In its response filed on 10 July, the NTA has stated that there have been 153 cases of unfair means – of which, results of 81 candidates have been withheld and 54 candidates have been debarred. However, it has not mentioned the status of the remaining 18 candidates.
The NTA also stated that a total of 16 FIRs have been registered regarding NEET UG 2024, and of these 14 have been registered by exam officials at the behest of the NTA.
But this is in contrast with its response submitted on 4 July, where the NTA had stated that there were 63 cases of unfair means – of which 22 candidates were debarred, the results of 33 candidates were withheld and those of nine were declared. At that time, the NTA had said that 13 FIRs have been registered in cases of unfair means.
The Supreme Court is scheduled to hear the matter next on 18 July.
(At The Quint, we are answerable only to our audience. Play an active role in shaping our journalism by becoming a member. Because the truth is worth it.)