The professor who taught the world the art of sampling
In the summer of 1946, at the ‘nuclear’ session of the United Nations Statistical Commission (UNSC), a representative of a British colony made an impassioned plea for laying down globally accepted standards for conducting large-scale sample surveys. Prasanta Chandra Mahalanobis argued that household surveys would become invaluable data sources for many developing countries that were not fortunate enough to have the kind of rich administrative datasets that advanced economies boasted.
Mahalanobis’ suggestion was accepted, and given his unique experience in conducting such surveys at the Indian Statistical Institute (ISI), he was asked to chair the first sub-committee of UNSC on sampling. The global manual on sampling which the world would follow for the next few decades owed a large debt to Mahalanobis’ and his colleagues work at ISI. ISI’S innovations - such as using a pilot survey to test a survey, and the use of replicating samples to ascertain the magnitude of different kinds of errors in surveying came to be adopted by countries across the world.
Mahalanobis’ contributions to the global statistical body were wide-ranging, and went beyond standardizing sampling norms. For instance, when the UNSC set up a committee to prepare a manual for computing national accounts (through which a country’s gross domestic product or GDP is calculated), Mahalanobis’ protege and a national income expert, Moni Mohan Mukherjee, was asked to join the taskforce set up for that purpose. Mahalanobis would also go on to chair the 8th and 9th sessions of the UNSC in the 1950s, and attended all its sessions till his death. When he passed away in 1972, UNSC’S condolence note said that it had lost its ‘doyen’.
Mahalanobis’ interventions on the global stage were not designed to appropriate the tag of ‘Vishwa Guru’ for himself or for his institute. It was motivated by a desire to fill the yawning data gaps that India and much of the developing world faced, without blindly copying the statistical norms and practices followed in the West. Ronald Fisher - arguably the greatest statistician of the twentieth century - once said that what impressed him most about Mahalanobis’ work was that it was not “imitative”.
Like most intellectuals of that era, Mahalanobis had a profound faith in planning as an instrument of growth and development. Good plans needed good data, and that motivated him to set up a number of datagathering institutions in the first few years after India’s independence. The national income unit of the finance ministry, headed by Mukherjee was hived off into a separate department called the Central Statistical Organization (CSO). To scale up ISI’S early surveys and fill the data holes in computing national income, the National Sample Survey (NSS) system was established, providing a model for the rest of the world. The Annual Survey of Industries (ASI) was set up to collect regular data on industrial activity.
Mahalanobis’ ideas on planning have not aged well. But his statistical interventions have stood the test of time. The manner in which India and much of the world collect data is shaped by the foundational ideas of Mahalanobis. Where Mahalanobis (and India) led, the rest of the world followed, the Nobelwinning economist Angus Deaton wrote in a 2005 article. Mahalanobis had a few things going for him. Born in a prominent Bengali Brahmo family, Mahalanobis’ social network was an important asset. He was very close to Rabindranath Tagore, at whose house he would first get to spend long hours with Jawaharlal Nehru. As chairman of the Congress’ Planning Committee, Nehru was worried about data gaps much before India won independence, and sought Mahalanobis’ counsel on ways to fill such gaps.
The unstinting support of Chintaman Dwarkanath Deshmukh was another important factor. As a civil servant and central banker, Deshmukh helped Mahalanobis access government funds to tide over the recurrent financial crises that the ISI faced in its early years. Later, as finance minister of independent India, it was Deshmukh who helped fund Mahalanobis’ ambitious survey plans. Deshmukh tutored the professor on ways to deal with the labyrinthine British bureaucracy and later its Indianized version, Mahalanobis’ biographer Ashok Rudra wrote.
Despite these advantages, it was never an easy ride for
Mahalanobis. Rudra documented a life-long struggle to stabilize ISI’S finances. As late as 1954, Mahalanobis felt frustrated enough to consider resigning as ISI’S director. “My struggles have been mostly against a machine which is impersonal, and incapable of responding to changing needs,” Mahalanobis wrote in a letter to Deshmukh (then the finance minister as well as ISI’S president) explaining how red-tapism was throttling ISI’S work and why he felt compelled to resign. In his recent book, Planning Democracy, the historian Nikhil Menon documents Mahalanobis’ epic struggle to obtain computers to handle the ‘big data’ of his era. Mahalanobis realized that without computers, NSS data could never be tabulated in time. He moved heaven and earth to get them. The first digital computer in the country was installed at ISI in 1956 to process NSS data. Soon it was processing data for other scientific institutions and for the Planning Commission.
As we celebrate the 75th anniversary of India’s independence, the statistical achievements of a poor country in the early post-independent years should make us genuinely proud. But we must also ask why India’s official statistical system no longer commands the respect it did in the early years of the republic.
This is the first of a five-part series on the founding fathers of India’s once-renowned statistical system.