Varanasi cleaner than Bhubaneswar? Swachh Survekshan needs to come clean on city ranking

How could Indore be India’s cleanest city? Is Varanasi really cleaner than Bhubaneswar? Why is Leh seen to be dirtier than Gangtok? What is keeping Mumbai and Navi Mumbai 20 ranks apart on cleanliness scores? These questions have come up since the government released the Swachh Survekshan 2017 – a survey of cleanliness across 434 cities, intended to foster competition among urban bodies and play a catalyst for Prime Minister Narendra Modi’s flagship Swachh Bharat mission. The findings of the survey made for interesting headlines and infographics for news publications, but the puzzle remains.

While the parameters of cleanliness chosen for the survey seem adequate and its assessment that Swachh Bharat Mission is succeeding in making urban India cleaner might be correct, the methodology used to rank cities appears flawed.

The survey rated cities on five parameters – sweeping, collection and transportation of solid waste, processing and disposal of solid waste, construction of toilets to stop open defecation, capacity building and lastly, communication and awareness campaigns to change behavior. Municipal authorities from cities covered by the survey provided data on the level of their preparedness on all these parameters. Surveyors were deployed to verify the municipal administration’s claims and make independent assessment. Citizen feedback was taken through phone calls, social media and digital apps.

The survey assigned 45% weight to self-declaration by municipalities, 25% to independent assessment by surveyors and 30% to citizen feedback.

The problem with such a methodology is that it leans heavily on declarations made by municipalities with respect to management of solid waste, community and public toilets and efforts undertaken to raise awareness and effective engagement of sanitation works. There is neither any penalty for furnishing inaccurate information nor a credible system to check the claims municipalities make. A poll earlier this year by an independent citizen engagement platform, www.localcircles.com, showed 75% of residents in 200 cities didn’t see their municipal bodies fully engaged in the Swachh Bharat mission. The surveyors, 421 of them, took less than six weeks to give their independent assessment. In such a short time, they managed to inspect 17,500 locations, 2,582 community and public toilets, 2,560 residential areas and 2,560 commercial hubs across 434 cities. We do not have information about their qualifications or methods they deployed to make their assessment. There have been news reports of some of them being caught for allegedly taking bribes from municipal officials.

Mistaking relative outcomes for absolute values:

The most striking drawback of the survey, however, has been that it mixes up absolute values with relative assessments. Let’s imagine a city, X, where 90% of its residents have access to toilets and no toilets were built in the past year. Now, let’s take another city, Y, where the same share increased from 60% last year to 70% this year because it built new toilets. City Y still has lower coverage of toilets than City X, but the survey would rank it higher on the open defecation parameter.

Similarly, residents were asked if cleanliness, hygiene and sanitation in their city had improved from last year. The questionnaire is designed such that it does not tell us “how much cleaner” a city has become. It may happen that the city is getting cleaner every year, but never getting clean enough. That, perhaps, explains why Varanasi, at 32, ranks higher than Bhubaneswar, which is placed at 94, or how Chandigarh slipped to 11th position from the second place it had won in the previous edition of Swachh Survekshan. Chandigarh and Indore are 106 marks apart in overall cleanliness scores, primarily because the northern city didn’t get a thumbs up from its residents for its cleanliness effort in the past year. On citizen feedback, Punjab scored 379 marks, 118 short of Indore’s score of 497.

Varanasi cleaner than Bhubaneswar?

Varanasi has seen much action by civic authorities and NGOs in recent times because it happens to the prime minister’s constituency. The holy city has surely got cleaner than what it was one or two years ago, but few would agree that Bhubaneswar is dirty enough to be 260 marks behind Varanasi. Even PM Modi, who chose to make the capital of Odisha the venue for his party’s executive meeting last month, would not buy this. Varanasi also scored much higher marks on account of solid waste management not necessarily because it does a better job in that segment, but for the facilities it has. Varanasi has access to a waste processing plant and operates it at more than 80% efficiency, while Bhubaneswar has none. As a result, the former gets to tick 7 out of the 10 boxes for solid waste management, while the latter has to settle for five.

Bhubaneswar also did poorly on citizen feedback, with a score of 303 out of 600, while Varanasi scored 388 marks on the same count. Although population of both cities are comparable, Varanasi saw a much higher number of its residents, 32,054, participating in the survey than Bhubaneswar where only 13,084 responded. If the higher participation is a result of the efforts undertaken by the advocacy groups, it is possible that the feedback from residents would have a positive bias.

Moreover, when citizen feedback is sourced mostly through digital media and phone calls and is therefore vulnerable to a higher degree of manipulation compared to a situation where one leans more on in-person interviews. Such a risk has already been acknowledged by officials associated with Swachh Survekshan. Also, it is unlikely that people in the lower social and economic strata, who bear the brunt of an unclean city, would have had representative participation in the survey.

Another weakness of the survey lies in the typology of cities. Different types of cities will have different standards and challenges. For example a town of 20,000 people is a vastly different entity than a metro like Mumbai. Similarly a temple town such as Tirupati should be considered separately from an industrial city such as Coimbatore or Sivakashi. Faridabad, a favoured destination for migrant workers, will have challenges very different from that of Kanpur that once was a manufacturing hub. Finally the base value should be taken as a determinant to measure change. A newer town like Navi Mumbai would have started from a higher cleanliness base level than its famous neighbor, Mumbai. And finally we need to set and measure against benchmarks, for different typologies.

These are complex issues, but critical to establishing the credibility of the survey, which is “intended to foster a spirit of healthy competition among towns and cities.” Swachh Surveskhan will fall short of that expectation if the shortcomings in its methodology are not addressed.

Rajesh Mahapatra is chief content officer, Hindustan Times.

He tweets at @RajeshMahapatra

(The copy has been updated to reflect the correct weightage values assigned to survey parameters which appeared in an abridged version of the column in the Hindustan Times on May 7)