Trial and error with digital technology in Sri Lanka
The success of the Asian Development Bank’s (ADB) relatively new results-based lending instrument hinges on accurate validation of outcome and output indicators. But this process can delay the programme if it is not conducted expediently.
Maybe digital technology can help to accelerate verification, I surmised when I was working on the Skills Sector Enhancement Programme (SSEP) in Sri Lanka.
The goal was to improve the employability of Sri Lankan youth that graduated from government technical and vocational education (TVET) programmes supported by ADB. More specifically, the target was to raise the employment rate of TVET graduates to 52 percent by 2016. A tracer study is the conventional approach to assess its effectiveness, but the employment rate might be inflated if teachers ask students about their job status.
To prevent systematic positive reporting bias, I conducted interactive voice response (IVR) surveys. I was particularly intrigued about this digital technology, which uses automated phone calls to reach out to people. The IVR technology is incredibly fast and affordable compared to manual telephone interviews.
Digital technology faces with social norms
Another advantage of IVR is its simplicity. The process requires questionnaire recording and its uploading to a special website with TVET graduates’ mobile phone numbers. In this case, I am interested in the employment rate, so I record the relevant questions, such as Are you currently employed? If yes, press 1; if no, press 2 in the country’s two official languages, Sinhala and Tamil. SMS was also used for prior notification so that the respondents knew when the IVR surveys were initiated. The IVR surveys were rolled out typically on evenings or weekends to improve the response rate.
It seemed like all the ideal conditions were in place for piloting IVR surveys under the SSEP. Mobile penetration in Sri Lanka reached 115 percent in 2015 and young people are more adept at using mobile phones than senior citizens. The IVR surveys had to be short (three to five minutes) to obtain a good response rate, but asking about employment status did not require many questions I was very excited to find out if this digital technology would work for us.
Unfortunately, it did not. The IVR surveys were not a complete fiasco, but certainly not up to the mark to make a good judgment on outcome validation. Why? Among several reasons, let me share three social barriers that caught my attention.
First, family members often share the same mobile phone number, so the target respondent is not necessarily the one to pick up. Second, the respondents fear they will be charged if they make key presses with a call from an unknown number, so many do not pick up. Third, some respondents shared a social preference to use landlines for official surveys. Yet another challenge was that many mobile phone numbers registered in the database were not necessarily correct, in some cases the person who picked up the call was not even a family member but the trusted village leader.
Trial and error works
I addressed the shortcomings of IVR surveys by deploying another form of digital data collection: the computerassisted telephone interview (CATI). CATI is a manual telephone interview conducted by human enumerators with data input digitized through a tablet. CATI surveys were carried out in parallel with IVR surveys to also cover other topics, such as the socioeconomic background of graduates and quality of courses, to better understand the broader TVET system in Sri Lanka.
This technology is expensive and you have to train enumerators and procure tablets. The upside is that enumerators can ask longer questions and responses are recorded automatically when the enumerators input them. After each survey is completed, the responses are instantly transmitted to a cloud server, significantly reducing the risk of data transcription error encountered in paperbased surveys. CATI can also solve the social barriers encountered through IVR surveys by flexibly addressing the respondents’ concerns.
Without CATI, I could not have completed the verification on schedule.
Fortunately, the assessment confirmed that the employment rate for TVET graduates surpassed the 52 percent target, so the government received US $ 5 million through the SSEP. Trial and error, I learned, is an inevitable process to harness the power of the digital revolution. (Ryotaro Hayashi is a Social Sector Specialist, South Asia Regional Department at the Asian Development Bank)