White Paper The Effect of Survey Length of Data Quality: An Asia Pacific Perspective By Jennifer Serrano, Knowledge Manager APAC, SSI Singapore July 2015 © Survey Sampling International, 2015
ABOUT THE AUTHOR Jennifer Serrano, Knowledge Manager APAC, SSI Singapore Jennifer has over nine years of research experience in three countries – Philippines, Malaysia and Singapore. Prior to joining SSI, she worked with HB&A Research International, Nielsen and Added Value Saffron Hill. At Nielsen she managed large-­‐scale tracking studies in the retail industry and worked with the Measurement Science and Retail Optimization teams to monitor data quality and implement improvements in protocols and processes. Jennifer also has experience in tracking customized studies with both private and public sectors in Singapore. Her private sector experience centers on advertising and campaign evaluations as well as new product innovation research, while her public sector experience has been primarily around market understanding and customer engagement. In her personal life, Jennifer is an avid traveller, particularly when it comes to looking for new experiences within Asia. Offices worldwide | [email protected] | surveysampling.com | 2 INTRODUCTION All investigators are interested in high quality data. And in order to get high quality research data, we need people to pay attention throughout the survey. But what happens when people get tired or bored while answering our questions? It is not uncommon to see surveys that exceed 25 minutes. Some even take as long as 40 minutes. And respondents will answer 40 minutes of questions. But the question is – will we get the same quality of data? To understand human behaviour – and survey taking is a human behaviour – it is often necessary to step outside our own discipline. We in market research tend to fondly imagine a respondent being fully rational in their answering and having the answers to our questions immediately at hand, however they are formulated. The reality could not be further from this. Herbert Simon, an American scientist and Nobel-­‐laureate, first coined the term “satisficing”. He used it to explain decision-­‐making behaviour under conditions where an optimal solution cannot be obtained, and yet a decision must be made. He called this type of rational decision making “bounded rationality”. This way of looking at human behaviour is not far from Jon Krosnick’s statistical survey theory of satisficing which says that optimal question answering by a survey respondent involves a great deal of cognitive work, and some people will “satisfice” to reduce that burden. The likelihood of satisficing is linked to respondent ability, respondent motivation and task difficulty. He further illustrates that some people may shortcut their cognitive processes in two ways: 1. Weak satisficing: Respondent executes all cognitive steps involved in optimizing, but less completely and with bias 2. Strong satisficing: Respondent offers responses that will seem reasonable to the interviewer without any memory search or information integration. Our study then asks itself – to what extent does the length of the survey add to the burden of the survey, affecting motivation and the task difficulty? The impact of survey length on data quality is a topic that SSI has been tracking across several years. The first study in 2004 covered the UK, France and the Netherlands and won the award for best paper at the 2005 ESOMAR Panel Conference. This study was revisited in 2009, and we likewise found that respondents get fatigued as the interview progresses and produce poor quality data towards the end of the survey. RESEARCH DESIGN In 2015, we used the same approach in examining the impact of survey length on fatigue and response quality in the Asia Pacific market. We designed two surveys: a long and a short version (an extract of the long survey), and each survey consisted of four sets or blocks of questions on different topics. In order to see the point where respondents get cognitively fatigued and exhibit satisficing behavior, the blocks were randomized for each respondent, and their response patterns were examined in relation to position of the blocks. Offices worldwide | [email protected] | surveysampling.com | 3 The study looked at different measures to examine data quality: ü
Dropout rates ü
Average interview and block length ü
Satisficing behavior trap questions o
Instrument non-­‐use o
Qualification question o
Open-­‐ended question DROPOUT RATES People tend to look at dropout rate as it is somehow related to data quality, expecting longer surveys to have higher dropout rate. But there is no clear relationship between interview length and dropout rate. Table 1 shows that overall dropout rate is almost the same for both the long and the short surveys with no discernible difference across countries, except for an atypical dropout rate in China. Table 1 – Dropout rate Singapore China India Average Long Short Long Short Long Short Long Short Dropout Rate 2015 11% 18% 48% 41% 10% 9% 27% 26% While there is generally a slight relationship between dropout rate and interview length as seen in the average dropout rates within the project over time (in Chart 1), there is no way to predict what the dropout rate will be on any particular survey. If we look at China specifically, the average dropout rate is actually 30% for a 20-­‐minute survey, but can vary very highly up to 40%. This is because there is no strong relationship between the dropout rate and interview length, and the variance is high -­‐ one can run the same survey twice and get a completely different dropout rate, as happened when we re-­‐ran the 2005 study in 2009. This explains why in Singapore the long survey shows a higher dropout than the short survey. Offices worldwide | [email protected] | surveysampling.com | 4 Chart 1-­‐ Average Dropout Rate per Project This large variance implies that there are external drivers to dropout rates that tend not to be related to the survey. This can be further seen in Chart 2, which shows the cumulative distribution of dropouts for the long survey. The straight diagonal lines indicate that drop out occurs constantly during the survey, perhaps due to some random life events. The majority of dropout respondents, in fact, dropped out by halfway through the survey for both long and short surveys. Chart 2 – Cumulative Dropout by Country Offices worldwide | [email protected] | surveysampling.com | 5 Chart 3 – Cumulative Dropout by Country Based on the results, it appears that dropout rates are not synonymous with mental dropout which we would hypothesize ought to “kick-­‐in” at some discrete point in time, leading to a J shaped distribution of cumulative dropout. While long surveys can lead to increased drop out, there are various external factors that come into play across the length of the survey. This makes it very difficult to disentangle the impact of these random events on dropout, thereby suggesting that dropout rates may not be an accurate indicator of data quality. INTERVIEW AND BLOCK LENGTH Another dimension that we considered to test data quality is the speed of response as measured by the average interview and block length. In the 2004 study, one of the hypotheses was that people would increase their speed of response and exert less effort as the interview progresses due to fatigue. And this was evidenced by the significant decrease in interview length for each section as the block moved further towards the end of the long survey. However, in the 2015 long survey, the declines were less dramatic and less clear. The same had been seen in the 2009 survey; people did not seem to speed up towards the end of the survey. If we look back at the significant changes that happened over the past ten years, many would agree that the speed of internet connection has given us the flexibility to do things much faster, particularly in terms of online survey-­‐taking. In 2004, we know that a large proportion of the participants had a dial-­‐up connection. In those days, respondents had to wait longer for the next page to load on their screen. They would look at the progress bar only to see that it had barely moved and they were still far from completing the survey. Five years later, the majority of the respondents were experiencing the benefits of having a faster broadband connection, which allows them to perceive as more reasonable the time taken in completing the task. This explains perhaps why there seems to be less tendency to speed up across the length of the long survey in 2009 and 2015. That said, however, we can still see a slight overall increase in speed in the 2015 long survey. Offices worldwide | [email protected] | surveysampling.com | 6 Chart 4 – Block Lengths (minutes) This trend was not at all visible in any of the short surveys done in previous years. Respondents completed the short survey in a more even pace. We hypothesize that this is primarily due to the shorter amount of time it takes to finish each of the sections. Progress is seen to be being made, and they are able to pause and have a little refresher in between sections. Chart 5 – Block Lengths (minutes) Offices worldwide | [email protected] | surveysampling.com | 7 While speed of internet connection is considered to be one of the drivers that influenced the shorter average survey length in 2015, interview length could also be a reflection of various elements like languages, cultures, infrastructures and even the different devices used in survey-­‐taking in each of the markets. This could be well illustrated in Singapore where the speed of internet connection and near ubiquity of mobile devices has created variation in people’s survey experience. Although faster internet connection may have very well influenced a shorter average interview length in Singapore compared to China and India, taking the survey on a smaller screen or typing in their answers via touchscreen may have caused some to complete the survey much slower, hence hiding the variances in the average interview length between the countries (see Table 2). Given the limitations in the base size, this study was not able to clearly assess the impact of the different devices used on survey length. Further research should be conducted to create a better understanding of the possible implications for data quality of survey taking on different devices. Table 2 -­‐ Average Interview Lengths Total Singapore China India Long Short Long Short Long Short Long Short 2015 27 15 29 19 25 16 27 17 In hindsight, we see that cognitive length is the issue and not the physical length. While respondents were able to complete the survey faster in 2015 than in 2004 and at a more even pace, it does not necessarily mean that the data gathered are of better quality. The evidence presented in the next section shows that panellists do get fatigued. They do not drop out from the survey, they keep going but they exhibit satisficing behaviour, which could seriously impact data quality. PANELIST FATIGUE AND SATISFICING Satisficing is a term that was first coined by Herbert Simon to explain the behaviour of decision makers under circumstances in which an optimal solution cannot be determined. He pointed out that human being are bounded by “cognitive limits”, which compel them to seek a satisfactory or adequate result rather than an optimal solution. This phenomenon also manifests itself in survey data. One of the measures we might use to look for satisficing behaviour is instrument non-­‐use analogous to item non-­‐response. In this metric, we gave people the opportunity to not answer the question. The question involved sliders, which people use to provide their response to a set of scaled items. The slider bar is positioned at the midpoint, and programming allowed respondents to hit the next button and go on with the survey without having to move the sliders. Even with the slider question in the first block, i.e. at the start of the survey, around 10% of respondents actually did not move the slider at all, indicating that “three” or “Neutral” is either correct or a “good enough” answer for them. As the question was encountered in the latter part of the long survey, the number not moving the slider increased. We can also see the same pattern in the short survey, but the effect is less clear. Here they start with an even pace and then start to show satisficing behaviour only towards the end of the survey. Offices worldwide | [email protected] | surveysampling.com | 8 Chart 6 -­‐ Instrument Non Response by Position From the results of the long survey, we see that respondents are more likely to use their knowledge of survey mechanics and structures to skip a section as they encounter it further into the survey. In our survey set-­‐up, respondents could skip an entire section if they claimed not to have been on a short break holiday. In this question, we ask them: “When was the last time they had a short break holiday?” and they can key in the year or click an option to say they have not been on any short holiday. Since the question was randomly positioned at different points in the survey for each of the respondents, we should expect the incidence of qualification to be consistent across all the positions in the survey to discount any evidence of satisficing behaviour. However, findings suggest otherwise. Table 3 shows a high qualification rate when the question was first encountered in both long and short surveys. This dropped significantly from 90% to 80% when the question was positioned at the end of the long survey. It seems that people find it harder to think and recall their past experience as they go further into the survey, becoming cognitively tired. And so they start to satisfice by choosing the easier option of saying that they have not taken any holiday. In the short survey, we do not see the same pattern. The data appears more consistent compared to the studies done in 2004 and 2009. It appears that there is no clear satisficing behaviour except when the survey is very long. Table 3: Qualification for Short Break Holiday section Position 1 Position 2 Position 3 Position 4 2015 Long Survey 90% 90% 89% 80% 2015 Short Survey 95% 84% 92% 82% Offices worldwide | [email protected] | surveysampling.com | 9 Another indicator we used to test data quality effects is the respondent’s involvement in answering open questions. People who are satisficing would tend to write less in an open-­‐ended question. In the chart below, we see that as the open question is positioned further towards the end of the survey, we get fewer characters. This was primarily seen in the long survey. Moreover, the number of characters in the long survey is generally lower compared to the short survey. Chart 7 – Number of Characters per Open Question CONCLUSION The average attention span of an adult is estimated to be around 20 minutes (Cornish and Dukette, 2009). This is the amount of time that you can sustain your attention on a task that you freely choose to do. This coincided with our experiment quite luckily in 2004, as the author of the experiment, Sandra Rathod, marks 20 minutes as the critical time for attention to decrease in an online survey. In 2009, we demonstrated that the survey was still too long even though the physical length of the survey was shorter. It was still producing poor data quality towards the end. Now, we are cognizant that this is mainly caused by the cognitive length and not necessarily physical length. It is the mental effort of answering each of the questions. So while we see how the speed of Internet connection influenced a shorter average time in completing the survey in 2009 and 2015, respondents still find the long survey at 25 minutes too long. It is how many questions are being asked which determines the effort required, and that has not changed. Findings from this study show that fatigue effects when taking long surveys is a phenomenon that spans across Asia Pacific markets and will recur if we re-­‐field the same surveys in Europe and other markets. We will see similar patterns of satisficing behaviour wherein respondents will tend to choose the easiest response option, provide just ‘good enough’ answers and less full answers as they go through the long survey. The impact on data quality as we saw in the short break holiday example could be serious if you are trying to estimate what the market penetration is. The substantial difference we see in the data as respondents Offices worldwide | [email protected] | surveysampling.com | 10 deal with the cognitive burden of answering more and more questions gives clear evidence for how long surveys can prevent us from getting the optimal data that all of us want to achieve. As we gained more insight from the supplementary information we have on average dropout rate per project, we saw that long surveys leads to increased dropouts. While respondents do not drop out because they are mentally fatigued, we can easily see how a long survey can subject them to a multitude of random events that could happen throughout the survey and cause them to drop out. Another interesting point that we uncovered in this study is the crucial role of technology in influencing the respondent’s survey experience. While we know that vast majority of surveys are still being done at home on laptops or desktops, we can see the fast adoption of internet-­‐enabled mobile devices in survey taking. This reality tells us we need to optimize our surveys to leverage these platforms and reach our intended respondents. We need to help them complete the surveys quicker and ease their cognitive burden in answering our questionnaires. We demonstrated this by employing a short survey which consists of short series of questions with breaks in between. While our goal is to ask people questions to predict things about their behaviour, we need to make their experience engaging and compelling. By reducing the physical and cognitive length of the survey, we can help engage respondents and produce high quality data. ABOUT SSI SSI is the premier global provider of data solutions and technology for consumer and business-­‐to-­‐
business survey research, reaching respondents in 100+ countries via Internet, telephone, mobile/wireless and mixed-­‐access offerings. SSI staff operates from 30 offices in 21 countries, offering sample, data collection, CATI, questionnaire design consultation, programming and hosting, online custom reporting and data processing. SSI’s 3,600 employees serve more than 2,500 clients worldwide. Visit SSI at www.surveysampling.com. Offices worldwide | [email protected] | surveysampling.com | 11