Handbook on Data Collection / Phase Four B: Design Surveys

Photo: sdgactioncampaign.org

Continued from Phase Four A: Design Samples

Survey design

Step one: Outline the design of your survey

After you’ve defined your sample, you can decide how you’ll collect the data from the respondents. Your survey design will include a survey format with a list of questions which correspond to your data needs and the frequency at which the data will be collected. The frequency of data collection depends on the type of data being collected. Baseline or mapping studies are one time surveys while tracking/monitoring surveys are conducted at time intervals. The periodicity of data collection is determined by the project goals and objectives and the related set of indicators listed in your Theory of Change.

Step two: Adopt good practices while designing your questionnaire

A questionnaire is likely to be most effective if you follow KISS: Keep it short and simple. If you don’t have a satisfactory answer to what you will do with the answer to a question, leave it out. Avoid the temptation to add a few more questions just because you are doing a questionnaire anyway. If necessary, place your questions into three groups: must know, useful to know, and nice to know. Discard the last group, unless the previous two groups are very short.

Start your questionnaire with an introduction or welcome message clearly stating who you are and why you want the information in the survey. A good introduction or welcome message will encourage respondents to cooperate and participate. In case of sensitive or private information, reassure your respondent that their responses will not be revealed. In some cases e.g. child/underage surveys, it may be mandatory to seek the consent of the respondent or guardian.

While designing a questionnaire, it’s important to reflect on how the order of the questions can impact the results of your survey. Ideally, you should:

• Place the easiest and most pleasant to answer questions at the beginning of your survey.
• Group together questions on the same topic.
• Leave difficult or sensitive questions until near the end of your survey.
• Address the data collector observations, validation issues, GPS readings, photographs, testing (e.g. water quality testing) at the end of the survey.
• Avoid breaks while interviewing the respondent
• Use a logical or natural order to answer choices, presenting "positive to negative" or "excellent to poor" scales and agree/disagree choices in that order.

Step three: Design your questionnaire according to the data type

Broadly speaking, there are two types of data: quantitative data and qualitative data.

Quantitative data is collected with a structured questionnaire which may have closed ended questions (i.e. with a list of options to choose from) and/or open ended questions, depending on the type of information you need.

Qualitative data is often essential to understanding the context and explaining the quantitative data. It is generally collected as free text, which may be translated into numbers by classifying the information or assigning codes. It is recommended that qualitative information be used sparingly, for example, use it where the possible responses are not known in advance, if will add value to your survey. This is because qualitative answers tend to take longer to check, clean, and process.

Most questionnaires will gain value by using a combination of both types of data. Your questionnaire will depend on the objectives of your project. For instance, if you need data to monitor the status of water points across a city, you are likely to ask the following questions:

• What type of waterpoint is it? Respondents can select from a list of options, e.g. hand pump, well, or tap.
• Where is the waterpoint located? Respondents can provide the name of the city/village and a GPS reading.
• Is the waterpoint functioning? You’ll need to clearly define functionality to ensure a common understanding for all data collectors.
• Is the water safe to drink? For this, you may need to test the water for certain parameters, document perceptions on water safety from respondents, or collect healthcare information from existing records.
• Who owns or is responsible for this water point and is it maintained? You can ask whether it is publicly owned (government) or privately owned, what type of repair (major or minor) has been done in the last few years, how much it cost and who paid for it. Again, define what major and minor repairs mean to you.

In the above example of the waterpoint monitoring, the first two questions are examples of (structured) questions to collect quantitative data. Questions after that could be framed to collect both quantitative and qualitative information. The fourth question, where you ask about perceptions of safe water, is an example of qualitative information, where you will record the responses verbatim and enter the data as free text.

Step four: Choose the question type to match your data needs

There are three basic ways in which questions are designed in surveys:

1. Multiple choice e.g. Have you watched this movie?

 ☐ Yes
 ☐ No

2. Numeric open ended e.g. How many times have you watched the movie?

  _______________________________________

3.Text open ended e.g. What did you like about this movie?

  _______________________________________

Rating Scales and Agreement Scales are two common types of questions also used to qualify multiple choice questions.
1. How would you rate the movie?

  ☐ Excellent
  ☐ Good
  ☐ Fair
  ☐ Poor

2. On a scale, where ‘10’ means you have enjoyed this aspect the most, how would you rate the movie?

  Acting ___
  Editing ___
  Casting ___
  Direction ___
  Production ___
  Cinematography ___

3. How much do you agree/disagree with the following statements?

	Strongly Agree	Agree	Disagree	Strongly Disagree
a) The movie has a strong social message
b) Children should not watch the movie
c) There is unnecessary violence in the movie
d) The storyline of the movie is weak

While designing a close-ended questionnaire, you should try to include the maximum possible list of relevant alternatives as answer choices. This helps to systemise and categorise respondent’s answers and saves time on text entries. However, this also reduces the scope for capturing detail and you will need to decide how flexible you want to be and to what extent the additional detail will improve the findings of the survey. Choosing a question type is largely based on how you want the data to come out, which depends on how you want to use the data.

You can pretest the questionnaire before the survey if you want to generate a list of alternative question types. When you’re unsure about the possible answer choices, use an open ended format by adding “other (specify)” as one of the alternatives. Also allow a “don’t know” or “not applicable” response to all questions, except to those in which you are certain that all respondents will have a clear answer.

In your survey, some questions may be dependent on responses to other questions. For example, in our case of the moviegoer, if the respondent’s answer to question one is “no,” i.e. they have not watched the movie, the rest of the questions would be irrelevant. In this case, you would add an instruction “continue survey only if the response is ‘yes’ in question one.”

Data collector selection

Step one: Select and prepare data collectors

The next step will be to define who will collect your data. Emails, Internet and telephone conversations are popular methods of collecting data. However, in this chapter, we will talk about data gathering with data collectors using mobile/tablet-based surveys or paper surveys.

Data collectors should be selected with care and should be sufficiently briefed/trained on the questionnaire before data collection. A guide with a set of instructions and an explanation for each question is always advisable. The guide should also provide definitions for each alternative to ensure a common understanding across data collectors. For example, if the questionnaire requires the data collector to list the type of a water source as improved or unimproved, the guide should clearly mention what is an improved source (i.e. hand pump, tap) and what defines an unimproved source (open well, pond).

Step two: Guidelines for working with data collectors

Guidelines	Importance
Identify and train more data collectors than you need.	Enables you to replace data collectors on short notice without having to train them again.
Select data collectors who have an understanding/familiarity with the local culture and sociology.	Helps to build rapport quickly with the respondents and demonstrates sensitivity to local cultures.
Select data collectors who are conversant in the local language/dialect.	Helps communication during the survey and minimises the use of interpreters/translators, which saves time and resources.
Brief your data collectors well before the survey. Pretest the questionnaire through sample visits or simulation exercises.	Ensures the data collectors have understood the questionnaire properly. During field tests or simulations, judge their capacities and train/guide them further, if needed.
Advise data collectors to be sensitive and value the time being given by the respondents.	Demonstrating sensitivity is the best way of thanking a respondent for the time they are voluntarily giving you.
Advise data collectors to be courteous and respectful to the respondents.	Remember that a respondent is not obliged to answer your questions. Reassure them their responses matter.
Advise data collectors to select time slots when the respondent is free to talk.	Timing of your survey is important because if respondents are busy or preoccupied, they are more likely to give an incorrect response.
Pretest the questionnaire through sample visits or simulation exercises.	Ensure the data collectors have understood the questionnaire properly. During field tests or simulations, judge their capacities and train them, if needed.

Conclusion

Regardless of the field of study, the goal of all data collection projects is to capture convincing and credible evidence. A variety of accepted data collection techniques are available. You can use your judgement to select the best method, based on the objectives of your research and the available resources. The key to robust data collection is to select the most appropriate method that strikes a balance between improving accuracy while protecting the credibility and reliability of your data. A clean and representative data set is essential in analysing data and helps to reduce time spent on preparing the data before analysis, which you can read more about in phase five of the Handbook. The choice of survey methods, questionnaire design and data collectors’ capacities largely contribute to this.

Suggested reading

• Sampling Techniques and Sample Types
• Statistical Methods: 13 Sampling Techniques
• What is Qualitative Research? - Definition, Sources & Examples

Acknowledgements

Author: Rajashi Mukherjee (Akvo.org)
Contributors: Camille Clerx (Akvo.org), Hans van der Kwast (IHE Delft Institute for Water Education), Nikki Sloan (Akvo.org), Stefan Kraus (Akvo.org)

AfriAlliance

The Africa-EU Innovation Alliance for Water and Climate (AfriAlliance), is a 5-year project funded by the European Union’s H2020 Research and Innovation Programme. It aims to improve African preparedness for climate change challenges by stimulating knowledge sharing and collaboration between African and European stakeholders. Rather than creating new networks, the 16 EU and African partners in this project will consolidate existing ones, consisting of scientists, decision makers, practitioners, citizens and other key stakeholders, into an effective, problem-focused knowledge sharing mechanism.

AfriAlliance is lead by the IHE Delft Institute for Water Education (Project Director: Dr. Uta Wehn) and runs from 2016 to 2021. The project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 689162.