QUESTIONNAIRE DESIGN FOR OBTAINING SUBJECTIVE DATA
(ASKING PEOPLE WHAT THEY THINK)

QUESTIONNAIRES

  • Written format, needs very careful work (including piloting and refining a questionnaire) to ensure that all questions are absolutely clear to all respondents and that you are collecting all the information you need.
  • Good when the issues you want to address are well defined (e.g. finding out the problems users are having with a current system which needs improvement)
  • Less good for situations where the questions are not well defined (e.g. when developing a novel system)
  • Good way to get a lot of information from a lot of people in a way that can be relatively quickly analysed.
  • But the response rate may seem to be low - getting 40% of questionnaires returned is considered very good for a postal questionnaire.
  • The more effort you put in, the higher the response rate you will get and the better the data will be. If you write follow up, reminder letters, emails, phone calls, and offer raffles…all improve response rate. Questionnaires should be too long to complete-length needs to be commensurate with the importance of the topic (and the reward being offered), but 15 minutes is LONG.
  • Online questionnaires are interesting, but do not think that this will make it any easier, there is just as much work, but a different set of problems.

QUESTION FORMATS

Although the information is subjective it can produce both qualitative and quantitative data.
There are 3 formats of questions:

  • Open-ended
  • Closed
  • Rating scales

and (Ranks - avoid, very difficult to analyse fully)

OPEN-ENDED FORMAT

  • OPEN-ENDED - completely free response
    "what did you like about ….."
    Good when you want to elicit all kinds of information, want respondents to be creative (they often enjoy this), and don't know what they might say.
  • Difficult to analyse, particularly if you have lots of respondents.
  • Usefulness of dummy analysis run - fill out a few questionnaires, what will you do with them?
  • Need to develop your own categories to group the answers.
  • The problem is then to decide whether two differently worded answers are the same category.
  • Often need two people to go through the answers, establish inter-coder reliability etc (in effect, conduct a content analysis)
  • Can use open-ended questions in a pilot version of the questionnaire given to a small number of people, then turn into a list of options.

CLOSED FORMAT

Yes/no, or limited set of options, with perhaps a "don't know" or "other" response to catch cases you haven't thought of (you don't want more than 10%in these categories)

  • Closed format is easy for respondents to answer and good for getting lots of data that's easy to process and understand.
  • But fundamentally categorical in nature (answers "what?" questions), doesn't help with how much people liked something or how easy people found something to use.
  • These latter, quantitative questions are very useful in designing systems - may want to compare to designs, or compare a design over time.

LICKERT SCALES

These are a very simple and neat way of measuring the 'how much' type questions.
Lickert or rating scale (named after Mr.Lickert)

e.g. "The arrangement of icons was easy to remember"
(circle one cross on the scale)

agree +-----+-----+-----+-----+-----+-----+ disagree

Can use 5, 7, 9 graduations depending on what you think the respondents can discriminate.

  • Can very easily compare different arrangements of icons and say:
  • Arrangement A - mean rating 4.0/7
  • Arrangement B - mean rating 5.2/7
  • Arrangement C - mean rating 5.4/7
  • Indifferent statistics (e.g. one way ANOVA) to show whether arrangement B was any better than arrangement C, or just a random difference.

Combination of a Lickert scale with an open-ended comment where respondents can justify their ratings can be very useful;
You get a quantitative measure for statistical analysis and comparison
You get some words from respondents to give meanings to the numbers (or alternatively you can ignore them)

QUESTIONNAIRE CHECKLIST

CLARITY

  • Make questions clear
  • Avoid double-barrelled questions where the answers will be ambiguous
    e.g. "Do you think it would be helpful to make the icons or buttons larger"
    Yes (the icons or the buttons?)

SPECIFICITY

  • Make the questions specific
  • You do the work to generalise, not the respondents
    e.g. "How many hours do you use the Internet in a week?"
    Better to ask specifically about yesterday, otherwise respondent is trying to average/add up in their mind - you then multiply by 7 (although you need to think about weekdays and weekends)

ORDERING

  • Think about the ordering of questions
  • Ask easy questions first, get the respondent relaxed (lulled into a false sense of security)
  • Ask personal questions last (people don't like answering these, but if they've put effort into all the other questions they will probably do a few more)
  • Does answering one question have an impact on subsequent ones?

EXAMPLE

Communist reporter item:
Do you think the United States should let Communist newspaper reporters from other countries come here and send back to their papers the news as they see it?

American reporter item:
Do you think a Communist country like Russia should let American newspaper reporters in and send back to America the news as they see it?

% Yes to Communist reporter item:

Order
Comm/Amer Amer/Comm
1948   36   73
1980   55   75

% Yes to American reporter item

Order
Comm/Amer Amer/Comm
1948   66   90
1980   64   82

Note the change from 36% agreement to 90% agreement in 1948 and 55% agreement to 82% agreement in 1980 on fundamentally the same moral question, depending on how it was posed.

Moral - statistics don't lie, but it does depend on how you ask the question.

SCALES

  • Make scales intuitive, clear and consistent
  • If you use numbers, 1=low agreement, 5=high agreement and is usually intuitive (like an exam score)
  • It's easier for the respondents if you don't mix negatives and positives, but you may want to, to keep respondents on their toes (avoid "blind agreement", make them think!)

LANGUAGE

  • Avoid technical or HCl jargon - speak the language used by respondents (may need a pilot)
    e.g. "Is this web site easy to navigate around?"
  • Do the users of e-learning web sites actually think about "navigation", what language should we be using instead?

INSTRUCTIONS

  • Provide clear instructions on how to complete the questionnaire and particular questions
      (people will ignore them anyway, but at least you tried!)

    (circle one cross on the scale)

    agree +-----+-----+-----+-----+-----+-----+ disagree

    Any number of people will put crosses between the crosses (an example often helps)

LAYOUT

  • Clear layout is also important and helpful (plus enough room to answer open-ended questions), but you need a balance between a lot of white space and a long looking questionnaire which will be off-putting.