The difficulty of asking easy questions

I recently carried out some user testing for FoodNetwork.co.uk, a question I asked was how often people cooked. I wanted to learn about people’s cooking habits and how they approached cookbooks or websites with recipes. Despite seeming like a straightforward question I realised that there were a number of challenges to overcome with regards the wording.
“How often do you cook?”

This is confusing because of what the word ‘cook’ means, is making a sandwich or a salad cooking? To some cooking means making something hot, but that might not be sufficient, others don’t consider a toasted sandwich cooking because the word ‘cooking’ may imply a certain level of complexity or effort.

Solution – use the word ‘prepare’, which is more neutral.
“How often do you prepare a meal?”

To some this could include ready-meals (as opposed to eating out or ordering take-aways), which I did not consider preparing a meal for this context.

Solution – add in the clause ‘excluding ready-meals’.
“How often do you prepare a meal, excluding ready-meals”

Now there is the problem of which meals I’m talking about, some people assume it means just an evening meal, for others it includes breakfast or lunch.

Solution – add in the clause ‘evening meal’
“How often do you prepare an evening meal, excluding ready meals?”

Human memory is notoriously unsuited to answering questions such as this, memory can be remarkably malleable (read more here by leading psychologist Elizabeth Loftus). Because the question is about preparing meals in general participants will tend to think in general and will be subject to biases, such as demand characteristics (for an amazing and amusing paper highlighting this problem see here) where they may try to please the experimenter. For example, participants may describe an idealised scenario where they prepare more food than they actually do.

Solution – anchor the question to a specific, unambiguous and recent time frame to encourage participants to search their memory for actual instances of preparing food. I avoided using the word week as some people may have thought that I meant only during the week the interviews took place.
“How many times did you prepare an evening meal, excluding ready-meals in the last 7 days?”

Unfortunately, this is a leading question, it implies that the person did prepare food, rather than was cooked for, ate out or ordered take-aways. This may make some feel that they should have cooked consequently leading to demand characteristics.

The final question(s) I went with was:
“Did you prepare any evening meals, excluding ready-meals, in the last 7 days?” then, if so, “How many did you prepare?”

There were other issues as well but I think I tightened it up quite a bit. It certainly illustrates some of the methodological challenges facing researchers when investigating people.