So what does your “error message” look like when a respondent makes a mistake?

In the largest Usability study conducted on this issue (“Eyes Don’t Lie” – 2009), we have come across many ways in which marketing researchers likely compound the issue of respondent errors. The above image shows a sample “error message” from one of the studies’ interfaces. Note that the errors are highlighted with an arrow, and text saying “your answer is incomplete”.

• From a Usability point of view, this is what needs to be done – the respondent is directed to the specific problem lines where they had omitted an answer (which is much better than an error message on top of the screen saying “you must complete all questions below”).
• However, what is problematic is the visual cue to potentially ‘tick’ the answer that is closest to the red arrow (as the video of this respondent does show to be the case).

Recommendation: highlight the row, or indicate at the question/statement level where there is an omission or error. Anything else is likely adding error on top of error.

Data suggests an increasing number of companies are either already, or considering, offshoring different aspects of the MR process. This is a complex issue, with many angles & perspectives.

element54 has registered with the Foundation for Transparency in Offshoring (FTO). We chose to declare a “position” early in this discussion for 2 reasons:

(1) We believe that MR companies should be transparent with clients about how we handle their projects. Regardless of why someone chooses to offshore (cost, speed, or any other reason), it is our stated position to always be accountable to our end-clients about where THEIR data is, at all phases of the Marketing Research process.

(2) element54 declared a position of “responsible offshoring” because we live in a global village, and we have learned  you can find trusted business partners around the world, regardless of distance or culture. We’ve worked with recruiters in Asia, online sample providers throughout East/West Europe … and more recently, a firm in India. The experience was positive, our expectations were clearly aligned, and the end-result was successful.

And, to be clear (and transparent) – our client was made aware of exactly what was being offshored, and the credentials of the company we had chosen to work with.

In the field of usability, there is the notion of “Inference vs. Instruction” – ie, how well did the instructions we have provided lead everyone to explicitly understand the same set of parameters in how to answer the question.

In the past couple of years, surveys have become increasingly visual & eye-catching, with the goal of “engaging” respondents.

Our research into this issue, presented around North America – and most recently at The Market Research Event in Las Vegas, has shown that up to 42% of respondents don’t read marketing research survey questions, and highest when visuals dominated the screen.

What can we do? For starters, a simple tip –> ensure the font is large enough (often, survey software has default font sizes that are too small) … and use BOLD and underline in your questions to highlight important information. For example: “Which of these three ads did you prefer most?”

Next time you get a survey link … look beyond typos and skip logic … look for ways to also make sure respondents are clear on what you are asking.

Had a discussion with a client the other day over “vertical” vs. “horizontal” layout of online questions. We were discussing potential issues around the “never” option always appearing off the screen, requiring scrolling to see that answer choice.

During the programming of a recent ad test, the “ad preference” question was programmed vertically – ie. 3 print ads were shown, but only 1 visible at a time, and 2 could only be seen if the respondent scrolled down. I’ll save the Usability lesson on ‘Task Workload” for another post, but wanted to ask this:

If what we are asking a respondent to rate, or comment on is not visible without scrolling up, down and all around – is that a source of potential bias?

To mitigate this, make sure that what you are showing/asking fits on one screen. In the case of this print ad test, I asked the programmer to resize the images so they could appear horizontally (and rotated) onscreen, so no scrolling was required for respondents to evaluate their options.

“Data Consistency” has been a key issue in the discussion of how we can learn about the impact of technology in marketing research.

In dozens of presentations, to hundreds of researchers and marketers this year – the issue of data consistency has gained lots of attention.  That’s because it is incumbent upon all of us to deploy the best solution (both research design, and technology) for our clients.

Most researchers are adept at the ‘science’ of what we do – but what about the ‘art’. It is this latter aspect which has been transformed in recent years, thanks to advances in rich media technology.

Case in point – our firm asked 2000 people the following: “are you an optimist or a pessimist?”. We observed answers which ranged from 5% to 21% (agreement of “10″ on a 1-10 scale), depending on the survey interface (a ‘radio button’, or a ‘visual slider’ scale).

So, your choice of how to design the visual aspect of a question for your survey will have an impact on the results. The question is by how much. Stay tuned for more tips and learning.

The researchers toolkit has become increasingly complex and sophisticated, and a large segment of the research business who were trained during the “telephone” generation, are finding themselves ill-equipped to handle many of the usability aspects of the Online survey design process.

I’ve shown a video during many presentations this year – of Joe, a typical respondent who had an unfortunate experience trying handle a complex question. It shows the last portion of the 5 minutes that were spent on a multiple choice grid question (since not all MR questions are yes/no). The findings highlight important usability learning, but also important DATA QUALITY considerations – what to do with data when they have clearly changed their answers after not being able to click “next”? Stay tuned for more usability tips, and contact us to learn more.

Spend a few minutes with researchers from 400+ client and supplier organizations, and you get a feel for how complex Online survey research has become, and the growing need to allow standards to catch up with technology.

Only a few years ago, most online surveys were quite similar, in both look and feel.  What many refer to as “radio buttons”, or the standard blank circles/squares – most online surveys were adapted from telephone surveys by an entire generation of Marketing Researchers. These phone surveys did not need to be either creative, engaging, or intuitive, since they were administered by an interviewer.

What we are witnessing today, is a technology “space race” in the marketing research business. Many researchers have embraced new technology. element54 has led two massive studies in this area (“Sexy Questions, Dangerous Results” and “Eyes Don’t Lie”), with proactive and practical suggestions for how to maximize the confidence in the survey, and ultimately deliver powerful and actionable data.  Stay tuned for a series of thought-provoking experiences and insights which have emerged from these studies, which have been presented to hundreds at conferences around North America, and dozens of private sessions.

An interesting session at The Market Research Event in Las Vegas (Oct ’09) dealt with the issue of Off-shoring Transparency. In an ever-increasing business climate of needing to deliver on the “better, faster & cheaper“, there has been a significant increase, and focus on the issue of off-shoring various aspects of the marketing research process.  To be clear, our position on this issue is that it must not be construed as “anti” any country, since the objective is to better understand, and promote safety and trust to off-shoring practices.

The following site has been setup to encourage companies to adopt a stated (and transparent) policy. http://www.offshoringtransparency.org/index.php?page=home