The question posed to the group of 5 Bloggers this month was: “Mobile surveys – For/Against,
Pros/Cons, Right Situations/Wrong Situations?” Links to my fellow bloggers Annie Pettit,
Joel Rubinson, Josh Mendelsohn and Brandon Bertelsen can be found below.

There’s no debating the societal impact of Mobile: Nokia stated at this weeks’ CES conference that now, over 50% of internet services are accessed on mobile phones, not from PCs. And, we all know that the level of mobile-only continues to rise (well past 20% in some demographic segments), forever changing landline survey options.

So what does that mean for Survey Researchers? Should each of us have our own $.99 survey App on iTunes? Not so fast, let’s think about this from a few perspectives.

Consistency: if the objective is to empower respondents with another mode of completing surveys, mobile certainly offers some interesting benefits. But as we’ve learned from the results of 1.0 vs. 2.0 web-based surveys (”Sexy Questions Dangerous Results?”), there will certainly be differences in the data when comparing mobile to web – and a range of new issues for which we have limited best practices (sampling, questionnaire length, visual layout & usability differences between BlackBerry and iPhones etc). As with each of the new tech options/advances, it’s critical to think through the “when, why and how” before we push the send button. As an industry, let’s avoid the temptation to over-focus on the ‘consistency’ of survey learning from Mobile vs PC, it’ll just get us into trouble, and distract us from creating end-client insight value from this exciting new channel.

Context: the benefits of using a mobile mode (either exclusively, or in conjunction with other methodologies) needs to take into account the contextual relevancy of using this platform. If you’re trying to get conference attendees to tell you what they thought of the speaker they just heard, that would be a intuitive application of a mobile platform. But don’t try to be cute, and offer people the choice to do a PC or mobile survey. A recent client survey found that under 5% of respondents chose the mobile option when given the choice between PC & mobile. And, you incur additional programming costs/delays to deploy the additional mode, for limited response rate upside, and questionable data consistency downside.

Future Focus: I recognize the research importance of making sure we know whether the “Next” button should be on the top or bottom of an iPhone screen. But the VP Marketing couldn’t care less – she/he wants to know how this mobile survey will help them better understand consumer needs. Consider the following App, which allows an iPhone user to scan a barcode, and get product rating information. This is where I believe mobile gets exciting for researchers, by linking real-time consumer behavior with attitudinal research communities (MROC’s). And dare I say, leveraging the power of the channel for exciting Qual opportunities from a new generation of Internet-savvy ethnographic researchers.

So, I say:  Dear <insert client name>, the next time I’m sitting on a plane, use my BlackBerry GPS locator (you can access it) and Frequent Flyer number (you have it) to send me a 5 question CSAT survey. I promise to respond. Sincerely, Bernie.

Note: element54 will be presenting new Mobile R&D – “The Moment of Truth” at the MRIA’s 50th annual National Conference on Monday May 30/2010. Stay tuned for details, and visit the conference website.

Links to Annie, Joel, Josh and Brandon (each to be updated as soon as available)

Annie Pettit of Lovestats: http://lovestats.wordpress.com/2010/01/15/1topic5blogs-the-only-thing-cell-phone-surveys-are-good-fer/

Joel Rubinson of the ARF:
Josh Mendelsohn of Chadwick Martin Bailey: http://lovestats.wordpress.com/2010/01/15/1topic5blogs-the-only-thing-cell-phone-surveys-are-good-fer/
Brandon Bertelsen:

I am privileged to be one of 5 bloggers who, each 15th of the month – will produce a POV on an issue facing the Marketing Research industry. You’ll also be hearing from Annie Pettit (organizer), Josh Mendelsohn, Joel Rubinson and Brandon Bertelsen. Links to their posts will follow.

Our first topic, is something which has been of utmost importance and urgency to element54 this year, and one where my partners and I have invested considerably, to advance the issue in our business (some previous posts on this site offer practical & prescriptive solutions).

Background/Issue:

We are witnessing an explosion of engaging new question types – intended to captivate respondents, and ensure the long-term viability of an online survey platform.

Research to-date has described “respondent engagement” as the notion that improving the usability and interest of Online question types will enhance respondents’ enjoyment, and the thoughtfulness of their survey experience. However, there has been little work to validate the myriad of new question types, and specifically, how response patterns vary across each type.

element54 conducted 2 transparent studies in 2009, which have been presented at several North American conferences (most recently, The Market Research Event, Las Vegas – Oct ’09)

Study #1 – “Sexy Questions, Dangerous Results?” (Data Consistency):

In January 2009, element54 and ResearchNow, conducted the first-ever fully transparent research study (full dataset available to MR peers), which examined the response patterns to various Online question types. The findings from this 2000 interview study were clear. Changing the visual layout of an online survey leads to differences in the data.

• 36% variance in how often people say they brush their teeth,
• Up to 8% overclaiming of behavioral product category consumption,
• Up to 10% understating of attitudinal “issue importance”.

Study #2 – “Eyes Don’t Lie” (Respondent Usability):

In May 2009, element54, along with partners UX Research & Consulting, and MBA Research, conducted the largest transparent “Eye Tracking” study on survey usability, with 100 one-on-one Qual/Quant interviews, using SMI Vision eye tracking software to validate respondent eye movements & patterns. The findings from this second study leveraged how Usability insights and applications can be applied to designing better surveys:

• Inference vs. Instruction; up to 40% of “2.0” survey questions aren’t read when visuals dominate the screen. It is certainly concerning if respondents are inferring our intent.
• Error Prevention & Recovery; ever tried to get off an elevator when you miss your floor – yes, you push other buttons to get off. When survey “error messages” are not linked to where the problem is, respondents are likely to change their answers to escape. This issue raises a host of related questions around how we treat “error” data once the respondent successfully clicks “Next”.

Next Steps:

Technology is headed in the right direction. To sustain the long-term viability of Online surveys, we do need to create appropriately engaging platforms for respondents. However, in the current space race, and range of available DIY software platforms – there’s no glory in getting to the moon first, if you crash land.  This is where best practices and standards can, and must catch up with all this exciting technology.

Here are the links to Annie, Joel, Josh and Brandon (soon).

Annie Pettit of Lovestats: http://lovestats.wordpress.com/2009/12/21/1-topic-5-blogs-rich-media-in-surveys/
Joel Rubinson of the ARF: http://blog.joelrubinson.net/2009/12/getting-the-most-out-of-online-research/
Josh Mendelsohn of Chadwick Martin Bailey:http://betterresearch.blogspot.com/2009/12/1-topic-5-blogs-interactive-questions.html
Brandon Bertelsen: link posted shortly at http://bertelsen.ca/

So what does your “error message” look like when a respondent makes a mistake?

In the largest Usability study conducted on this issue (”Eyes Don’t Lie” – 2009), we have come across many ways in which marketing researchers likely compound the issue of respondent errors. The above image shows a sample “error message” from one of the studies’ interfaces. Note that the errors are highlighted with an arrow, and text saying “your answer is incomplete”.

• From a Usability point of view, this is what needs to be done – the respondent is directed to the specific problem lines where they had omitted an answer (which is much better than an error message on top of the screen saying “you must complete all questions below”).
• However, what is problematic is the visual cue to potentially ‘tick’ the answer that is closest to the red arrow (as the video of this respondent does show to be the case).

Recommendation: highlight the row, or indicate at the question/statement level where there is an omission or error. Anything else is likely adding error on top of error.

Data suggests an increasing number of companies are either already, or considering, offshoring different aspects of the MR process. This is a complex issue, with many angles & perspectives.

element54 has registered with the Foundation for Transparency in Offshoring (FTO). We chose to declare a “position” early in this discussion for 2 reasons:

(1) We believe that MR companies should be transparent with clients about how we handle their projects. Regardless of why someone chooses to offshore (cost, speed, or any other reason), it is our stated position to always be accountable to our end-clients about where THEIR data is, at all phases of the Marketing Research process.

(2) element54 declared a position of “responsible offshoring” because we live in a global village, and we have learned  you can find trusted business partners around the world, regardless of distance or culture. We’ve worked with recruiters in Asia, online sample providers throughout East/West Europe … and more recently, a firm in India. The experience was positive, our expectations were clearly aligned, and the end-result was successful.

And, to be clear (and transparent) – our client was made aware of exactly what was being offshored, and the credentials of the company we had chosen to work with.

In the field of usability, there is the notion of “Inference vs. Instruction” – ie, how well did the instructions we have provided lead everyone to explicitly understand the same set of parameters in how to answer the question.

In the past couple of years, surveys have become increasingly visual & eye-catching, with the goal of “engaging” respondents.

Our research into this issue, presented around North America – and most recently at The Market Research Event in Las Vegas, has shown that up to 42% of respondents don’t read marketing research survey questions, and highest when visuals dominated the screen.

What can we do? For starters, a simple tip –> ensure the font is large enough (often, survey software has default font sizes that are too small) … and use BOLD and underline in your questions to highlight important information. For example: “Which of these three ads did you prefer most?”

Next time you get a survey link … look beyond typos and skip logic … look for ways to also make sure respondents are clear on what you are asking.

Had a discussion with a client the other day over “vertical” vs. “horizontal” layout of online questions. We were discussing potential issues around the “never” option always appearing off the screen, requiring scrolling to see that answer choice.

During the programming of a recent ad test, the “ad preference” question was programmed vertically – ie. 3 print ads were shown, but only 1 visible at a time, and 2 could only be seen if the respondent scrolled down. I’ll save the Usability lesson on ‘Task Workload” for another post, but wanted to ask this:

If what we are asking a respondent to rate, or comment on is not visible without scrolling up, down and all around – is that a source of potential bias?

To mitigate this, make sure that what you are showing/asking fits on one screen. In the case of this print ad test, I asked the programmer to resize the images so they could appear horizontally (and rotated) onscreen, so no scrolling was required for respondents to evaluate their options.

“Data Consistency” has been a key issue in the discussion of how we can learn about the impact of technology in marketing research.

In dozens of presentations, to hundreds of researchers and marketers this year – the issue of data consistency has gained lots of attention.  That’s because it is incumbent upon all of us to deploy the best solution (both research design, and technology) for our clients.

Most researchers are adept at the ’science’ of what we do – but what about the ‘art’. It is this latter aspect which has been transformed in recent years, thanks to advances in rich media technology.

Case in point – our firm asked 2000 people the following: “are you an optimist or a pessimist?”. We observed answers which ranged from 5% to 21% (agreement of “10″ on a 1-10 scale), depending on the survey interface (a ‘radio button’, or a ‘visual slider’ scale).

So, your choice of how to design the visual aspect of a question for your survey will have an impact on the results. The question is by how much. Stay tuned for more tips and learning.

The researchers toolkit has become increasingly complex and sophisticated, and a large segment of the research business who were trained during the “telephone” generation, are finding themselves ill-equipped to handle many of the usability aspects of the Online survey design process.

I’ve shown a video during many presentations this year – of Joe, a typical respondent who had an unfortunate experience trying handle a complex question. It shows the last portion of the 5 minutes that were spent on a multiple choice grid question (since not all MR questions are yes/no). The findings highlight important usability learning, but also important DATA QUALITY considerations – what to do with data when they have clearly changed their answers after not being able to click “next”? Stay tuned for more usability tips, and contact us to learn more.

Spend a few minutes with researchers from 400+ client and supplier organizations, and you get a feel for how complex Online survey research has become, and the growing need to allow standards to catch up with technology.

Only a few years ago, most online surveys were quite similar, in both look and feel.  What many refer to as “radio buttons”, or the standard blank circles/squares – most online surveys were adapted from telephone surveys by an entire generation of Marketing Researchers. These phone surveys did not need to be either creative, engaging, or intuitive, since they were administered by an interviewer.

What we are witnessing today, is a technology “space race” in the marketing research business. Many researchers have embraced new technology. element54 has led two massive studies in this area (”Sexy Questions, Dangerous Results” and “Eyes Don’t Lie”), with proactive and practical suggestions for how to maximize the confidence in the survey, and ultimately deliver powerful and actionable data.  Stay tuned for a series of thought-provoking experiences and insights which have emerged from these studies, which have been presented to hundreds at conferences around North America, and dozens of private sessions.

An interesting session at The Market Research Event in Las Vegas (Oct ‘09) dealt with the issue of Off-shoring Transparency. In an ever-increasing business climate of needing to deliver on the “better, faster & cheaper“, there has been a significant increase, and focus on the issue of off-shoring various aspects of the marketing research process.  To be clear, our position on this issue is that it must not be construed as “anti” any country, since the objective is to better understand, and promote safety and trust to off-shoring practices.

The following site has been setup to encourage companies to adopt a stated (and transparent) policy. http://www.offshoringtransparency.org/index.php?page=home