Ray Poynter used the term “Doing the Monkey” at the recent NetGain Toronto conference, and the larger issue of how “survey independence” is changing the research landscape. The serious side of this topic, is the growing trend of “Do It Yourself” Market Research, spawned by exciting new technology/software, in both the Qualitative and Quantitative fields.

Some have framed this discussion as Research Professionals vs. Non-Professionals, and who should be allowed to have the ‘keys to the car’. Over the past few years, some of my clients have deployed their own proprietary panels, others continue to rely on outside research agencies. There is no right or wrong … rather, just “Good or Bad research”.

In a recent LinkedIn thread posted by Ray Poynter (NewMR – Co-Creating the Future of Market Research), Mark Kupferman (Director of Consumer Insights, Universal Orlando) asked:

• “One has to ask where professional research ends and where “DIY” research begins. If I’m at a company and I do my own research with my own customers and/or with rented panels … am I doing DIY research because I’m not using an external research agency to write the questions and do the analysis? What if I or the people I hire have worked in a professional research firm in the past? And what is it that makes professional researchers better at executing research than an internal team of researchers?”

There is a time and a place for both DIY, and ‘Assisted’ projects … and there is both “Good” and “Bad” research being conducted. I’d encourage you to visit Jeffrey Hennings’ blog at Vovici, who has posted more on this topic than anyone else http://blog.vovici.com/blog/?Tag=Survey%20Monkey including this potential warning:

If you are a Client-Side researcher, Kathryn Korostoff (AKA, Research Rockstar) did an excellent piece summarizing the DIY short-list to ask yourself http://www.researchrockstar.com/diy-or-hire-a-market-research-company/

• Are you confident that you can write a questionnaire such that it will capture information objectively.
• You realistically have time to do the project management in-house
• You have the tools and skills in-house to clean the collected data and analyze it
• You have resources in-house that can report the findings in a way that will be credible to your internal colleagues

If you are a Supplier-Side researcher, stop dissing DIY platforms just because they are “DIY” instead of full service, and focus on evolving a solutions toolbox that gives clients your expertise, in situations where they have reframed the costing model for how you can add value to their decision making (ie – that “full serve” project might now become a consulting mandate). Focus on adding value, helping to develop/build question libraries, conduct advanced statistical analyses of Attitudinal & Behavioral databases, and help those clients establish best practices. This paradigm shift is happening for lots of different reasons, not just economics. Embrace the wave.

If you are a Software Provider, I believe it is time to practice “responsible innovation”. Having led 2 research-on-research studies over the past year (Sexy Questions, Dangerous Results and Eyes Don’t Lie), it’s clear that software developers must re-take the high ground and guide their end-users with how and why to choose different question types. Most researchers lack the Usability experience to guide clients on how/when different question types are best used, and the risks associated with survey layout.

But, at the heart of every good survey is a good question … and we can’t blame software providers when someone just asks a bad question …

Read the other blogs:

• Annie Petit (AKA “LoveStats”) at Conversition Strategies: http://lovestats.wordpress.com/2010/02/14/1-topic-5-blogs-diy-surveys/
• Josh Mendelsohn of Chadwick Martin Bailey http://blog.cmbinfo.com/bid/33687/1-Topic-5-Blogs-Why-market-research-professionals-should-embrace-DIY-Surveys
• Brandon Bertelsen: http://www.bertelsen.ca/market-research/1-topic-5-blogs-diy-surveys
• Joel Rubinson of the ARF: post coming shortly at http://blog.joelrubinson.net/

I learned that Jeffrey Henning (@JHenning) of Vovici and I have something in common – we both have 16 year olds at home. I was recounting a recent story, and decided to share, you know – for the market research implications.

My eldest has been studying hard lately. I was on my way home, and decided to pop into a Pizza place to pick up for him and his friend who was over at our house studying w/him.

Price/Inventory: If you call ahead, a large veggie pizza goes for $7.99. As it turned out, they they had one under the warmer, except that it was missing 2 slices.

Conversation with Pizza Guy:

• Me says: I’ll take that partial large pizza (6 slices) instead of 8 slices … And pay $7.99.
• Pizza dude: sorry, I have to charge you $2.25 per slice (x 6 = $13.50).
• Me says: but I’ll pay you full price for only 75% of that pizza.
• Pizza dude: sorry, if you want a whole pizza –  I can make one for you in 15 minutes.
• Me says: thanks, but sorry dude, gotta go.
  

So how much is 75% of a pizza worth?

Technically, I think he was correct, and I admired his holding the party line on the “pay per slice” take-out-premium-pricing model. Yet, it was an interesting Customer Service experience – since the opportunity to generate a guaranteed $8, and still have another full pie out within 15 minutes (on a not so busy night) was lost.

Mr. Pizza Dude, I will be back again, and might even call ahead next time.

There’s no debating the societal impact of Mobile: Nokia stated at this weeks’ CES conference that now, over 50% of internet services are accessed on mobile phones, not from PCs. And, we all know that the level of mobile-only continues to rise (well past 20% in some demographic segments), forever changing landline survey options.

So what does that mean for Survey Researchers? Should each of us have our own $.99 survey App on iTunes? Not so fast, let’s think about this from a few perspectives.

Consistency: if the objective is to empower respondents with another mode of completing surveys, mobile certainly offers some interesting benefits. But as we’ve learned from the results of 1.0 vs. 2.0 web-based surveys (“Sexy Questions Dangerous Results?”), there will certainly be differences in the data when comparing mobile to web – and a range of new issues for which we have limited best practices (sampling, questionnaire length, visual layout & usability differences between BlackBerry and iPhones etc). As with each of the new tech options/advances, it’s critical to think through the “when, why and how” before we push the send button. As an industry, let’s avoid the temptation to over-focus on the ‘consistency’ of survey learning from Mobile vs PC, it’ll just get us into trouble, and distract us from creating end-client insight value from this exciting new channel.

Context: the benefits of using a mobile mode (either exclusively, or in conjunction with other methodologies) needs to take into account the contextual relevancy of using this platform. If you’re trying to get conference attendees to tell you what they thought of the speaker they just heard, that would be a intuitive application of a mobile platform. But don’t try to be cute, and offer people the choice to do a PC or mobile survey. A recent client survey found that under 5% of respondents chose the mobile option when given the choice between PC & mobile. And, you incur additional programming costs/delays to deploy the additional mode, for limited response rate upside, and questionable data consistency downside.

Future Focus: I recognize the research importance of making sure we know whether the “Next” button should be on the top or bottom of an iPhone screen. But the VP Marketing couldn’t care less – she/he wants to know how this mobile survey will help them better understand consumer needs. Consider the following App, which allows an iPhone user to scan a barcode, and get product rating information. This is where I believe mobile gets exciting for researchers, by linking real-time consumer behavior with attitudinal research communities (MROC’s). And dare I say, leveraging the power of the channel for exciting Qual opportunities from a new generation of Internet-savvy ethnographic researchers.

So, I say:  Dear <insert client name>, the next time I’m sitting on a plane, use my BlackBerry GPS locator (you can access it) and Frequent Flyer number (you have it) to send me a 5 question CSAT survey. I promise to respond. Sincerely, Bernie.

Note: element54 will be presenting new Mobile R&D – “The Moment of Truth” at the MRIA’s 50th annual National Conference on Monday May 30/2010. Stay tuned for details, and visit the conference website.

Links to Annie, Joel, Josh and Brandon (each to be updated as soon as available)

Annie Pettit of Lovestats: http://lovestats.wordpress.com/2010/01/15/1topic5blogs-the-only-thing-cell-phone-surveys-are-good-fer/

Joel Rubinson of the ARF:
Josh Mendelsohn of Chadwick Martin Bailey: http://lovestats.wordpress.com/2010/01/15/1topic5blogs-the-only-thing-cell-phone-surveys-are-good-fer/
Brandon Bertelsen:

Our first topic, is something which has been of utmost importance and urgency to element54 this year, and one where my partners and I have invested considerably, to advance the issue in our business (some previous posts on this site offer practical & prescriptive solutions).

Background/Issue:

We are witnessing an explosion of engaging new question types – intended to captivate respondents, and ensure the long-term viability of an online survey platform.

Research to-date has described “respondent engagement” as the notion that improving the usability and interest of Online question types will enhance respondents’ enjoyment, and the thoughtfulness of their survey experience. However, there has been little work to validate the myriad of new question types, and specifically, how response patterns vary across each type.

element54 conducted 2 transparent studies in 2009, which have been presented at several North American conferences (most recently, The Market Research Event, Las Vegas – Oct ’09)

Study #1 – “Sexy Questions, Dangerous Results?” (Data Consistency):

In January 2009, element54 and ResearchNow, conducted the first-ever fully transparent research study (full dataset available to MR peers), which examined the response patterns to various Online question types. The findings from this 2000 interview study were clear. Changing the visual layout of an online survey leads to differences in the data.

• 36% variance in how often people say they brush their teeth,
• Up to 8% overclaiming of behavioral product category consumption,
• Up to 10% understating of attitudinal “issue importance”.

Study #2 – “Eyes Don’t Lie” (Respondent Usability):

In May 2009, element54, along with partners UX Research & Consulting, and MBA Research, conducted the largest transparent “Eye Tracking” study on survey usability, with 100 one-on-one Qual/Quant interviews, using SMI Vision eye tracking software to validate respondent eye movements & patterns. The findings from this second study leveraged how Usability insights and applications can be applied to designing better surveys:

• Inference vs. Instruction; up to 40% of “2.0” survey questions aren’t read when visuals dominate the screen. It is certainly concerning if respondents are inferring our intent.
• Error Prevention & Recovery; ever tried to get off an elevator when you miss your floor – yes, you push other buttons to get off. When survey “error messages” are not linked to where the problem is, respondents are likely to change their answers to escape. This issue raises a host of related questions around how we treat “error” data once the respondent successfully clicks “Next”.

Next Steps:

Technology is headed in the right direction. To sustain the long-term viability of Online surveys, we do need to create appropriately engaging platforms for respondents. However, in the current space race, and range of available DIY software platforms – there’s no glory in getting to the moon first, if you crash land.  This is where best practices and standards can, and must catch up with all this exciting technology.

Here are the links to Annie, Joel, Josh and Brandon (soon).

Annie Pettit of Lovestats: http://lovestats.wordpress.com/2009/12/21/1-topic-5-blogs-rich-media-in-surveys/
Joel Rubinson of the ARF: http://blog.joelrubinson.net/2009/12/getting-the-most-out-of-online-research/
Josh Mendelsohn of Chadwick Martin Bailey:http://betterresearch.blogspot.com/2009/12/1-topic-5-blogs-interactive-questions.html
Brandon Bertelsen: link posted shortly at http://bertelsen.ca/

In the largest Usability study conducted on this issue (“Eyes Don’t Lie” – 2009), we have come across many ways in which marketing researchers likely compound the issue of respondent errors. The above image shows a sample “error message” from one of the studies’ interfaces. Note that the errors are highlighted with an arrow, and text saying “your answer is incomplete”.

• From a Usability point of view, this is what needs to be done – the respondent is directed to the specific problem lines where they had omitted an answer (which is much better than an error message on top of the screen saying “you must complete all questions below”).
• However, what is problematic is the visual cue to potentially ‘tick’ the answer that is closest to the red arrow (as the video of this respondent does show to be the case).

Recommendation: highlight the row, or indicate at the question/statement level where there is an omission or error. Anything else is likely adding error on top of error.

element54 has registered with the Foundation for Transparency in Offshoring (FTO). We chose to declare a “position” early in this discussion for 2 reasons:

(1) We believe that MR companies should be transparent with clients about how we handle their projects. Regardless of why someone chooses to offshore (cost, speed, or any other reason), it is our stated position to always be accountable to our end-clients about where THEIR data is, at all phases of the Marketing Research process.

(2) element54 declared a position of “responsible offshoring” because we live in a global village, and we have learned  you can find trusted business partners around the world, regardless of distance or culture. We’ve worked with recruiters in Asia, online sample providers throughout East/West Europe … and more recently, a firm in India. The experience was positive, our expectations were clearly aligned, and the end-result was successful.

And, to be clear (and transparent) – our client was made aware of exactly what was being offshored, and the credentials of the company we had chosen to work with.

In the past couple of years, surveys have become increasingly visual & eye-catching, with the goal of “engaging” respondents.

Our research into this issue, presented around North America – and most recently at The Market Research Event in Las Vegas, has shown that up to 42% of respondents don’t read marketing research survey questions, and highest when visuals dominated the screen.

What can we do? For starters, a simple tip –> ensure the font is large enough (often, survey software has default font sizes that are too small) … and use BOLD and underline in your questions to highlight important information. For example: “Which of these three ads did you prefer most?”

Next time you get a survey link … look beyond typos and skip logic … look for ways to also make sure respondents are clear on what you are asking.

During the programming of a recent ad test, the “ad preference” question was programmed vertically – ie. 3 print ads were shown, but only 1 visible at a time, and 2 could only be seen if the respondent scrolled down. I’ll save the Usability lesson on ‘Task Workload” for another post, but wanted to ask this:

If what we are asking a respondent to rate, or comment on is not visible without scrolling up, down and all around – is that a source of potential bias?

To mitigate this, make sure that what you are showing/asking fits on one screen. In the case of this print ad test, I asked the programmer to resize the images so they could appear horizontally (and rotated) onscreen, so no scrolling was required for respondents to evaluate their options.

In dozens of presentations, to hundreds of researchers and marketers this year – the issue of data consistency has gained lots of attention.  That’s because it is incumbent upon all of us to deploy the best solution (both research design, and technology) for our clients.

Most researchers are adept at the ’science’ of what we do – but what about the ‘art’. It is this latter aspect which has been transformed in recent years, thanks to advances in rich media technology.

Case in point – our firm asked 2000 people the following: “are you an optimist or a pessimist?”. We observed answers which ranged from 5% to 21% (agreement of “10″ on a 1-10 scale), depending on the survey interface (a ‘radio button’, or a ‘visual slider’ scale).

So, your choice of how to design the visual aspect of a question for your survey will have an impact on the results. The question is by how much. Stay tuned for more tips and learning.

I’ve shown a video during many presentations this year – of Joe, a typical respondent who had an unfortunate experience trying handle a complex question. It shows the last portion of the 5 minutes that were spent on a multiple choice grid question (since not all MR questions are yes/no). The findings highlight important usability learning, but also important DATA QUALITY considerations – what to do with data when they have clearly changed their answers after not being able to click “next”? Stay tuned for more usability tips, and contact us to learn more.