Last night (with 2 of my 3 kids just about ready for bedtime), I somewhat reluctantly took a telephone survey.  Given that it was a seemingly nice & polite interviewer, on a topic of interest (Travel), by a reputable (Top 10) research firm, and a promised short (5-6 mins), I agreed (Note: I was not asked any disqualification questions).

• After about 8 mins, I asked: “approximately how many questions do you have left?”
• Interviewer: “we’re almost done, and I’ve been trying to get someone to qualify for 3 hours, so this is a really important interview for me to complete” (Red flag alert).
• After about 10 mins, I ask again: “are we almost done, this is longer than you had said it would be”.
• After 12 mins, I politely say “if we are not on our last 2-3 question, I am sorry, but must now end our call”.

I remember being trained as a junior researcher in the late 80′s, and spending 3 nights as a telephone interviewer to round out my training.  It was hard back then, dealing with the challenge of making a 15-20 minute survey about cookies or toilet paper hold interruption value during the golden hours of 5-8pm … and I recall developing a real appreciation for the front line of survey research.

Fast forward to 2010, and I can appreciate that the job of a Telephone interviewer is no less easy, and I’m sure even harder.  Consumers (Note: not “respondents”) are increasingly unwilling to answer the phone, and when they do, are certainly less and less interested to give up their time.

Conclusion #1: All forms of Survey Research are based upon terms & trust.  I was let down, because someone thought that I could be snagged with a promise, and out of guilt would finish the interview.  That was bad for all involved, and made me wonder how many people have had similar experiences, which impact all of us doing survey research.

Conclusion #2: Perhaps nostalgically speaking, I still believe there is a future for telephone surveys, but one where researchers cannot rely on the same tools, approaches and interviewer scripts that existed in the 1970′s and 80′s.  Doing so, will just hasten the demise of that methodological option in the survey toolkit.

The question posed to the group of 5 Bloggers this month was: “Qualitative Research: How has the art of listening changed, and what can we do to leverage new listening tools in a Social Media landscape.”

Links to my fellow bloggers Annie Pettit, Joel Rubinson, Josh Mendelsohn and Brandon Bertelsen can be found below.

An impetus for the selection of this topic was an as-usually well written post by Joel Rubinson on “Transforming Research Through Listening”.

“Listening for the unexpected should be at the heart of the innovation process.  It takes research from the back end and places it squarely at the front end.  It says our role doesn’t kick in only when the marketing team is ready to “order up” a concept test, a commercial test, etc.  Our role is to anticipate the next move of consumers and to help the marketing teams turn that into innovation (Joel Rubinson, Dec 11/09)

There is an oft-cited stat that “80% of usable business information originates in an unstructured form” (Source: Merrill Lynch). Whether there is support of not for this figure, the art of listening has evolved by giving structure, to the unstructured.

I believe there are 2 areas where Market Research solutions will, and must rapidly evolve in the next 12-24 months:

“Mobile Listening”; location based social networks like Brightkite, GyPSii, Pelago and Loopt will generate over $3.0 billion of revenue by 2013. The tie-in between these services, and the evolution of listening should be of great interest to researchers.

Mobile Ethnography, and other evolved forms of Qualitative Research have been empowered by such mobile platforms. While these options by no means replace traditional Qualitative solutions, the expanded toolkit offers us an unparalleled perspective at the consumer “moment of truth”, and unlocking real-time motivations for purchase decisions will significantly enhance the value and visibility of Market Research within the client organization.

“Social Media Listening”; there are no shortage of listening solutions available to marketers, and I was intrugued by a blog from last year which asked: “How valuable are qualitative and personal focus groups compared to online sentiment analysis”?  The notion that these are potentially swappable methodologies raises a pile of issues and flags, but does point to the practical reality that the lines between options continue to blur. Those trained in the art of listening, are now able to mine through thousands of Qualitative comments, to help build the brand story, and glean tremendously valuable insights.

Aggregated metrics like sentiment analysis are interesting, in the way that a Net Promoter score is interesting – it is easy to explain, and can be trended over time. Yet, on its’ own, the metric offers only part of the brand story. Those who capture the online, volunteered conversations through the new paradigm lens of survey research will help their clients uncover incredibly valuable learnings about topics we haven’t asked about in our traditional survey methods.

New and exciting insights will come from going deeper, exploring the moment of truth, and structuring those unstructured Qualitative conversations.

More to come from element54 on both these topics, and please visit my fellow bloggers and their perspectives on this issue.

• Annie Petit, Chief Research Officer at Conversition: post coming shortly at http://lovestats.wordpress.com/
• Brandon Bertelsen: post coming shortly at  http://www.bertelsen.ca/
• Josh Mendelsohn, VP Marketing at Chadwick Martin Bailey: post coming shortly at  http://blog.cmbinfo.com/bid/36062/1-Topic-5-Blogs-Qualitative-Market-Research-in-2012
• Joel Rubinson, Chief Research Officer at The ARF: post coming shortly at http://blog.joelrubinson.net/

The question posed to the group of 5 Bloggers this month was: “The survey monkey conundrum:  the upsides, downsides and opportunities for researchers that access to self survey tools creates.” Links to my fellow bloggers Annie Pettit, Joel Rubinson, Josh Mendelsohn and Brandon Bertelsen can be found below.

Ray Poynter used the term “Doing the Monkey” at the recent NetGain Toronto conference, and the larger issue of how “survey independence” is changing the research landscape. The serious side of this topic, is the growing trend of “Do It Yourself” Market Research, spawned by exciting new technology/software, in both the Qualitative and Quantitative fields.

Some have framed this discussion as Research Professionals vs. Non-Professionals, and who should be allowed to have the ‘keys to the car’. Over the past few years, some of my clients have deployed their own proprietary panels, others continue to rely on outside research agencies. There is no right or wrong … rather, just “Good or Bad research”.

In a recent LinkedIn thread posted by Ray Poynter (NewMR – Co-Creating the Future of Market Research), Mark Kupferman (Director of Consumer Insights, Universal Orlando) asked:

• “One has to ask where professional research ends and where “DIY” research begins. If I’m at a company and I do my own research with my own customers and/or with rented panels … am I doing DIY research because I’m not using an external research agency to write the questions and do the analysis? What if I or the people I hire have worked in a professional research firm in the past? And what is it that makes professional researchers better at executing research than an internal team of researchers?”

There is a time and a place for both DIY, and ‘Assisted’ projects … and there is both “Good” and “Bad” research being conducted. I’d encourage you to visit Jeffrey Hennings’ blog at Vovici, who has posted more on this topic than anyone else http://blog.vovici.com/blog/?Tag=Survey%20Monkey including this potential warning:

If you are a Client-Side researcher, Kathryn Korostoff (AKA, Research Rockstar) did an excellent piece summarizing the DIY short-list to ask yourself http://www.researchrockstar.com/diy-or-hire-a-market-research-company/

• Are you confident that you can write a questionnaire such that it will capture information objectively.
• You realistically have time to do the project management in-house
• You have the tools and skills in-house to clean the collected data and analyze it
• You have resources in-house that can report the findings in a way that will be credible to your internal colleagues

If you are a Supplier-Side researcher, stop dissing DIY platforms just because they are “DIY” instead of full service, and focus on evolving a solutions toolbox that gives clients your expertise, in situations where they have reframed the costing model for how you can add value to their decision making (ie – that “full serve” project might now become a consulting mandate). Focus on adding value, helping to develop/build question libraries, conduct advanced statistical analyses of Attitudinal & Behavioral databases, and help those clients establish best practices. This paradigm shift is happening for lots of different reasons, not just economics. Embrace the wave.

If you are a Software Provider, I believe it is time to practice “responsible innovation”. Having led 2 research-on-research studies over the past year (Sexy Questions, Dangerous Results and Eyes Don’t Lie), it’s clear that software developers must re-take the high ground and guide their end-users with how and why to choose different question types. Most researchers lack the Usability experience to guide clients on how/when different question types are best used, and the risks associated with survey layout.

But, at the heart of every good survey is a good question … and we can’t blame software providers when someone just asks a bad question …

Read the other blogs:

• Annie Petit (AKA “LoveStats”) at Conversition Strategies: http://lovestats.wordpress.com/2010/02/14/1-topic-5-blogs-diy-surveys/
• Josh Mendelsohn of Chadwick Martin Bailey http://blog.cmbinfo.com/bid/33687/1-Topic-5-Blogs-Why-market-research-professionals-should-embrace-DIY-Surveys
• Brandon Bertelsen: http://www.bertelsen.ca/market-research/1-topic-5-blogs-diy-surveys
• Joel Rubinson of the ARF: post coming shortly at http://blog.joelrubinson.net/

I learned that Jeffrey Henning (@JHenning) of Vovici and I have something in common – we both have 16 year olds at home. I was recounting a recent story, and decided to share, you know – for the market research implications.

My eldest has been studying hard lately. I was on my way home, and decided to pop into a Pizza place to pick up for him and his friend who was over at our house studying w/him.

Price/Inventory: If you call ahead, a large veggie pizza goes for $7.99. As it turned out, they they had one under the warmer, except that it was missing 2 slices.

Conversation with Pizza Guy:

• Me says: I’ll take that partial large pizza (6 slices) instead of 8 slices … And pay $7.99.
• Pizza dude: sorry, I have to charge you $2.25 per slice (x 6 = $13.50).
• Me says: but I’ll pay you full price for only 75% of that pizza.
• Pizza dude: sorry, if you want a whole pizza –  I can make one for you in 15 minutes.
• Me says: thanks, but sorry dude, gotta go.
  

So how much is 75% of a pizza worth?

Technically, I think he was correct, and I admired his holding the party line on the “pay per slice” take-out-premium-pricing model. Yet, it was an interesting Customer Service experience – since the opportunity to generate a guaranteed $8, and still have another full pie out within 15 minutes (on a not so busy night) was lost.

Mr. Pizza Dude, I will be back again, and might even call ahead next time.

The question posed to the group of 5 Bloggers this month was: “Mobile surveys – For/Against,
Pros/Cons, Right Situations/Wrong Situations?” Links to my fellow bloggers Annie Pettit,
Joel Rubinson, Josh Mendelsohn and Brandon Bertelsen can be found below.

There’s no debating the societal impact of Mobile: Nokia stated at this weeks’ CES conference that now, over 50% of internet services are accessed on mobile phones, not from PCs. And, we all know that the level of mobile-only continues to rise (well past 20% in some demographic segments), forever changing landline survey options.

So what does that mean for Survey Researchers? Should each of us have our own $.99 survey App on iTunes? Not so fast, let’s think about this from a few perspectives.

Consistency: if the objective is to empower respondents with another mode of completing surveys, mobile certainly offers some interesting benefits. But as we’ve learned from the results of 1.0 vs. 2.0 web-based surveys (“Sexy Questions Dangerous Results?”), there will certainly be differences in the data when comparing mobile to web – and a range of new issues for which we have limited best practices (sampling, questionnaire length, visual layout & usability differences between BlackBerry and iPhones etc). As with each of the new tech options/advances, it’s critical to think through the “when, why and how” before we push the send button. As an industry, let’s avoid the temptation to over-focus on the ‘consistency’ of survey learning from Mobile vs PC, it’ll just get us into trouble, and distract us from creating end-client insight value from this exciting new channel.

Context: the benefits of using a mobile mode (either exclusively, or in conjunction with other methodologies) needs to take into account the contextual relevancy of using this platform. If you’re trying to get conference attendees to tell you what they thought of the speaker they just heard, that would be a intuitive application of a mobile platform. But don’t try to be cute, and offer people the choice to do a PC or mobile survey. A recent client survey found that under 5% of respondents chose the mobile option when given the choice between PC & mobile. And, you incur additional programming costs/delays to deploy the additional mode, for limited response rate upside, and questionable data consistency downside.

Future Focus: I recognize the research importance of making sure we know whether the “Next” button should be on the top or bottom of an iPhone screen. But the VP Marketing couldn’t care less – she/he wants to know how this mobile survey will help them better understand consumer needs. Consider the following App, which allows an iPhone user to scan a barcode, and get product rating information. This is where I believe mobile gets exciting for researchers, by linking real-time consumer behavior with attitudinal research communities (MROC’s). And dare I say, leveraging the power of the channel for exciting Qual opportunities from a new generation of Internet-savvy ethnographic researchers.

So, I say:  Dear <insert client name>, the next time I’m sitting on a plane, use my BlackBerry GPS locator (you can access it) and Frequent Flyer number (you have it) to send me a 5 question CSAT survey. I promise to respond. Sincerely, Bernie.

Note: element54 will be presenting new Mobile R&D – “The Moment of Truth” at the MRIA’s 50th annual National Conference on Monday May 30/2010. Stay tuned for details, and visit the conference website.

Links to Annie, Joel, Josh and Brandon (each to be updated as soon as available)

Annie Pettit of Lovestats: http://lovestats.wordpress.com/2010/01/15/1topic5blogs-the-only-thing-cell-phone-surveys-are-good-fer/

Joel Rubinson of the ARF:
Josh Mendelsohn of Chadwick Martin Bailey: http://lovestats.wordpress.com/2010/01/15/1topic5blogs-the-only-thing-cell-phone-surveys-are-good-fer/
Brandon Bertelsen:

I am privileged to be one of 5 bloggers who, each 15th of the month – will produce a POV on an issue facing the Marketing Research industry. You’ll also be hearing from Annie Pettit (organizer), Josh Mendelsohn, Joel Rubinson and Brandon Bertelsen. Links to their posts will follow.

Our first topic, is something which has been of utmost importance and urgency to element54 this year, and one where my partners and I have invested considerably, to advance the issue in our business (some previous posts on this site offer practical & prescriptive solutions).

Background/Issue:

We are witnessing an explosion of engaging new question types – intended to captivate respondents, and ensure the long-term viability of an online survey platform.

Research to-date has described “respondent engagement” as the notion that improving the usability and interest of Online question types will enhance respondents’ enjoyment, and the thoughtfulness of their survey experience. However, there has been little work to validate the myriad of new question types, and specifically, how response patterns vary across each type.

element54 conducted 2 transparent studies in 2009, which have been presented at several North American conferences (most recently, The Market Research Event, Las Vegas – Oct ’09)

Study #1 – “Sexy Questions, Dangerous Results?” (Data Consistency):

In January 2009, element54 and ResearchNow, conducted the first-ever fully transparent research study (full dataset available to MR peers), which examined the response patterns to various Online question types. The findings from this 2000 interview study were clear. Changing the visual layout of an online survey leads to differences in the data.

• 36% variance in how often people say they brush their teeth,
• Up to 8% overclaiming of behavioral product category consumption,
• Up to 10% understating of attitudinal “issue importance”.

Study #2 – “Eyes Don’t Lie” (Respondent Usability):

In May 2009, element54, along with partners UX Research & Consulting, and MBA Research, conducted the largest transparent “Eye Tracking” study on survey usability, with 100 one-on-one Qual/Quant interviews, using SMI Vision eye tracking software to validate respondent eye movements & patterns. The findings from this second study leveraged how Usability insights and applications can be applied to designing better surveys:

• Inference vs. Instruction; up to 40% of “2.0” survey questions aren’t read when visuals dominate the screen. It is certainly concerning if respondents are inferring our intent.
• Error Prevention & Recovery; ever tried to get off an elevator when you miss your floor – yes, you push other buttons to get off. When survey “error messages” are not linked to where the problem is, respondents are likely to change their answers to escape. This issue raises a host of related questions around how we treat “error” data once the respondent successfully clicks “Next”.

Next Steps:

Technology is headed in the right direction. To sustain the long-term viability of Online surveys, we do need to create appropriately engaging platforms for respondents. However, in the current space race, and range of available DIY software platforms – there’s no glory in getting to the moon first, if you crash land.  This is where best practices and standards can, and must catch up with all this exciting technology.

Here are the links to Annie, Joel, Josh and Brandon (soon).

Annie Pettit of Lovestats: http://lovestats.wordpress.com/2009/12/21/1-topic-5-blogs-rich-media-in-surveys/
Joel Rubinson of the ARF: http://blog.joelrubinson.net/2009/12/getting-the-most-out-of-online-research/
Josh Mendelsohn of Chadwick Martin Bailey:http://betterresearch.blogspot.com/2009/12/1-topic-5-blogs-interactive-questions.html
Brandon Bertelsen: link posted shortly at http://bertelsen.ca/

So what does your “error message” look like when a respondent makes a mistake?

In the largest Usability study conducted on this issue (“Eyes Don’t Lie” – 2009), we have come across many ways in which marketing researchers likely compound the issue of respondent errors. The above image shows a sample “error message” from one of the studies’ interfaces. Note that the errors are highlighted with an arrow, and text saying “your answer is incomplete”.

• From a Usability point of view, this is what needs to be done – the respondent is directed to the specific problem lines where they had omitted an answer (which is much better than an error message on top of the screen saying “you must complete all questions below”).
• However, what is problematic is the visual cue to potentially ‘tick’ the answer that is closest to the red arrow (as the video of this respondent does show to be the case).

Recommendation: highlight the row, or indicate at the question/statement level where there is an omission or error. Anything else is likely adding error on top of error.

Data suggests an increasing number of companies are either already, or considering, offshoring different aspects of the MR process. This is a complex issue, with many angles & perspectives.

element54 has registered with the Foundation for Transparency in Offshoring (FTO). We chose to declare a “position” early in this discussion for 2 reasons:

(1) We believe that MR companies should be transparent with clients about how we handle their projects. Regardless of why someone chooses to offshore (cost, speed, or any other reason), it is our stated position to always be accountable to our end-clients about where THEIR data is, at all phases of the Marketing Research process.

(2) element54 declared a position of “responsible offshoring” because we live in a global village, and we have learned  you can find trusted business partners around the world, regardless of distance or culture. We’ve worked with recruiters in Asia, online sample providers throughout East/West Europe … and more recently, a firm in India. The experience was positive, our expectations were clearly aligned, and the end-result was successful.

And, to be clear (and transparent) – our client was made aware of exactly what was being offshored, and the credentials of the company we had chosen to work with.

In the field of usability, there is the notion of “Inference vs. Instruction” – ie, how well did the instructions we have provided lead everyone to explicitly understand the same set of parameters in how to answer the question.

In the past couple of years, surveys have become increasingly visual & eye-catching, with the goal of “engaging” respondents.

Our research into this issue, presented around North America – and most recently at The Market Research Event in Las Vegas, has shown that up to 42% of respondents don’t read marketing research survey questions, and highest when visuals dominated the screen.

What can we do? For starters, a simple tip –> ensure the font is large enough (often, survey software has default font sizes that are too small) … and use BOLD and underline in your questions to highlight important information. For example: “Which of these three ads did you prefer most?”

Next time you get a survey link … look beyond typos and skip logic … look for ways to also make sure respondents are clear on what you are asking.

Had a discussion with a client the other day over “vertical” vs. “horizontal” layout of online questions. We were discussing potential issues around the “never” option always appearing off the screen, requiring scrolling to see that answer choice.

During the programming of a recent ad test, the “ad preference” question was programmed vertically – ie. 3 print ads were shown, but only 1 visible at a time, and 2 could only be seen if the respondent scrolled down. I’ll save the Usability lesson on ‘Task Workload” for another post, but wanted to ask this:

If what we are asking a respondent to rate, or comment on is not visible without scrolling up, down and all around – is that a source of potential bias?

To mitigate this, make sure that what you are showing/asking fits on one screen. In the case of this print ad test, I asked the programmer to resize the images so they could appear horizontally (and rotated) onscreen, so no scrolling was required for respondents to evaluate their options.