LEARN:

Make a pledge to learn something new, by reaching out across departments, and/or taking a professional development course via any of our trade associations (MRA, MRIA, CASRO etc).  Many offer online courses, and many firms hold regular expert webinars (eg – Vovici, Peanut Labs, Communispace).

TEACH:

If you possess subject matter expertise, share it with others within your organization, and/or with your industry peers.  Sharing knowledge is a rewarding experience on so many levels.  Take it a step further, and seek out teaching opportunities at your local colleges/Universities, which helps us inspire the next generation of Market Researchers. I frequently guest-lecture at McGill University on Research/Marketing issues, and was thrilled to see Annie Pettit from Conversition Strategies guest lecture at Georgian College last week.

So beyond the earlier wake-ups and after school homework, how will you “Go Back to School?”

The core message of “Sexy Questions, Dangerous Results?” has been the importance of understanding when and how to leverage rich media survey technology (see previous posts on this site for other examples).

In recent presentations, I’ve shared some examples from our Eye Tracking survey, where survey participants were asked to complete the same question, in either a “1.0″ or “2.0″ format.  What is clear from both, is the importance of better understanding the role of Usability in designing the survey interface, and the new complexities of the survey toolbox.

To illustrate the point, I’ve shared 2 videos below (click on 2.o Survey … and 1.o Survey to see them play).

2.0 Survey In the first video (a “2.0″ or enhanced survey interface), our survey participant tries in vain to complete the task of a complex question, but keeps getting messaged “Please select 9 Answers”. What results is a series of errors, an almost random assignment of answers just to try to get out of the question, and needless to say … questionnable data.  This in turn raises additional issues around how to treat data from a multiple error survey (more on that in another post).

1.0 Survey In the second video (a “1.0″ or basic survey interface), our survey participant is faced with 3 unanswered questions.  The error messaging with this interface shows  a RED ARROW message saying “<—– Your Answer is Incomplete”, which in all 3 error messages, our participant assumed meant “you should now click on the closest answer to this message”. Another clear issue of not knowing how to respond to an error message, and picking up on a visual cue.

IMPLICATIONS:

The digital divide in our industry has created opportunities to engage and excite survey participants who we rely on to complete online surveys.  However, the Market Research toolbox is filled with some new power tools that not everyone is familiar with, and for which it is incumbent upon all of us to develop best practices and standards. Thoughts? Email bernie.malinoff@element-54DOTcom with any thoughts.

After a fascinating call today with a client, I was intrigued by their question “Do Norms Matter?”

As we talked about the issue, I was struck by how ingrained the “norms” were as milestones that were either missed, on-norm, or achieved. For this client, norms have become a crutch – where corporate intuition, experience and judgment are secondary to the “normative” threshold.

As an example of their concern, “norms” are rarely challenged, and more likely to be taken as gospel.

Question #1: do the norms always make sense? eg: for a Cheese client, are all the norms from a Category Leader, or is there dispersion amongst the cases – to include other brands in the category.  What if you’re a smaller share marketer, and all/most of the norms are from the category leader (who is more likely to invest in repeat-type methodologies that sustain normative databases than the smaller brands anyways). Are norms as relevant/applicable if you’re being compared to the category leader?

Question #2: how are the norms actioned? Meaning, is the Norm Bar set at 50% (ie an average of the cases), or higher (eg. Top 25% of Cases).  Hopefully, your brand is being compared to some higher standard as a success threshold.

Question #3: have you asked for recent validation? Many of the “norms” quoted today source from the 80′s and 90′s.  In some cases (eg: advertising), norms might not be as relevant today as they were 20 years ago.  It’s a fair question to ask how some norms have changed over time, and to be certain that the standards you are trying to reach, are based on current market conditions/relevancy.

In your experience, when do “norms” matter – and where have they become less relevant?

PS – the guy in the pic is “Norm” (AKA George Wendt) from Cheers.

• After about 8 mins, I asked: “approximately how many questions do you have left?”
• Interviewer: “we’re almost done, and I’ve been trying to get someone to qualify for 3 hours, so this is a really important interview for me to complete” (Red flag alert).
• After about 10 mins, I ask again: “are we almost done, this is longer than you had said it would be”.
• After 12 mins, I politely say “if we are not on our last 2-3 question, I am sorry, but must now end our call”.

I remember being trained as a junior researcher in the late 80′s, and spending 3 nights as a telephone interviewer to round out my training.  It was hard back then, dealing with the challenge of making a 15-20 minute survey about cookies or toilet paper hold interruption value during the golden hours of 5-8pm … and I recall developing a real appreciation for the front line of survey research.

Fast forward to 2010, and I can appreciate that the job of a Telephone interviewer is no less easy, and I’m sure even harder.  Consumers (Note: not “respondents”) are increasingly unwilling to answer the phone, and when they do, are certainly less and less interested to give up their time.

Conclusion #1: All forms of Survey Research are based upon terms & trust.  I was let down, because someone thought that I could be snagged with a promise, and out of guilt would finish the interview.  That was bad for all involved, and made me wonder how many people have had similar experiences, which impact all of us doing survey research.

Conclusion #2: Perhaps nostalgically speaking, I still believe there is a future for telephone surveys, but one where researchers cannot rely on the same tools, approaches and interviewer scripts that existed in the 1970′s and 80′s.  Doing so, will just hasten the demise of that methodological option in the survey toolkit.

Links to my fellow bloggers Annie Pettit, Joel Rubinson, Josh Mendelsohn and Brandon Bertelsen can be found below.

An impetus for the selection of this topic was an as-usually well written post by Joel Rubinson on “Transforming Research Through Listening”.

“Listening for the unexpected should be at the heart of the innovation process.  It takes research from the back end and places it squarely at the front end.  It says our role doesn’t kick in only when the marketing team is ready to “order up” a concept test, a commercial test, etc.  Our role is to anticipate the next move of consumers and to help the marketing teams turn that into innovation (Joel Rubinson, Dec 11/09)

There is an oft-cited stat that “80% of usable business information originates in an unstructured form” (Source: Merrill Lynch). Whether there is support of not for this figure, the art of listening has evolved by giving structure, to the unstructured.

I believe there are 2 areas where Market Research solutions will, and must rapidly evolve in the next 12-24 months:

“Mobile Listening”; location based social networks like Brightkite, GyPSii, Pelago and Loopt will generate over $3.0 billion of revenue by 2013. The tie-in between these services, and the evolution of listening should be of great interest to researchers.

Mobile Ethnography, and other evolved forms of Qualitative Research have been empowered by such mobile platforms. While these options by no means replace traditional Qualitative solutions, the expanded toolkit offers us an unparalleled perspective at the consumer “moment of truth”, and unlocking real-time motivations for purchase decisions will significantly enhance the value and visibility of Market Research within the client organization.

“Social Media Listening”; there are no shortage of listening solutions available to marketers, and I was intrugued by a blog from last year which asked: “How valuable are qualitative and personal focus groups compared to online sentiment analysis”?  The notion that these are potentially swappable methodologies raises a pile of issues and flags, but does point to the practical reality that the lines between options continue to blur. Those trained in the art of listening, are now able to mine through thousands of Qualitative comments, to help build the brand story, and glean tremendously valuable insights.

Aggregated metrics like sentiment analysis are interesting, in the way that a Net Promoter score is interesting – it is easy to explain, and can be trended over time. Yet, on its’ own, the metric offers only part of the brand story. Those who capture the online, volunteered conversations through the new paradigm lens of survey research will help their clients uncover incredibly valuable learnings about topics we haven’t asked about in our traditional survey methods.

New and exciting insights will come from going deeper, exploring the moment of truth, and structuring those unstructured Qualitative conversations.

More to come from element54 on both these topics, and please visit my fellow bloggers and their perspectives on this issue.

• Annie Petit, Chief Research Officer at Conversition: post coming shortly at http://lovestats.wordpress.com/
• Brandon Bertelsen: post coming shortly at  http://www.bertelsen.ca/
• Josh Mendelsohn, VP Marketing at Chadwick Martin Bailey: post coming shortly at  http://blog.cmbinfo.com/bid/36062/1-Topic-5-Blogs-Qualitative-Market-Research-in-2012
• Joel Rubinson, Chief Research Officer at The ARF: post coming shortly at http://blog.joelrubinson.net/

Ray Poynter used the term “Doing the Monkey” at the recent NetGain Toronto conference, and the larger issue of how “survey independence” is changing the research landscape. The serious side of this topic, is the growing trend of “Do It Yourself” Market Research, spawned by exciting new technology/software, in both the Qualitative and Quantitative fields.

Some have framed this discussion as Research Professionals vs. Non-Professionals, and who should be allowed to have the ‘keys to the car’. Over the past few years, some of my clients have deployed their own proprietary panels, others continue to rely on outside research agencies. There is no right or wrong … rather, just “Good or Bad research”.

In a recent LinkedIn thread posted by Ray Poynter (NewMR – Co-Creating the Future of Market Research), Mark Kupferman (Director of Consumer Insights, Universal Orlando) asked:

• “One has to ask where professional research ends and where “DIY” research begins. If I’m at a company and I do my own research with my own customers and/or with rented panels … am I doing DIY research because I’m not using an external research agency to write the questions and do the analysis? What if I or the people I hire have worked in a professional research firm in the past? And what is it that makes professional researchers better at executing research than an internal team of researchers?”

There is a time and a place for both DIY, and ‘Assisted’ projects … and there is both “Good” and “Bad” research being conducted. I’d encourage you to visit Jeffrey Hennings’ blog at Vovici, who has posted more on this topic than anyone else http://blog.vovici.com/blog/?Tag=Survey%20Monkey including this potential warning:

If you are a Client-Side researcher, Kathryn Korostoff (AKA, Research Rockstar) did an excellent piece summarizing the DIY short-list to ask yourself http://www.researchrockstar.com/diy-or-hire-a-market-research-company/

• Are you confident that you can write a questionnaire such that it will capture information objectively.
• You realistically have time to do the project management in-house
• You have the tools and skills in-house to clean the collected data and analyze it
• You have resources in-house that can report the findings in a way that will be credible to your internal colleagues

If you are a Supplier-Side researcher, stop dissing DIY platforms just because they are “DIY” instead of full service, and focus on evolving a solutions toolbox that gives clients your expertise, in situations where they have reframed the costing model for how you can add value to their decision making (ie – that “full serve” project might now become a consulting mandate). Focus on adding value, helping to develop/build question libraries, conduct advanced statistical analyses of Attitudinal & Behavioral databases, and help those clients establish best practices. This paradigm shift is happening for lots of different reasons, not just economics. Embrace the wave.

If you are a Software Provider, I believe it is time to practice “responsible innovation”. Having led 2 research-on-research studies over the past year (Sexy Questions, Dangerous Results and Eyes Don’t Lie), it’s clear that software developers must re-take the high ground and guide their end-users with how and why to choose different question types. Most researchers lack the Usability experience to guide clients on how/when different question types are best used, and the risks associated with survey layout.

But, at the heart of every good survey is a good question … and we can’t blame software providers when someone just asks a bad question …

Read the other blogs:

• Annie Petit (AKA “LoveStats”) at Conversition Strategies: http://lovestats.wordpress.com/2010/02/14/1-topic-5-blogs-diy-surveys/
• Josh Mendelsohn of Chadwick Martin Bailey http://blog.cmbinfo.com/bid/33687/1-Topic-5-Blogs-Why-market-research-professionals-should-embrace-DIY-Surveys
• Brandon Bertelsen: http://www.bertelsen.ca/market-research/1-topic-5-blogs-diy-surveys
• Joel Rubinson of the ARF: post coming shortly at http://blog.joelrubinson.net/

I learned that Jeffrey Henning (@JHenning) of Vovici and I have something in common – we both have 16 year olds at home. I was recounting a recent story, and decided to share, you know – for the market research implications.

My eldest has been studying hard lately. I was on my way home, and decided to pop into a Pizza place to pick up for him and his friend who was over at our house studying w/him.

Price/Inventory: If you call ahead, a large veggie pizza goes for $7.99. As it turned out, they they had one under the warmer, except that it was missing 2 slices.

Conversation with Pizza Guy:

• Me says: I’ll take that partial large pizza (6 slices) instead of 8 slices … And pay $7.99.
• Pizza dude: sorry, I have to charge you $2.25 per slice (x 6 = $13.50).
• Me says: but I’ll pay you full price for only 75% of that pizza.
• Pizza dude: sorry, if you want a whole pizza –  I can make one for you in 15 minutes.
• Me says: thanks, but sorry dude, gotta go.
  

So how much is 75% of a pizza worth?

Technically, I think he was correct, and I admired his holding the party line on the “pay per slice” take-out-premium-pricing model. Yet, it was an interesting Customer Service experience – since the opportunity to generate a guaranteed $8, and still have another full pie out within 15 minutes (on a not so busy night) was lost.

Mr. Pizza Dude, I will be back again, and might even call ahead next time.

There’s no debating the societal impact of Mobile: Nokia stated at this weeks’ CES conference that now, over 50% of internet services are accessed on mobile phones, not from PCs. And, we all know that the level of mobile-only continues to rise (well past 20% in some demographic segments), forever changing landline survey options.

So what does that mean for Survey Researchers? Should each of us have our own $.99 survey App on iTunes? Not so fast, let’s think about this from a few perspectives.

Consistency: if the objective is to empower respondents with another mode of completing surveys, mobile certainly offers some interesting benefits. But as we’ve learned from the results of 1.0 vs. 2.0 web-based surveys (“Sexy Questions Dangerous Results?”), there will certainly be differences in the data when comparing mobile to web – and a range of new issues for which we have limited best practices (sampling, questionnaire length, visual layout & usability differences between BlackBerry and iPhones etc). As with each of the new tech options/advances, it’s critical to think through the “when, why and how” before we push the send button. As an industry, let’s avoid the temptation to over-focus on the ‘consistency’ of survey learning from Mobile vs PC, it’ll just get us into trouble, and distract us from creating end-client insight value from this exciting new channel.

Context: the benefits of using a mobile mode (either exclusively, or in conjunction with other methodologies) needs to take into account the contextual relevancy of using this platform. If you’re trying to get conference attendees to tell you what they thought of the speaker they just heard, that would be a intuitive application of a mobile platform. But don’t try to be cute, and offer people the choice to do a PC or mobile survey. A recent client survey found that under 5% of respondents chose the mobile option when given the choice between PC & mobile. And, you incur additional programming costs/delays to deploy the additional mode, for limited response rate upside, and questionable data consistency downside.

Future Focus: I recognize the research importance of making sure we know whether the “Next” button should be on the top or bottom of an iPhone screen. But the VP Marketing couldn’t care less – she/he wants to know how this mobile survey will help them better understand consumer needs. Consider the following App, which allows an iPhone user to scan a barcode, and get product rating information. This is where I believe mobile gets exciting for researchers, by linking real-time consumer behavior with attitudinal research communities (MROC’s). And dare I say, leveraging the power of the channel for exciting Qual opportunities from a new generation of Internet-savvy ethnographic researchers.

So, I say:  Dear <insert client name>, the next time I’m sitting on a plane, use my BlackBerry GPS locator (you can access it) and Frequent Flyer number (you have it) to send me a 5 question CSAT survey. I promise to respond. Sincerely, Bernie.

Note: element54 will be presenting new Mobile R&D – “The Moment of Truth” at the MRIA’s 50th annual National Conference on Monday May 30/2010. Stay tuned for details, and visit the conference website.

Links to Annie, Joel, Josh and Brandon (each to be updated as soon as available)

Annie Pettit of Lovestats: http://lovestats.wordpress.com/2010/01/15/1topic5blogs-the-only-thing-cell-phone-surveys-are-good-fer/

Joel Rubinson of the ARF:
Josh Mendelsohn of Chadwick Martin Bailey: http://lovestats.wordpress.com/2010/01/15/1topic5blogs-the-only-thing-cell-phone-surveys-are-good-fer/
Brandon Bertelsen:

Our first topic, is something which has been of utmost importance and urgency to element54 this year, and one where my partners and I have invested considerably, to advance the issue in our business (some previous posts on this site offer practical & prescriptive solutions).

Background/Issue:

We are witnessing an explosion of engaging new question types – intended to captivate respondents, and ensure the long-term viability of an online survey platform.

Research to-date has described “respondent engagement” as the notion that improving the usability and interest of Online question types will enhance respondents’ enjoyment, and the thoughtfulness of their survey experience. However, there has been little work to validate the myriad of new question types, and specifically, how response patterns vary across each type.

element54 conducted 2 transparent studies in 2009, which have been presented at several North American conferences (most recently, The Market Research Event, Las Vegas – Oct ’09)

Study #1 – “Sexy Questions, Dangerous Results?” (Data Consistency):

In January 2009, element54 and ResearchNow, conducted the first-ever fully transparent research study (full dataset available to MR peers), which examined the response patterns to various Online question types. The findings from this 2000 interview study were clear. Changing the visual layout of an online survey leads to differences in the data.

• 36% variance in how often people say they brush their teeth,
• Up to 8% overclaiming of behavioral product category consumption,
• Up to 10% understating of attitudinal “issue importance”.

Study #2 – “Eyes Don’t Lie” (Respondent Usability):

In May 2009, element54, along with partners UX Research & Consulting, and MBA Research, conducted the largest transparent “Eye Tracking” study on survey usability, with 100 one-on-one Qual/Quant interviews, using SMI Vision eye tracking software to validate respondent eye movements & patterns. The findings from this second study leveraged how Usability insights and applications can be applied to designing better surveys:

• Inference vs. Instruction; up to 40% of “2.0” survey questions aren’t read when visuals dominate the screen. It is certainly concerning if respondents are inferring our intent.
• Error Prevention & Recovery; ever tried to get off an elevator when you miss your floor – yes, you push other buttons to get off. When survey “error messages” are not linked to where the problem is, respondents are likely to change their answers to escape. This issue raises a host of related questions around how we treat “error” data once the respondent successfully clicks “Next”.

Next Steps:

Technology is headed in the right direction. To sustain the long-term viability of Online surveys, we do need to create appropriately engaging platforms for respondents. However, in the current space race, and range of available DIY software platforms – there’s no glory in getting to the moon first, if you crash land.  This is where best practices and standards can, and must catch up with all this exciting technology.

Here are the links to Annie, Joel, Josh and Brandon (soon).

Annie Pettit of Lovestats: http://lovestats.wordpress.com/2009/12/21/1-topic-5-blogs-rich-media-in-surveys/
Joel Rubinson of the ARF: http://blog.joelrubinson.net/2009/12/getting-the-most-out-of-online-research/
Josh Mendelsohn of Chadwick Martin Bailey:http://betterresearch.blogspot.com/2009/12/1-topic-5-blogs-interactive-questions.html
Brandon Bertelsen: link posted shortly at http://bertelsen.ca/

In the largest Usability study conducted on this issue (“Eyes Don’t Lie” – 2009), we have come across many ways in which marketing researchers likely compound the issue of respondent errors. The above image shows a sample “error message” from one of the studies’ interfaces. Note that the errors are highlighted with an arrow, and text saying “your answer is incomplete”.

• From a Usability point of view, this is what needs to be done – the respondent is directed to the specific problem lines where they had omitted an answer (which is much better than an error message on top of the screen saying “you must complete all questions below”).
• However, what is problematic is the visual cue to potentially ‘tick’ the answer that is closest to the red arrow (as the video of this respondent does show to be the case).

Recommendation: highlight the row, or indicate at the question/statement level where there is an omission or error. Anything else is likely adding error on top of error.