TRUTH AND LIES: CONSUMER PERCEPTION VS. DATA

TRUTH AND LIES: CONSUMER PERCEPTION VS. DATA Rosario Murguia, Consumer and Product Research Manager, Procter & Gamble Diana Ballard, Senior Consulting Statistician, Predictum Inc. Michael E. Haslam, PhD, Vice President of Application Development, Predictum Inc. Topic: Data Exploration Level: JMP Discovery Summit 2015 September 17 th 1

Background Motivation: Guide product design choices of rechargeable batteries and chargers Hypothesis: Consumer perception of their habits will be different vs. measured behavior o How often they used the charger? o How long it takes to charge the batteries? o How satisfied are they with speed of charging? Methodology: o 2 device Types o Central Location Three Month Sequential Monadic Use Test o Low, Mid, High Battery Capacities 2

Background Limitations: o Participants were not randomly chosen from the population Rechargeable battery users or open to buy Claimed to charge at least 3 times per month o Controlled use situation Minimum charging events Number of batteries Charger plugged all the time o Diary heightens attention to battery and charger performance Scope: Perception Reality Screener Survey Recall Survey Diary Charger Data 3

Four Data Sources Screening Survey Data (user) o o What are my perceptions of how I might respond to the testing; as seen on a pre-test survey. This survey screens for the appropriate types of users. Am I a heavy user? What type of devices do I use? etc. Diary Survey Data (user) o What I say I did during the test; using a self-collected diary. Recall Survey Data (user) o How I feel about what I did after the test; as seen on a post-test survey. Charger Data (device) o Time stamped event data from the battery chargers, collected during testing. 4

Survey Based Data Comes in Triple-S (SSS) survey data format. o o Platform independent, multi-lingual encoding of surveys, consisting of a file pair XML data tags file CSV raw data file Data constructs (XML tags) denote what is in the survey Questions Possible Answers Scoring etc. Converted to JMP tables for analysis using a custom built survey converter script. 5

Triple-S Format Example 6

Charger Data Consists of Time stamped, key events that denote changes in the charger. For example: o o Power On Charging Complete. Is captured as one data file per physical charger unit. Is not captured at uniform time intervals. Is not necessarily clean; events may be missing. Charger event change data must be converted to a form suitable for analysis; a Flat File (JMP) with State Changes and Counts. 7

Charger Data Example Cell Values are in voltage in Counts. 8

Converted Charger Data o Cell and LED States o Charging Status and Percent Charged 9

Analysis Challenges: Many datasets Screening survey Charger data Diary survey Recall survey Screening survey + Charger Screening survey + Diary survey Screening survey + Recall survey Charger + Diary survey Charger + Recall survey Diary survey + Recall survey 10

Key Questions 1. How often did they use the charger? 2. How long does it take to charge the batteries? 3. How satisfied are you with the charging speed? 11

Question 1: How often did they use the charger? We expected the surveys will not match the actual behavior, but we didn t know that consumers would tend to under report the number of charging events. Perception Screener Survey Recall Survey Diary Reality Charger Data 12

Screener Survey Decide which potential participants will charge the most frequently. 13

Did they use the charger at least 3 times? Perception Recall Survey Diary Reality 10% of the perception responses were not in agreement: 8% people saying YES, and 24% saying NO 14

Did they use the charger at least 3 times? Perception Recall Survey Reality Charger Data 31% of the perception responses were not accurate: 28% people saying YES, and 48% saying NO 15

How many times did they use the charger? Perception Reality Diary Charger Data In general, a person claimed fewer charging events in the diary than those measured by the charger. 16

Question 2: How long does it take to charge the batteries? We expected respondents to claim that charging time is important, but we found evidence that many forget the batteries in the charger. Perception Diary Reality Charger Data 17

How long does it take to charge the batteries? We expect the diary data to be closer to reality, and yet 74% of diary responses did not match the measured charging time. Points on the graph are the difference between participants claim in the Diary survey vs actual value calculated in the charger data. Zero values indicate that a participant told the truth. 18

Question 3: How satisfied are you with the charging speed? The responses in the survey and diary are not statistically independent. Perception Recall Survey Diary Reality 19

Question 3: How satisfied are you with the charging speed? 20

Question 3: How satisfied are you with the charging speed? 21

Conclusions 1. Quantified differences between claimed and actual use o Over 70% of respondents charged the batteries at least 3 times in a month as instructed. o Respondents under reported the number of charging events. o Charging speed perception does not match reality: 74% of diary responses did not match the measured charging time. 2. Insights about the methodology o Greater confidence in recruiting criteria and testing time. o We have the technology to read the charger every few seconds, but we don t need 5+ million rows of data! o Less value obtained from diary data. If used, need to include date and time to better match with measured data. 3. Few surprises o About 80% of the panel unplugged the charger during the time of the test. About 20% unplugged/replugged 10 or more times! 22

Questions? 23