Online Shopper Experience Comparative Survey

A comparative survey of the online shopper experience on Facebook Marketplace and eBay.

Context

Organization

This project was part of a quantitative research course from the UC Berkeley Extension Program. My goal was to expand my skillsets into mixed methods research, starting with surveys.

To guide my project, I designed a hypothetical research scenario. I imagined that I was back on the Facebook Marketplace buyer team (see examples of my past Marketplace projects here). In this scenario, our team wanted to understand how easily shoppers could discover items they were interested in on Marketplace (a key user goal in the discovery phase of the shopper journey). After months of investment, we were confident that we had identified and rectified the major UX issues (e.g., through the addition of search filters) and that we had surpassed our competitor, eBay. 

The goal of the research was to benchmark the current experience, with insights informing the product roadmap. For example, if Marketplace outperformed eBay in product discovery, we would shift our focus to other parts of the shopper journey. But if Marketplace was behind eBay, we would re-focus on helping shoppers find items they were interested in.

Goals

Business goal: Determine priorities for the product roadmap.

Research goals: Assess and compare ease of item discovery and shopper satisfaction on Marketplace and eBay. 

Team

My role: Lead researcher

Partners: Research advisors

Stakeholders: N/A

Study Design

Method

I chose surveys for this project. I wanted to understand how well shoppers could find items they were interested in, and confidently extrapolate to the broader population. This required asking different kind of questions and a larger sample than interviews. I fielded two surveys, one for Marketplace and one for eBay. Both surveys had identical questions and response options, with the platform name adjusted accordingly.

Sample

48 online shoppers (24 per survey) who had shopped on Marketplace or eBay in the last six months. I recruited respondents via my professional and personal network, as well as SurveySwap (a platform for finding survey respondents).

Tools

Qualtrics, SurveySwap, Google Suite

Process

Sample Size

An important part of the study design was determining my sample size. I determined this by starting with the confidence levels I wanted to produce. My goal was an 80% confidence level and 20% margin of error. To reach this, I would need at least 23 respondents per survey.

Survey Questions

Each survey consisted of 21 questions. To measure ease of item discovery, I asked respondents to rate how often they found items that matched their interests when shopping on the platform. To measure shopper satisfaction, I adapted three items from the System Usability Questionnaire, focusing on simplicity, ease of use, and intuitiveness. The remaining questions addressed respondent eligibility, secondary research questions (e.g., barriers to item discovery), context of use, respondent demographics, and other topics.

Analysis

After cleaning and coding the data, I used two-tailed t-tests, visualizations, and thematic analysis of open-ended responses to analyze the data. T-tests enabled me to identify whether there was a significant difference between platforms in ease of item discovery and shopper satisfaction. I used the standard p<0.05 threshold to detect significance, as well as effect size to quantify the size of the difference.

Key Insights

Demographics

Respondents tended to be female, White or Asian, and under 45 years old.

Marketplace shoppers were more likely to be Asian and older than eBay shoppers, who were more likely to be White and younger.

Ease of product discovery & satisfaction

It was easier for shoppers to find items they were interested in on eBay than Marketplace.

*P-value: 0.01, medium effect size. Question text: "When you shop on [platform] (including browsing), how often do you find items that match your interests?". Response options: Always (5), Often (4), Sometimes (3), Rarely (2), Never (1)

Shoppers found eBay to be simpler than Marketplace.

*P-value: 0.007, large effect size. Question text: "To what extent would you agree or disagree with the following statements, based on the last time you shopped on Facebook Marketplace? 'I found the website to be simple.'" Response options from 5 (Strongly Agree) to 1 (Strongly Disagree)

Shoppers thought the eBay website was easier to use than Marketplace.

*P-value: 0.004, large effect size. Question text: "To what extent would you agree or disagree with the following statements, based on the last time you shopped on Facebook Marketplace? 'I thought the website was easy to use.'" Response options from 5 (Strongly Agree) to 1 (Strongly Disagree)

Shoppers also thought the eBay website was more intuitive than Marketplace.

*P-value: 0.004, large effect size. Question text: "To what extent would you agree or disagree with the following statements, based on the last time you shopped on Facebook Marketplace? 'I found the website very intuitive.'" Response options from 5 (Strongly Agree) to 1 (Strongly Disagree)

Product discovery barriers

Low trust in sellers was a top barrier to finding items of interest on both Marketplace and eBay. Other top barriers were sellers being too far away and lack of search results (Marketplace) and high shipping costs and lack of item information (eBay).

Recommendations

Based on the insights, my primary recommendation would be to prioritize product discovery (i.e., helping shoppers find items they were interested in) in the product roadmap. I also identified opportunity areas such as:

  • Connecting shoppers with sellers closer to them

  • Increasing the volume of search results

  • Increasing trust in sellers

For example, perhaps the distance filter was not working as intended and shoppers were inadvertently being shown sellers too far away from them. Alternatively, maybe we did not have enough listing volume across certain geographic areas or in certain high-demand categories. These were areas we could investigate through product testing and analyzing supply and demand. In addition, how did shoppers evaluate seller trustworthiness, and what helped or hindered sellers from signaling their trustworthiness? Further research would be needed to answer these questions.

Finally, I would recommend running a similar benchmark study after implementing changes, to see how much progress we had made.

Impact

Although this project was based on a hypothetical scenario (and thus did not have actual impact), I would have measured impact in the following ways:

  • Product discovery being called out as a priority on the product roadmap

  • Specific projects initiated related to seller location, search result volume, and seller trust

  • Improved results for Marketplace in future benchmark studies