Context

The product had lofty long-term revenue goals. To help meet those goals, an auction buying model was introduced, supplanting a fixed price buying model for ad placements. The business aim was to increase the revenue per user as well as set the stage to attract more diverse user groups who are cost-conscious. As the user researcher, over the course of 1 year, I de-risked the launch before going live, monitored it while it was live, and evaluated the feature after users had experience with it.

Research Goals

Provide end-to-end research support over the course of the product development cycle to ensure the best possible user experience. This included:

1. De-risk the launch by understanding the concerns and expectations of the auction buying model and how user segments differ.

2. Evaluate designs for usability and clarity.

3. Monitor the experience of auction after launch to improve the experience as needed.

STAKEHOLDERS

MethodS

Pre-launch:

1. 12 interviews with users of varying expertise to understand their experience with auctions on other platforms, including the pros, cons, concerns, and expectations.

2. Survey with 300 users to validate and quantify findings from the interviews, and understand differences by user type.

3. Concept testing designs (skewing toward novice users) to ensure the auction UI is easy to use and understand.

Post-launch:

1. Work with sales and marketing to monitor customer service tickets themes and changes in the customer satisfaction survey.

2. Partner with data science to understand changes in behavior (e.g. spend) by user type.

3. 10 interviews with users who have used auction to deeply understand the pros, cons, and challenges with the new buying model.

impact

Pre-launch:

User concerns with switching to auction informed reactive communications for the customer support team to prepare ahead of launch, allowing more thoughtful and consistent communication.

User expectations and mental models informed the initial designs by helping order new information and the terminology used in the UI.

Post-launch:

Issues users were having using the product centered around how the product worked from a technical perspective. These issues were brought to life in presentations to leadership to ultimately change the ad serving algorithm in favor of our self-serve advertisers.

Additionally, messaging was added to the UI to better set users’ expectations while a technical fix was being worked on.

I noted that future launches with such a fundamental switch more strongly message the switch ahead of time, such as interrupting product usage with messaging ahead of launch.